Expressway card receiving and sending machine service life prediction method based on multi-modal data fusion and double-branch LSTM

Abstract

According to the highway card receiving and sending machine service life prediction method based on the multi-modal data fusion and the double-branch LSTM, the association degree of heterogeneous data can be improved through a space-time alignment algorithm, the problem that information redundancy and missing coexist in a traditional single-modal sensor is effectively solved, and the integrity and accuracy of the data are improved; a low-frequency trend item and a high-frequency transient item are respectively processed through the double-branch LSTM architecture, the capturing capability of the model for fault characteristics of different time scales is enhanced, the time-frequency characteristics of the fault characteristics can be distinguished, the response speed of the model to vibration signal sudden change is improved, the double-branch LSTM architecture can better capture the fault characteristics of different frequencies of the highway card receiving and sending machine under the complex working condition, and the fault detection accuracy of the highway card receiving and sending machine is improved. The prediction accuracy is improved; a weighted gating fusion mechanism is utilized, the fusion proportion of trend features and transient features is dynamically adjusted, the residual life probability distribution of the equipment is output, the weighted gating fusion mechanism better conforms to the life prediction requirement of the highway card receiving and sending machine, and a fusion strategy can be adjusted in real time according to the running state of the equipment.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation technology, and in particular to a method for predicting the life of a highway card transmitter and receiver based on multimodal data fusion and a dual-branch LSTM. Background Art

[0002] The lifespan prediction system for toll lane card dispensers is a solution based on intelligent monitoring and data analysis technologies. It aims to predict the remaining useful life of equipment by monitoring equipment status in real time and analyzing key parameters, thereby optimizing maintenance strategies, reducing operating costs, and improving equipment reliability and stability. However, existing toll lane card dispenser lifespan prediction systems still have the following drawbacks:

[0003] (1) Single data dimension: Traditional systems rely on only a single sensor (such as a current sensor) to monitor equipment status. This makes it difficult to fully capture multi-source heterogeneous features such as mechanical component wear, bearing vibration, and electromagnetic interference, resulting in incomplete monitoring information and affecting the accurate judgment of the equipment health status.

[0004] (2) Insufficient timing modeling capabilities: The prediction models used by existing prediction systems (such as linear regression and SVM) have difficulties in processing nonlinear timing relationships under the dynamic working conditions of card transceivers. For example, they cannot effectively analyze the coupling effect between high-frequency vibration signals and low-frequency temperature changes, making it difficult to accurately predict the life of the equipment.

[0005] (3) Poor adaptability to scenarios: Expressway card transmitters face extreme environments (-40℃~85℃, high dust pollution) and complex working conditions (average daily processing volume ≥100,000 times). The existing system is not stable enough and it is difficult to operate stably and provide accurate life prediction under such harsh conditions. Summary of the Invention

[0006] The present invention aims to solve at least one of the technical problems raised in the above-mentioned background technology, and provide a highway card transceiver life prediction method based on multimodal data fusion and dual-branch LSTM, which can improve the accuracy of highway card transceiver life prediction.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is:

[0008] A highway card transceiver life prediction method based on multimodal data fusion and dual-branch LSTM includes the following steps:

[0009] S1, collecting multimodal data related to the operating status of the highway card transceiver, the multimodal data including cumulative operating time, vibration signal, temperature signal and visual image signal, and achieving spatiotemporal alignment of the data through timestamp synchronization;

[0010] S2, preprocessing the collected multimodal data to obtain data features, and dividing the obtained data features into low-frequency trend data and high-frequency transient data according to the frequency characteristics of the data;

[0011] S3, constructing a two-branch LSTM network having a low-frequency trend branch and a high-frequency transient branch, performing feature extraction on the low-frequency trend data through the low-frequency trend branch to obtain low-frequency trend features; and performing feature extraction on the high-frequency transient data through the high-frequency transient branch to obtain high-frequency transient features;

[0012] S4, dynamically adjusts the fusion ratio of low-frequency trend features and high-frequency transient features through the weighted gated fusion mechanism;

[0013] S5, performing feature fusion on the low-frequency trend feature and the high-frequency transient feature according to the obtained fusion ratio to obtain a fusion feature, and finally outputting a probability distribution of the remaining life of the highway card transceiver according to the fusion feature.

[0014] Furthermore, in step S1, the vibration signal collection method is: collecting vibration data of the highway card transceiver through a vibration sensor; the temperature signal collection method is: using a temperature sensor with an accuracy of ±0.5°C to collect temperature data of the highway card transceiver; the visual image collection method is: using an industrial camera to collect image data of the highway card transceiver; and the cumulative operating time data of the highway card transceiver is collected through a timing module installed on the highway card transceiver.

[0015] Furthermore, in step S2, the method for preprocessing the collected multimodal data is:

[0016] Processing the accumulated running time to remove possible noise or outliers in the accumulated running time signal;

[0017] The vibration signal preprocessing method involves removing high-frequency noise above 20 kHz through wavelet packet decomposition and calculating the energy proportion in the 1-100 Hz frequency band to characterize the defect characteristics of the bearing rolling elements used in highway card transceivers. The bearing rolling element defect characteristics are used as the vibration time-frequency domain characteristics.

[0018] The temperature signal preprocessing method is as follows: the temperature signal is filtered using a sliding window mean filter to obtain the sliding window mean at different times. The temperature rise rate is calculated based on the sliding window mean at different times. The obtained temperature rise rate is compared with a set threshold. If the temperature rise rate is greater than the set threshold, it is determined that an abnormal heating event has been detected. When an abnormal heating event is detected, the dynamic range of the temperature signal is automatically adjusted according to the ambient temperature.

[0019] The preprocessing method of the visual image signal is to use YOLOv5s to detect the surface cracks of the gear set and calculate the crack area growth rate.

[0020] Furthermore, the low-frequency trend data includes the cumulative running time and the sliding window mean; the high-frequency transient data includes the vibration time-frequency domain characteristics and the crack area growth rate.

[0021] Furthermore, when the sliding window mean filter is used to filter the temperature signal, the window size is set to 60 seconds and the step size is set to 1 second.

[0022] Furthermore, in step S3, the low-frequency trend branch and the high-frequency transient branch both include an input layer, a hidden layer, and an output layer. The input layer of the low-frequency trend branch takes low-frequency trend data as input, performs feature extraction on the input low-frequency trend data through the hidden layer, and finally outputs the low-frequency trend features through the output layer; the input layer of the high-frequency transient branch takes high-frequency transient data as input, performs feature extraction on the input high-frequency transient data through the hidden layer, and finally outputs the probability of sudden failure as a high-frequency trend feature through the output layer.

[0023] Furthermore, the hidden layer of the low-frequency trend branch includes two LSTM layers, each LSTM layer includes 128 LSTM units, and the activation function used in the gating mechanism inside the LSTM unit is tanh.

[0024] Furthermore, in step S3, the hidden layer of the high-frequency trend branch is an LSTM network structure that combines a one-dimensional convolutional neural network 1DCNN and an attention mechanism.

[0025] Furthermore, the following fusion formula W is adopted fusion Dynamically adjust the fusion ratio of low-frequency trend features and high-frequency transient features:

[0026] W fusion =σ(Wg·[trend_score; transient_score])

[0027] Where the dynamic weight Wg is optimized by gradient backpropagation; trend_score refers to the output of the low-frequency trend branch; transient_score refers to the output of the high-frequency transient branch; σ(*) represents the Sigmoid function.

[0028] Furthermore, the vibration sensor adopts a three-axis MEMS accelerometer with a sampling rate of 10kHz and a range of ±50g; the temperature sensor adopts a K-type thermocouple and uses a MAX31855 chip for cold-end compensation with an accuracy of ±0.5°C; the vibration sensor and the temperature sensor are both protected by an IP68 protection grade casing.

[0029] Due to the adoption of the above technical solution, the present invention has the following beneficial effects:

[0030] The highway card transceiver life prediction method based on multimodal data fusion and dual-branch LSTM of the present invention can improve the correlation of heterogeneous data through the spatiotemporal alignment algorithm, effectively solve the problem of coexistence of information redundancy and missing of traditional single-modal sensors, and improve the integrity and accuracy of data; through the dual-branch LSTM architecture, low-frequency trend items (temperature, cumulative operating time) and high-frequency transient items (vibration, local image anomalies) are processed respectively, the model's ability to capture fault characteristics of different time scales is enhanced, the time-frequency characteristics of fault characteristics can be distinguished, and the model's response speed to sudden changes in vibration signals is improved. The dual-branch LSTM architecture can better capture the fault characteristics of different frequencies under complex working conditions of highway card transceivers and improve prediction accuracy; using the weighted gated fusion mechanism, the fusion ratio of trend characteristics and transient characteristics is dynamically adjusted, and the probability distribution of the remaining life of the equipment is output. The weighted gated fusion mechanism is more in line with the life prediction needs of highway card transceivers and can adjust the fusion strategy in real time according to the operating status of the equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 The present invention is a flowchart of a highway card transceiver life prediction method based on multimodal data fusion and dual-branch LSTM according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this invention pertains. The terms used in this specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0034] See Figure 1 A preferred embodiment of the present invention provides a highway card transceiver life prediction method based on multimodal data fusion and dual-branch LSTM, which is used for health status monitoring and preventive maintenance of toll lane card transceivers. The highway card transceiver life prediction method based on multimodal data fusion and dual-branch LSTM includes the following steps:

[0035] S1, collects multimodal data related to the operating status of the highway card transceiver, the multimodal data including cumulative operating time, vibration signal, temperature signal and visual image signal, and synchronizes the timestamp to achieve spatiotemporal alignment of the data.

[0036] In this embodiment, a timing module can be integrated into the control system of the expressway card transmitter / receiver. This module begins operating from the moment the expressway card transmitter / receiver is first installed and activated, continuously recording the expressway card transmitter / receiver's operating time. Whenever the expressway card transmitter / receiver is in operation, the timing module accumulates the time; when the expressway card transmitter / receiver stops operating, the timing is paused. Thus, by reading the data from the timing module, the accumulated operating time of the expressway card transmitter / receiver can be obtained. The structure of the timing module is conventional and will not be detailed here for the sake of brevity.

[0037] In this implementation, vibration data from highway card transceivers is collected using a vibration sensor. This vibration sensor utilizes a three-axis MEMS accelerometer, model ADXL357. This sensor has a sampling rate of 10kHz and a range of ±50g. Protected by an IP68-rated enclosure, it operates stably in environments with temperatures ranging from -40°C to 85°C and high dust pollution. The vibration sensor is installed within the highway card transceiver near key mechanical components (such as gears and bearings). This sensor can more directly and accurately capture vibration signals reflecting the equipment's operating status, such as abnormal vibrations caused by gear wear or bearing failure, facilitating the timely identification of potential equipment issues.

[0038] Temperature signal acquisition uses a temperature sensor with an accuracy of ±0.5°C to collect temperature data from the highway card transceiver. Installed in key heat-generating components within the card transceiver (such as motors and circuit boards), the temperature sensor provides timely information on the heating of the motors and circuit boards during operation. Any abnormal temperature rise can be quickly detected, allowing for identification of faults such as overload or short circuits in the motors and circuit boards. In this embodiment, the temperature sensor uses a K-type thermocouple and a MAX31855 chip for cold-junction compensation to ensure accurate temperature measurement. The sensor is protected by an IP68-rated enclosure, ensuring stable operation in environments with temperatures ranging from -40°C to 85°C and high dust pollution.

[0039] Industrial cameras are used to capture visual image signals from key locations within the highway card transceiver. These locations primarily include mechanical transmission components, key connections and moving parts, and key electrical control areas. Gear sets are a key area for capture. Gear sets frequently engage and rotate during operation, making them prone to wear and cracking. By capturing surface images of the gear sets with industrial cameras and calculating the crack area growth rate, potential faults can be detected promptly. Chains and sprockets are also important areas to monitor. Chains can stretch and break over time, while sprockets can wear. Capturing images of these locations allows for intuitive observation of their surface condition and operating status. Key connections and moving parts refer to rotating components within the highway card transceiver, such as bearings. Failure in these components can affect normal operation. Capturing images of bearings allows for verification of lubrication status and surface damage. Key connections and moving parts also include various connecting shafts and joints, which frequently move during operation and are prone to loosening and wear. Capturing images of these components helps detect anomalies promptly, ensuring stable and reliable operation. The key electrical control area refers to the circuit board, which is the core of the electrical control system for highway card transceivers and contains numerous electronic components. Capturing circuit board images can detect problems such as overheating, discoloration, and broken pins. Furthermore, poor contact at electrical connections, such as plugs and sockets, can easily lead to failures. Image capture allows visualization of the connection status and surface condition, helping to prevent electrical failures. Specifically, in this embodiment, a 6-megapixel industrial camera, model MV-CA060-10GM, with a frame rate of 30 fps, is used to capture visual image signals.

[0040] The embodiment of the present invention uses a sensor with the above parameters, which can more accurately collect multimodal data of highway card transceivers, meet the monitoring needs of highway card transceivers, and in the harsh working environment of highway card transceivers, the sensor with the above parameters can ensure the reliable operation of the sensor and improve the accuracy of data collection, which is the key to achieving accurate life prediction.

[0041] S2, preprocessing the collected multimodal data to obtain data features, and dividing the obtained data features into low-frequency trend data and high-frequency transient data according to the frequency characteristics of the data.

[0042] In this embodiment, the data features in step S2 include cumulative duration, vibration time-frequency domain features, sliding window mean of temperature, and crack area growth rate. Specifically, the method for preprocessing the collected multimodal data in step S2 is:

[0043] The accumulated runtime is processed to remove any noise or outliers that may be present in the accumulated runtime signal. While the collected accumulated runtime signal is inherently stable, preprocessing is typically required to ensure its integration into the overall data processing pipeline and improve model performance. Preprocessing is performed to remove any noise or outliers that may be present. For example, a timing device may experience a brief malfunction, causing unusual jumps in the accumulated runtime. Preprocessing can identify and correct these anomalies, allowing the model to more accurately learn the relationship between accumulated runtime and device status, thereby improving model performance.

[0044] The vibration signal preprocessing method is to remove high-frequency noise above 20kHz through wavelet packet decomposition and calculate the energy proportion of the 1-100Hz frequency band to characterize the defect characteristics of the bearing rolling elements used in highway card transceivers. The defect characteristics of the bearing rolling elements are used as the vibration time-frequency domain characteristics.

[0045] The preprocessing method of the temperature signal is as follows: the temperature signal is filtered using a sliding window mean filter to obtain the sliding window mean at different times, the temperature rise rate is calculated based on the sliding window mean at different times, and the obtained temperature rise rate is compared with the set threshold. If the temperature rise rate is greater than the set threshold, it is judged that an abnormal heating event has been detected. After an abnormal heating event is detected, the dynamic range of the temperature signal is automatically adjusted according to the ambient temperature (-40℃~85℃).

[0046] Specifically, when using a sliding window mean filter to filter the temperature signal, the window size is set to 60 seconds and the step size is set to 1 second. During this process, the collected raw temperature data is processed according to the set window size and step size to obtain the sliding window mean at different times. For example, a mean is calculated for the temperature data from the 1st to the 60th second. The window is then slid for 1 second, and another mean is calculated for the temperature data from the 2nd to the 61st second, and so on.

[0047] The temperature rise rate reflects how quickly the temperature changes over time. For example, if the current sliding window average is T1 and the next sliding window average (a step size of 1 second later) is T2, the temperature rise rate = (T2 - T1) / 1 second. This calculation method can monitor temperature changes in real time and detect abnormal heating events.

[0048] Detecting abnormal heating events in highway card transceivers is primarily based on the temperature rise rate, which is determined by a set threshold. Specifically, a threshold for the temperature rise rate is set. For example, under normal circumstances, the temperature rise rate fluctuates within a specific range (0.15°C / min-0.25°C / min). If the temperature rise rate exceeds this threshold, abnormal heating may occur. By continuously monitoring the temperature rise rate and comparing it with the set threshold, abnormal heating can be detected in a timely manner, allowing the device's health status to be assessed and appropriate measures to be taken.

[0049] Automatically adjust the dynamic range of the temperature signal based on the ambient temperature range (-40℃~85℃) of the highway card reader. Common adjustment methods include standardization and segmented mapping:

[0050] (1) Standardization adjustment method: This method unifies the temperature signal into a standard numerical range through calculation, which is convenient for subsequent processing and analysis. Assume that the collected temperature signal range is [T min ,T max ], to map it to the standard interval [0,1], for a certain temperature value T, the adjustment formula is For example, in winter, the ambient temperature is between -40℃ and 0℃, and the temperature value collected by the card transceiver temperature sensor ranges from -30℃ to -10℃. When the collected temperature T=-20℃, T min =-30℃, T max =-10℃, calculated according to the formula: This maps the original temperature value to the [0,1] interval, making it easier to perform subsequent comparisons and analyses.

[0051] (2) Segmented mapping adjustment method: Considering the differences in the impact of different temperature ranges on the equipment, the entire ambient temperature range (-40℃~85℃) is divided into multiple sub-ranges, and each sub-range corresponds to a different mapping rule. For example, three sub-ranges are divided: [-40℃, 0℃), [0℃, 50℃), and [50℃, 85℃]. In the [-40℃, 0℃) range, the temperature signal is mapped to [0, 0.3]; in the [0℃, 50℃) range, it is mapped to [0.3, 0.7]; in the [50℃, 85℃] range, it is mapped to [0.7, 1]. When the collected temperature T = 10℃, it is in the [0℃, 50℃) range. According to the mapping rule, assuming linear mapping: Through such segmented mapping, the changing characteristics of temperature signals in different temperature ranges can be reflected more finely, thereby improving the accuracy of equipment temperature monitoring.

[0052] After detecting an abnormal heating event, the dynamic range of the temperature signal is adjusted. The main purpose is to more accurately monitor and analyze the temperature status of the device to ensure the accuracy of subsequent device health status assessment (improve data monitoring accuracy, adapt to complex environmental changes, and assist in fault diagnosis and maintenance decision-making). The details are as follows:

[0053] (1) Improve data monitoring accuracy: Under normal circumstances, the device temperature is in a relatively stable range, and the temperature signal changes in a relatively regular manner. However, when an abnormal heating event is detected, the change amplitude and trend of the temperature signal will change significantly. At this time, the original temperature signal dynamic range may not accurately reflect the actual temperature changes of the device. For example, when abnormal heating occurs, the temperature may rise rapidly to a level close to or even exceeding the upper limit of the normal monitoring range. If the dynamic range is not adjusted, some temperature data may exceed the monitoring range, resulting in the loss of key information and affecting the judgment of equipment failure. By automatically adjusting the temperature signal dynamic range according to the ambient temperature, the monitoring range can be re-optimized to ensure that subtle changes in temperature during abnormal heating can be captured more accurately, providing more accurate data support for subsequent fault diagnosis.

[0054] (2) Adapting to complex environmental changes: Highway card transceivers operate in extreme environments (-40°C to 85°C), and the ambient temperature itself fluctuates greatly. Under different ambient temperatures, the normal operating temperature range of the equipment and the criteria for determining abnormal heating are also different. When an abnormal heating event is detected, adjusting the dynamic range of the temperature signal according to the current ambient temperature can enable the system to better adapt to environmental changes and more reasonably evaluate the temperature status of the equipment. For example, in a low-temperature environment (such as -40°C), the normal operating temperature of the equipment may be relatively low, and the threshold for abnormal heating is also correspondingly low; while in a high-temperature environment (such as 85°C), the normal operating temperature of the equipment is higher, and the criteria for determining abnormal heating are also higher. Automatically adjusting the dynamic range of the temperature signal allows the system to accurately determine whether the equipment is in an abnormal heating state under different environmental conditions, thereby improving the reliability of equipment health monitoring.

[0055] (3) Assisting fault diagnosis and maintenance decision-making: An accurate dynamic range of temperature signals is crucial for subsequent fault diagnosis and maintenance decisions. When an abnormal heating event is detected and the dynamic range is adjusted, the system can more clearly analyze the temperature change trend and the degree of abnormality. This helps maintenance personnel to more accurately determine the type, severity, and possible causes of equipment failures, so that they can formulate reasonable maintenance plans in a timely manner and take targeted maintenance measures. For example, if the temperature continues to rise rapidly and exceeds a certain threshold after adjusting the dynamic range, it may mean that there is a serious fault inside the equipment and it needs to be shut down for maintenance immediately; if the temperature only rises abnormally for a short time and quickly returns to normal, it may be just an accidental overload problem, and the equipment operating parameters can be appropriately adjusted or regular inspections can be carried out.

[0056] The preprocessing method of the visual image signal is to use YOLOv5s to detect the surface cracks of the gear set and calculate the crack area growth rate.

[0057] Furthermore, within the entire dataset, different types of data may exist at different magnitudes and ranges. During preprocessing, the accumulated runtime, vibration signals, temperature signals, and visual image signals are adjusted to appropriate scales to ensure that the magnitudes of different data types match. This prevents an imbalance in the model's focus on different features during training, thereby improving the model's accuracy and stability.

[0058] S3, construct a two-branch LSTM network with a low-frequency trend branch and a high-frequency transient branch, extract features from low-frequency trend data through the low-frequency trend branch to obtain low-frequency trend features; and extract features from high-frequency transient data through the high-frequency transient branch to obtain high-frequency transient features.

[0059] In this implementation, low-frequency trend data includes cumulative operating time and a sliding window mean; high-frequency transient data includes vibration time-frequency characteristics and crack area growth rate. Low-frequency trend data typically reflects the changing trends of the equipment over a longer time scale. The sliding window mean of temperature data reflects the overall temperature level and its slow changes over a period of time. Cumulative operating time records the degree of equipment usage from a temporal perspective. These data change relatively slowly over time, which is consistent with the characteristics of low-frequency trend data and therefore falls under this category.

[0060] Both the low-frequency trend branch and the high-frequency transient branch consist of an input layer, a hidden layer, and an output layer. The low-frequency trend branch's input layer takes low-frequency trend data as input, extracts features from this input through the hidden layer, and ultimately outputs low-frequency trend features through the output layer. These low-frequency trend features represent a score of the device's long-term degradation trend, taking into account multiple factors such as the sliding window mean of the temperature sensor and the accumulated operating time. Specifically, the low-frequency trend branch's hidden layer consists of two LSTM layers, each containing 128 LSTM units. The activation function used in the gating mechanism within the LSTM units is tanh.

[0061] The input layer of the high-frequency trend branch takes high-frequency transient data as input, extracts features from the input high-frequency transient data through the hidden layer, and finally outputs the probability of sudden failure as a high-frequency transient feature through the output layer. In this embodiment, the hidden layer of the high-frequency trend branch is an LSTM network structure that combines a one-dimensional convolutional neural network 1D CNN and an attention mechanism. In one embodiment, the local features of the input data can be extracted through the 1D CNN layer, the output of the 1D CNN can be received through the LSTM layer and long-term dependencies can be captured. Finally, the attention mechanism is applied to the output of the LSTM to highlight important features. In another embodiment, the local features of the input data can also be extracted through the 1D CNN layer, and the long-term dependencies of the input data can be captured through the LSTM layer. Then, the output features of the 1D CNN and LSTM are spliced ​​or weighted summed. Finally, the attention mechanism is applied to the spliced ​​or weighted summed output to highlight important features.

[0062] S4 dynamically adjusts the fusion ratio of low-frequency trend features and high-frequency transient features through the weighted gated fusion mechanism.

[0063] In this embodiment, the following fusion formula W is used fusion Dynamically adjust the fusion ratio of low-frequency trend features and high-frequency transient features:

[0064] W fusion =σ(Wg·[trend_score; transient_score])

[0065] Wherein, the dynamic weight Wg is optimized by gradient back propagation;

[0066] The trend_score is the output of the low-frequency trend branch, which represents the score of the device's long-term degradation trend. In a two-branch LSTM network, the low-frequency trend branch takes the sliding window mean of the temperature sensor and the accumulated runtime as input. The LSTM network (e.g., a two-layer, 128-unit, tanh activation function) calculates and outputs a result representing the device's long-term degradation trend. This result is the trend_score. The trend_score reflects the degree to which the device's overall performance has gradually declined over time due to factors such as temperature changes and increased accumulated runtime.

[0067] The transient_score represents the output of the high-frequency transient branch, which represents the score of sudden equipment failure signs. The high-frequency transient branch takes the time-frequency domain characteristics of the vibration signal and the crack area growth rate of the image as input. It processes the data through an LSTM network structure that includes a 1D CNN (3×3 convolution kernel, 2×2 pooling window) and an attention mechanism. It derives the results of sudden equipment failure signs (such as gear tooth breakage and impact). This result is represented by the transient_score, which can quickly capture sudden abnormalities during equipment operation, such as abnormal vibration caused by sudden damage to a certain equipment component or the sudden expansion of a crack in a critical part.

[0068] σ(*) represents the Sigmoid function, which maps the weighted calculation result (Wg ·〔trend_score;transient_score〕) to between 0 and 1, thereby obtaining the dynamic weight W fusion .

[0069] The dynamic weight Wg is optimized through gradient back propagation and is used to dynamically adjust the fusion ratio of low-frequency trend features and high-frequency transient features, and then output the probability distribution of the remaining life of the equipment to achieve accurate prediction of the life of highway card transceivers.

[0070] S5, based on the obtained fusion ratio, the low-frequency trend feature and the high-frequency transient feature are fused to obtain a fusion feature, and finally the remaining life probability distribution of the highway card transceiver is output based on the fusion feature.

[0071] For example, when the output remaining life (RUL) is 85% confidence interval [120, 150] hours, it means that when predicting the remaining life of a highway card transceiver, there is an 85% probability that its remaining life will be between 120 and 150 hours. For highway card transceivers, accurate RUL prediction is crucial for preventive maintenance. It allows for early maintenance planning, the preparation of spare parts, and the reduction of toll lane congestion caused by equipment failure. A confidence interval is an interval estimation method. An 85% confidence level indicates that if the same life prediction is performed multiple times, theoretically there is an 85% chance that the equipment will fail within the next 120 to 150 hours, providing operations and maintenance personnel with a more reliable timeframe within which to schedule maintenance work.

[0072] The embodiment of the present invention also puts the present invention's highway card transceiver life prediction method based on multimodal data fusion and dual-branch LSTM into practical application. On the card transceiver on the Guiliu Expressway section in Guangxi, according to the above-mentioned collection method, vibration, temperature and visual image data are synchronously collected. The collection is continued for one year, and 300,000 sample data are obtained. The collected data is preprocessed by wavelet packet decomposition, sliding window mean filtering, image crack detection and other preprocessing operations to extract the corresponding features. The dual-branch LSTM model is trained using the AdamW optimizer (learning rate 1e-4, weight decay = 1e-2) and the weighted MSE loss function. After training, the real-time preprocessed data is input to obtain the probability distribution of the remaining life of the equipment. After evaluation, the highway card transceiver life prediction method based on multimodal data fusion and dual-branch LSTM of the present invention increases the correlation of heterogeneous data to 92% through the spatiotemporal alignment algorithm, effectively solving the problem of coexistence of information redundancy and loss in traditional single-modal sensors, and improving data integrity and accuracy; the accuracy of early weak fault identification is increased by 38%, and the remaining service life (RUL) prediction error (MAE) is reduced to 1.8% (compared with the traditional ARIMA model remaining service life prediction error (MAE) of 12.3%), which significantly improves the accuracy of life prediction.

[0073] The highway card transceiver life prediction method based on multimodal data fusion and dual-branch LSTM of the present invention can improve the correlation of heterogeneous data through the spatiotemporal alignment algorithm, effectively solve the problem of coexistence of information redundancy and missing of traditional single-modal sensors, and improve the integrity and accuracy of data; through the dual-branch LSTM architecture, low-frequency trend items (temperature, cumulative operating time) and high-frequency transient items (vibration, local image anomalies) are processed respectively, the model's ability to capture fault characteristics of different time scales is enhanced, the time-frequency characteristics of fault characteristics can be distinguished, and the model's response speed to sudden changes in vibration signals is improved. The dual-branch LSTM architecture can better capture the fault characteristics of different frequencies under complex working conditions of highway card transceivers and improve prediction accuracy; using the weighted gated fusion mechanism, the fusion ratio of trend characteristics and transient characteristics is dynamically adjusted, and the probability distribution of the remaining life of the equipment is output. The weighted gated fusion mechanism is more in line with the life prediction needs of highway card transceivers and can adjust the fusion strategy in real time according to the operating status of the equipment.

[0074] In addition, this embodiment uses edge computing terminals (TensorRT acceleration) to process data, achieving a single-node 500TPS throughput, reducing data transmission pressure and improving system real-time performance. Edge computing and high-throughput design can reduce data transmission delays and enable the system to quickly respond to changes in device status. Edge computing and high throughput bring about the effects of reducing data transmission delays and quickly responding to changes in device status, mainly due to the collaborative work of multimodal data acquisition, preprocessing, and dual-branch LSTM feature extraction steps with edge computing terminals (TensorRT acceleration), specifically:

[0075] During the multimodal data acquisition and preprocessing phase, data is collected using a variety of sensors (vibration sensors, temperature sensors, and visual image acquisition devices). The NTP protocol is used to synchronize timestamps with an accuracy of ±1ms to achieve spatiotemporal alignment. Subsequently, in the preprocessing phase, the vibration signal, temperature signal, and visual image data are subjected to denoising and feature adjustment. These preliminary steps ensure data accuracy and availability, laying the foundation for efficient data processing on edge computing terminals. If the collected data is inaccurate or contains a large amount of noise, even with powerful edge computing capabilities, it will be difficult to quickly and accurately analyze changes in device status.

[0076] During the dual-branch LSTM feature extraction and dynamic fusion step, the dual-branch LSTM network extracts features from low-frequency trend data and high-frequency transient data, respectively. This process effectively captures device fault characteristics at different timescales. A weighted gated fusion mechanism then dynamically adjusts the feature fusion ratio to output a probability distribution for the device's remaining life. Edge computing terminals (with TensorRT acceleration) play a role in this process, accelerating the computation of the dual-branch LSTM network. TensorRT is optimized for deep learning models, improving model inference speed. For example, when processing large amounts of vibration, temperature, and image data, TensorRT acceleration enables the dual-branch LSTM network to quickly complete feature extraction and fusion calculations, reducing processing time and, in turn, data transmission latency. This allows the system to quickly respond to device status changes, achieving a single-node throughput of 500 TPS (Transactions Per Second), and alleviating data transmission pressure.

[0077] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, apparatus, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable non-transitory storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0078] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatuses, and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 A means for specifying functions in one or more processes and / or one or more blocks in a block diagram.

[0079] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 The functions specified in one or more blocks in the process or processes and / or block diagrams.

[0080] The above description is a detailed description of the preferred embodiments of the present invention, but the embodiments are not intended to limit the scope of the patent application of the present invention. Any equivalent changes or modifications completed under the technical spirit suggested by the present invention should fall within the patent scope covered by the present invention.

Claims

1. A highway card transceiver life prediction method based on multimodal data fusion and dual-branch LSTM, characterized by: The following steps are involved: S1, collecting multimodal data related to the operating status of the highway card transceiver, the multimodal data including cumulative operating time, vibration signal, temperature signal and visual image signal, and achieving spatiotemporal alignment of the data through timestamp synchronization; S2, preprocessing the collected multimodal data to obtain data features, and dividing the obtained data features into low-frequency trend data and high-frequency transient data according to the frequency characteristics of the data; S3, constructing a two-branch LSTM network having a low-frequency trend branch and a high-frequency transient branch, performing feature extraction on the low-frequency trend data through the low-frequency trend branch to obtain low-frequency trend features; and performing feature extraction on the high-frequency transient data through the high-frequency transient branch to obtain high-frequency transient features; S4, dynamically adjusts the fusion ratio of low-frequency trend features and high-frequency transient features through the weighted gated fusion mechanism; S5, performing feature fusion on the low-frequency trend feature and the high-frequency transient feature according to the obtained fusion ratio to obtain a fusion feature, and finally outputting a probability distribution of the remaining life of the highway card transceiver according to the fusion feature.

2. The highway card receiving and distributing machine life prediction method based on multimodal data fusion and dual-branch LSTM according to claim 1 is characterized in that: In step S1, the vibration signal collection method is: collecting vibration data of the highway card transceiver through a vibration sensor; the temperature signal collection method is: using a temperature sensor with an accuracy of ±0.5°C to collect temperature data of the highway card transceiver; the visual image collection method is: using an industrial camera to collect image data of the highway card transceiver; and the cumulative operating time data of the highway card transceiver is collected through a timing module installed on the highway card transceiver.

3. The highway card receiving and distributing machine life prediction method based on multimodal data fusion and dual-branch LSTM as claimed in claim 1 is characterized in that: In step S2, the method for preprocessing the collected multimodal data is: Processing the accumulated running time to remove possible noise or outliers in the accumulated running time signal; The vibration signal preprocessing method involves removing high-frequency noise above 20 kHz through wavelet packet decomposition and calculating the energy proportion in the 1-100 Hz frequency band to characterize the defect characteristics of the bearing rolling elements used in highway card transceivers. The bearing rolling element defect characteristics are used as the vibration time-frequency domain characteristics. The temperature signal preprocessing method is as follows: the temperature signal is filtered using a sliding window mean filter to obtain the sliding window mean at different times. The temperature rise rate is calculated based on the sliding window mean at different times. The obtained temperature rise rate is compared with a set threshold. If the temperature rise rate is greater than the set threshold, it is determined that an abnormal heating event has been detected. When an abnormal heating event is detected, the dynamic range of the temperature signal is adjusted according to the ambient temperature. The preprocessing method of the visual image signal is to use YOLOv5s to detect the surface cracks of the gear set and calculate the crack area growth rate.

4. The highway card receiving and distributing machine life prediction method based on multimodal data fusion and dual-branch LSTM as claimed in claim 3 is characterized in that: The low-frequency trend data includes the cumulative running time and the sliding window mean; the high-frequency transient data includes the vibration time-frequency domain characteristics and the crack area growth rate.

5. The highway card receiving and distributing machine life prediction method based on multimodal data fusion and dual-branch LSTM as claimed in claim 3 is characterized in that: When using sliding window mean filtering to filter the temperature signal, the window size is set to 60 seconds and the step size is set to 1 second.

6. The highway card receiving and distributing machine life prediction method based on multimodal data fusion and dual-branch LSTM as claimed in claim 1 is characterized in that: In step S3, the low-frequency trend branch and the high-frequency transient branch both include an input layer, a hidden layer, and an output layer. The input layer of the low-frequency trend branch takes low-frequency trend data as input, performs feature extraction on the input low-frequency trend data through the hidden layer, and finally outputs low-frequency trend features through the output layer. The input layer of the high-frequency transient branch takes high-frequency transient data as input, extracts features of the input high-frequency transient data through the hidden layer, and finally outputs the probability of sudden failure as a high-frequency trend feature through the output layer.

7. The highway card receiving and distributing machine life prediction method based on multimodal data fusion and dual-branch LSTM according to claim 6 is characterized in that: The hidden layer of the low-frequency trend branch includes two LSTM layers, each LSTM layer includes 128 LSTM units, and the activation function used in the gating mechanism inside the LSTM unit is tanh.

8. The highway card receiving and distributing machine life prediction method based on multimodal data fusion and dual-branch LSTM as claimed in claim 6 is characterized in that: In step S3, the hidden layer of the high-frequency trend branch is an LSTM network structure that combines a one-dimensional convolutional neural network 1DCNN and an attention mechanism.

9. The highway card receiving and distributing machine life prediction method based on multimodal data fusion and dual-branch LSTM as claimed in claim 1 is characterized in that: The following fusion formula W is used fusion Dynamically adjust the fusion ratio of low-frequency trend features and high-frequency transient features: W fusion =σ(Wg·〔trend_score;transient_score〕) Where the dynamic weight Wg is optimized by gradient backpropagation; trend_score refers to the output of the low-frequency trend branch; transient_score refers to the output of the high-frequency transient branch; σ(*) represents the Sigmoid function.

10. The highway card receiving and distributing machine life prediction method based on multimodal data fusion and dual-branch LSTM according to claim 1 is characterized in that: The vibration sensor uses a three-axis MEMS accelerometer with a sampling rate of 10kHz and a range of ±50g; the temperature sensor uses a K-type thermocouple and uses a MAX31855 chip for cold-end compensation with an accuracy of ±0.5°C; the vibration sensor and the temperature sensor are both protected by an IP68 protection grade casing.

Images