Real-time monitoring system for semi-trailer reversing folding based on multi-sensor fusion

Abstract

The present invention relates to the technical field of semi-trailer safety monitoring technology and discloses a real-time monitoring system for semi-trailer reversing folding based on multi-sensor fusion. A data acquisition module acquires sensor data such as position, angle, and speed, and divides normal and potential folding areas through time synchronization processing. A data fusion module generates a fusion state vector and calibrates the spatial coordinate system. A folding state recognition module determines folding risk based on the density of key monitoring point paths, the distribution of motion characteristics, and the span. A real-time monitoring module expands the data area to generate continuous trajectories and uses curve fitting slope change rate and a convolutional neural network model to assess folding probability in real time. The system integrates multiple types of sensors and employs dynamic filtering, multi-anchor point benchmarks, and intelligent algorithms to achieve accurate real-time monitoring of the risk of semi-trailer reversing folding, thereby improving driving safety.

Description

Technical Field

[0001] The present invention relates to the technical field of semi-trailer safety monitoring, and in particular to a semi-trailer reversing and folding real-time monitoring system based on multi-sensor fusion. Background Art

[0002] Semi-trailers are widely used in logistics and transportation due to their large cargo capacity and high transport efficiency. However, their unique articulated structure presents a significant risk of folding during reversing (i.e., abnormal angle deflection between the tractor and trailer, potentially leading to collisions, rollovers, and other accidents). Traditional monitoring methods rely primarily on single sensors (such as cameras or ultrasonic sensors), which suffer from limited monitoring range, insufficient data accuracy, and poor real-time performance, making it difficult to fully and accurately capture the complex motion of a vehicle during reversing.

[0003] In existing technologies, some monitoring systems fail to fully consider the time synchronization of multi-sensor data. This results in time deviations between data collected by different types of sensors (such as GPS positioning sensors, IMU angle sensors, and wheel speed sensors), preventing effective fusion and thus affecting the accuracy of monitoring results. Furthermore, traditional methods often rely solely on single-dimensional angular changes or simple geometric models when determining vehicle rollover risk, failing to incorporate multi-dimensional information such as the density distribution of the vehicle's motion trajectory and span analysis of motion characteristics. This can easily lead to misjudgments or missed detections. For example, when a vehicle's motion trajectory exhibits a nonlinear distribution or its longitudinal or lateral span exceeds a safety threshold, traditional systems struggle to quickly identify potentially high-risk conditions.

[0004] In terms of real-time performance, existing systems typically use fixed-parameter filtering algorithms or simple curve fitting methods, which are unable to dynamically adapt to the changing characteristics of data at different vehicle speeds. For example, if the window length of the moving average filter cannot be dynamically adjusted according to vehicle speed, it may lead to poor data smoothing, affecting the accuracy of the slope calculation of the state change curve, and thus failing to detect abnormal changes in the folding state in a timely manner. At the same time, the lack of dynamic expansion of the monitoring unit data area and continuous trajectory generation mechanism makes it difficult to effectively connect the data of adjacent monitoring units, making it difficult to form a complete motion state analysis chain.

[0005] Furthermore, existing technologies offer a relatively crude definition of preset safety angle intervals, lacking specific anchor point locations and distribution rules. This lacks a scientific basis for calculating path density and angle changes. For example, by not using the angle peak point in the front region and the angle change point in the rear region as key anchor points, it is difficult to accurately delineate between the normal driving area and the potential folding area. During the coordinate conversion process, using only a single-axis calibration or failing to consider the multi-axis orientation deviation of the boundary contour line can lead to significant errors in the spatial coordinate system calibration of the sensor data, affecting the accuracy of the subsequent fusion state vector generation. Summary of the Invention

[0006] The purpose of the present invention is to provide a real-time monitoring system for the reversing and folding of a semi-trailer based on multi-sensor fusion to solve the problems raised in the above background technology.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a real-time monitoring system for the reversing and folding of a semi-trailer based on multi-sensor fusion, the system comprising:

[0008] A data acquisition module is used to obtain multiple sensor data of the semi-trailer, including position data, angle data and speed data, and perform time synchronization processing on the sensor data to divide the vehicle into a normal driving area and a potential collapse area;

[0009] a data fusion module, configured to generate a fusion state vector based on the sensor data and perform coordinate transformation according to a vehicle geometric model;

[0010] The folding state recognition module is used to select any monitoring unit of the vehicle hinge point, calculate the starting angle value and ending angle value of each monitoring point based on the fusion state vector, mark them as reference points, obtain several reference points closest to the geometric center point of the monitoring unit, define them as key monitoring points, associate adjacent key monitoring points in sequence to generate a path trajectory, obtain the density of the path trajectory covering the preset safety angle range, and when the density is less than a preset threshold, identify whether it is a linear distribution or a curved distribution motion feature. If not, it is determined that the angle change of the monitoring unit is abnormal; if so, the span of the longitudinal or lateral distribution of the motion feature is detected. If the span is greater than the preset span value, it is determined that the unit has a high risk of folding;

[0011] The real-time monitoring module is used to select any one-dimensional fusion state vector sequence, expand the data area of ​​each monitoring unit longitudinally based on the axis where the unit center point is located, until the data areas in adjacent monitoring units form a continuous motion trajectory, generate a continuous state change curve, and calculate the slope change rate of the curve fitting equation. When the deviation between the change rate and the standard value exceeds a preset range, it is judged that the real-time performance of the folding state of this dimension does not meet the standard.

[0012] Preferably, the time synchronization process is as follows:

[0013] The multiple sensor data are timestamp aligned to generate synchronized data, the synchronized data are dynamically range adjusted, and the synchronized data are converted into normalized data by setting a data threshold. The normalized data are smoothed based on a moving average filter, and feature decomposition is performed on the smoothed normalized data to separate the normal driving area of ​​the vehicle from the potential folding area.

[0014] Preferably, the preset safety angle interval is composed of 6 reference anchor points, of which 2 anchor points are set in the front end area of ​​the monitoring unit, and the other 4 anchor points are set in the rear end area of ​​the monitoring unit. The two points in the front end area are positioned at the angle peak points of the axial distribution; the two anchor points in the rear end area are positioned at the intersection of the unit boundary line and the motion path, that is, the angle change point, and the remaining two points in the rear end area are positioned at the secondary uniformly distributed nodes of the line connecting the angle change points.

[0015] Preferably, the process of performing coordinate transformation on the sensor data is:

[0016] The boundary contour line of the normal driving area of ​​the vehicle is obtained, the azimuth deviation angles between the three main axes of the boundary and the sensor data coordinate system are calculated respectively, and the sensor data is calibrated in the spatial coordinate system according to the weighted average of the azimuth deviation angles.

[0017] Preferably, the process of detecting the longitudinal or transverse distribution span of the motion feature is:

[0018] The data area is traversed according to the detection direction, which is a longitudinal section or a transverse section. The angle change value of each group of data detected is counted. The angle value of the potential folding area is 180, and the angle value of the normal driving area is 0. The first detection group whose angle value exceeds the normal driving threshold is selected and marked as the risk starting group. The data area is continued to be traversed. When a detection group whose angle value returns to the normal driving threshold is detected, it is marked as a pending group. The spatial interval D1 between the pending group and the starting group is obtained. If the spatial interval D1 is greater than or equal to the preset span P, and no angle abnormality is found when continuing to detect along the detection direction, the pending group is marked as the termination group, and the motion is characterized by uniform distribution. If the spatial interval is less than D1, the data area is continued to be traversed. When a detection group whose angle exceeds the normal driving threshold is detected again, it is marked as the second starting group. The interval D2 between the second starting group and the pending group is obtained. If the interval D2 is greater than the preset span Q, it is determined that the motion distribution is abnormal.

[0019] Preferably, the specific process of determining whether the folding state real-time performance does not meet the standard is as follows:

[0020] For any selected monitoring unit, select the reference axis where the center point of the unit is located, obtain the offset of all angle points in the data area relative to the axis, adjust the offset of each angle point by a preset scaling factor, and continuously increase the value of the scaling factor until adjacent monitoring units form a data superposition area, obtain a continuous trajectory of state changes, obtain the corresponding fitting curve equation through median filtering of the trajectory, and obtain the slope change rate of the curve equation.

[0021] Preferably, the process of generating the path trajectory is:

[0022] First, the three reference points closest to the center point of the monitoring unit are selected, and the adjacent reference points are associated to generate an initial path structure. Then, the nearest points among the remaining reference points are selected one by one and added to the association of the path structure. When the length of the connection path corresponding to the original reference point in the associated path formed by the newly added reference point increases, the newly added point is removed, and the associated path formed by the original reference point is the path trajectory of the monitoring unit.

[0023] Preferably, the data acquisition module includes a GPS positioning sensor, an IMU angle sensor and a wheel speed sensor, wherein the GPS positioning sensor is arranged in a grid structure at key positions of the vehicle, the IMU angle sensor is distributed axially at equal intervals, and the wheel speed sensor covers all moving joint areas of the vehicle.

[0024] Preferably, the dynamic range adjustment adopts a linear transformation method to compress the range of the original data to a preset range while retaining the data distribution characteristics, and the window length of the moving average filter is dynamically adjusted according to the vehicle movement speed.

[0025] Preferably, the real-time monitoring module has a built-in convolutional neural network model, which is trained through historical vehicle folding data, with the slope change rate of the state change curve as the input feature and the probability of folding as the output result. An adaptive learning rate strategy is used to optimize the loss function during model training.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] During data acquisition and preprocessing, the data acquisition module integrates a GPS positioning sensor, an IMU angle sensor, and a wheel speed sensor to acquire position, angle, and speed data, respectively. Time synchronization (including timestamp alignment, dynamic range adjustment, moving average filtering, and feature decomposition) eliminates temporal deviations between sensors and separates normal driving areas from potential folding areas. Dynamic range adjustment uses linear transformation to compress data ranges while preserving distribution characteristics. The moving average filter window length dynamically adjusts with vehicle speed to ensure optimal data smoothing at different speeds, laying the foundation for subsequent fusion analysis.

[0028] The data fusion module generates a fusion state vector based on multi-source sensor data and performs coordinate transformation using the vehicle geometry model. Specifically, it calculates the azimuth deviation angle between the three principal axes of the normal driving area boundary contour and the sensor coordinate system, and uses a weighted average to calibrate the spatial coordinate system. This effectively reduces calculation errors caused by coordinate deviation and improves data fusion accuracy.

[0029] The folding state recognition module achieves accurate judgment of folding risks through an innovative path trajectory generation and risk assessment mechanism. First, the reference point of the monitoring unit hinge point is selected, and the key monitoring points are determined through distance screening. The path trajectory is generated and its density covering the preset safety angle range is calculated. When the density is lower than the threshold, the linear / curved distribution properties of the motion characteristics are further analyzed. If it is a nonlinear distribution, the angle change is directly determined to be abnormal; if it is a linear distribution, the longitudinal / lateral span is detected, and when the span exceeds the preset value, it is determined to be high risk. This process combines multi-dimensional features such as trajectory density, motion form and spatial span, avoids the limitations of a single indicator, and significantly improves the accuracy and robustness of risk identification.

[0030] The real-time monitoring module generates continuous motion trajectories by dynamically expanding the data region. It uses median filtering and curve fitting to calculate the slope change rate, and combines this with a convolutional neural network (CNN) model to predict the probability of folding. The data region is expanded longitudinally based on the axis of the cell center until adjacent cells form a continuous trajectory, ensuring the integrity of the state change curve. The CNN model is trained on historical folding data and uses an adaptive learning rate to optimize the loss function. It outputs risk probabilities in real time, providing drivers with forward-looking warnings and effectively improving the real-time and intelligent capabilities of the monitoring system.

[0031] The anchor point distribution rules for the preset safe angle range (two angle peaks at the front, two angle change points at the back, and two quadratically evenly distributed nodes) provide a scientific benchmark for trajectory density calculation, ensuring a clearer demarcation between normal driving and collapse risk. The motion feature span detection process (which traverses the data region to determine the risk starting group, pending group, and ending group, and compares the spatial interval with a preset threshold) quantitatively analyzes the distribution range of angle anomalies, further enhancing the credibility of risk assessments.

[0032] The present invention constructs a complete real-time monitoring system for the reversing and folding of semi-trailers through the organic combination of multi-sensor fusion, spatiotemporal data calibration, multi-dimensional risk feature extraction and intelligent prediction models. It effectively solves the problems of low data fusion accuracy, single risk identification, and insufficient real-time performance in the existing technology, and provides strong technical support for improving the driving safety of semi-trailers. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a working principle diagram of the real-time monitoring system for reversing and folding of a semi-trailer based on multi-sensor fusion according to the present invention;

[0034] Figure 2 This is the working principle diagram of sensor data time synchronization processing;

[0035] Figure 3 Flowchart for detecting the longitudinal / lateral distribution span of motion features;

[0036] Figure 4 This is a diagram showing the working principle of real-time judgment of folding status. DETAILED DESCRIPTION

[0037] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0038] See also Figures 1-4 The present invention relates to a real-time monitoring system for the folding of a semi-trailer vehicle when it is reversed based on multi-sensor fusion. The system comprises: a data acquisition module, a data fusion module, a folding state recognition module, and a real-time monitoring module. The specific implementation steps are as follows:

[0039] The data acquisition module obtains multiple sensor data of the semi-trailer, including position data, angle data and speed data, and performs time synchronization processing on the sensor data to divide the vehicle's normal driving area and potential folding area.

[0040] The data fusion module generates a fusion state vector based on the sensor data and performs coordinate transformation according to the vehicle geometric model.

[0041] The folding state recognition module selects any monitoring unit of the vehicle hinge point, calculates the starting angle value and ending angle value of each monitoring point based on the fusion state vector, and marks them as reference points. It obtains several reference points closest to the geometric center point of the monitoring unit, which are defined as key monitoring points. The adjacent key monitoring points are sequentially associated to generate a path trajectory, and the density of the path trajectory covering the preset safety angle range is obtained. When the density is less than the preset threshold, it is identified whether it is a linear distribution or a curved distribution motion feature. If not, it is judged that the angle change of the monitoring unit is abnormal; if so, the span of the longitudinal or lateral distribution of the motion feature is detected. If the span is greater than the preset span value, it is judged that the unit has a high risk of folding.

[0042] The real-time monitoring module selects any one-dimensional fusion state vector sequence, and vertically expands the data area of ​​each monitoring unit based on the axis of the unit center point until the data areas in adjacent monitoring units form a continuous motion trajectory, generates a continuous state change curve, and calculates the slope change rate of the curve fitting equation. When the deviation between the change rate and the standard value exceeds the preset range, it is judged that the real-time performance of the folding state of this dimension does not meet the standard.

[0043] The present invention will be further described below in conjunction with Examples 1 to 5:

[0044] Example 1: This example relates to the specific structure and time synchronization process of the data acquisition module. The data acquisition module includes GPS positioning sensors, IMU angle sensors, and wheel speed sensors. The GPS positioning sensors are arranged in a grid-like structure at key locations on the vehicle. This grid structure is designed based on the structural characteristics of a semi-trailer vehicle, forming regular or irregular grid nodes at key locations such as the front, middle, and rear of the vehicle. Each node is equipped with a GPS positioning sensor, ensuring coverage of all key vehicle areas. This allows for comprehensive collection of vehicle position data and accurate acquisition of the vehicle's coordinate position and movement trajectory in three-dimensional space. The IMU angle sensors are evenly spaced axially and deployed at uniform intervals along the vehicle's longitudinal axis (e.g., the centerline of the frame). For example, an IMU angle sensor is positioned at regular intervals (this distance can be set based on the vehicle's length and monitoring accuracy requirements). This allows for uniform collection of angle data from various vehicle components along the vehicle's axial direction, including pitch, roll, and yaw angles, to reflect changes in the vehicle's posture in real time. The wheel speed sensor covers all moving joint areas of the vehicle, including the wheel axle, the articulated joints of the semi-trailer and other parts. A wheel speed sensor is installed at the axle of each wheel to monitor the wheel speed and rotation direction. The wheel speed sensor at the articulated joint is used to monitor the movement speed of the joint, thereby ensuring the accurate collection of the speed data of the vehicle as a whole and each moving joint.

[0045] In terms of time synchronization processing, multiple sensor data are timestamp aligned to generate synchronized data. Since different types of sensors (such as GPS positioning sensors, IMU angle sensors, and wheel speed sensors) may have different sampling frequencies and time bases, it is necessary to add timestamps to the data output by each sensor, and then align different sensor data at the same moment through timestamp matching, so that sensor data from different sources are consistent in the time dimension, forming a synchronized data set with a unified time base. The synchronized data is then dynamically adjusted and processed using a linear transformation. This linear transformation uses a linear function to compress the range of the original data to the target range by determining the maximum and minimum values ​​of the original data, as well as a preset target interval range (such as [0,1] or other intervals set according to subsequent processing requirements). For example, if the minimum value of the original data is , the maximum value is , the target interval is , then for any original data , and its linearly transformed value is:

[0046]

[0047] This processing method can preserve the original distribution characteristics of the data while compressing the data range, avoiding the impact of large differences in data dimensions on subsequent data processing and analysis.

[0048] Synchronized data is converted into normalized data by setting data thresholds. Normalization is performed to unify sensor data of different types and dimensions into the same dimension, facilitating data fusion and comparative analysis. Specifically, thresholds are set based on the physical meaning and value range of each type of sensor data. Data exceeding the threshold is truncated or adjusted so that all data falls within a standard numerical range (such as [0, 1] or [-1, 1]). For example, for longitude and latitude information in GPS positioning data, thresholds can be set based on the vehicle's geographic range, with data outside this range deemed invalid or corrected. For angle data from an IMU angle sensor, thresholds can be set based on the vehicle's normal driving angle range to convert the angle values ​​into normalized values.

[0049] Normalized data is smoothed using a moving average filter, whose window length is dynamically adjusted based on vehicle speed. Moving average filtering is a commonly used signal smoothing method that reduces noise by calculating the average value of data within a window. When the vehicle is moving quickly, sensor data may be more susceptible to high-frequency noise. In this case, the window length is automatically increased to include more data points in the averaging calculation, thereby enhancing the suppression of high-frequency noise. When the vehicle is moving slowly, data changes are relatively gradual. In this case, the window length is reduced to more quickly respond to real-time data changes and avoid data lag caused by excessively long windows. For example, when the vehicle is reversing at high speed, the window length can be set to include the most recent 10 sampling points; when the vehicle is moving slowly at low speed, the window length can be adjusted to include the most recent 5 sampling points.

[0050] Eigen decomposition is performed on the smoothed, normalized data to separate the vehicle's normal driving area from its potential collapse area. Eigen decomposition uses mathematical methods (such as principal component analysis) to reduce the data's dimensionality and extract components that reflect the data's key characteristics. By analyzing the smoothed, normalized data, characteristic patterns of the vehicle's normal driving state (such as angle changes within a certain range and stable speed fluctuations) and characteristic patterns of its potential collapse state (such as sudden, large angle changes and abnormal speed fluctuations) are identified. Based on these characteristic patterns, the data space is divided into the normal driving area and the potential collapse area. For example, eigendecomposition can determine that when the articulation angle change detected by the IMU angle sensor exceeds a certain threshold and the speed detected by the wheel speed sensor suddenly decreases, the data point is in the potential collapse area. Conversely, when both the angle change and speed change are within normal range, the data point is in the normal driving area. This provides a clear data foundation for the subsequent data fusion and collapse state identification modules, enabling the system to more accurately determine the vehicle's driving state and collapse risk.

[0051] Example 2: This example focuses on the composition of the preset safety angle interval and the coordinate conversion process. The preset safety angle interval is composed of 6 reference anchor points, of which 2 anchor points are set in the front end area of ​​the monitoring unit, and the other 4 anchor points are set in the rear end area of ​​the monitoring unit. The two points in the front end area are positioned at the angle peak points of the axial distribution. The axial distribution refers to the angle change distribution along the central axis of the monitoring unit (such as the axis where the hinge point of the semi-trailer is located). The angle peak point is determined by analyzing the extreme angles that may appear at the front end of the monitoring unit during the reversing process. For example, when the vehicle is turning or reversing, the front end may produce the maximum angle deviation due to inertia or steering operation. These two peak points correspond to the maximum positive and negative angles in the axial distribution, respectively, which can reflect the extreme angle changes of the front end of the monitoring unit in the axial direction, and serve as key reference points for judging angle anomalies in the front end area.

[0052] The two anchor points in the rear-end area are located at the intersection of the unit boundary line and the motion path, which is the angle change point. The unit boundary line is a boundary pre-defined based on the physical structure and motion range of the monitoring unit, such as a virtual boundary line extending to both sides with the hinge point as the center; the motion path is the actual motion trajectory of the monitoring unit tracked in real time through sensor data. When the motion trajectory of the monitoring unit intersects with the unit boundary line, it indicates that the angle of the rear end of the monitoring unit begins to change significantly. The intersection is the starting point or turning point of the angle change, which can capture the key position of the rear-end angle change. The remaining two points in the rear-end area are located at the secondary uniformly distributed nodes of the angle change point line. Specifically, the two angle change points are first connected by a straight line to form a baseline, and then the baseline is evenly divided twice, that is, anchor points are set at the midpoint of the baseline and at the midpoint between the midpoint and the two end points, so that the two newly added anchor points divide the baseline into three equal segments. Through this quadratic uniform distribution setting, the details of the angle change can be captured more finely in the area between the angle change points, avoiding missing the angle anomalies in the middle area due to the sparse distribution of anchor points, so that the preset safety angle interval can fully and accurately cover the angle change range of the rear end of the monitoring unit, providing a more accurate reference basis for the density calculation and risk judgment of the path trajectory in subsequent folding state identification.

[0053] When performing coordinate transformation on sensor data, a boundary contour line of the vehicle's normal driving area is obtained. This boundary contour line is derived from statistical analysis of historical sensor data and represents the positional boundaries of various vehicle components during normal reverse driving. For example, it can be a closed contour formed by the position ranges of various monitoring units on a semi-trailer during normal reverse driving without the risk of jackknifing. This boundary contour line can be generated through data fitting, such as using polygon fitting or curve fitting methods, to integrate the position data of various monitoring points during normal driving to form a boundary curve or broken line that represents the normal driving area.

[0054] Calculate the azimuth deviation angles between the three principal axes of the boundary and the sensor data coordinate system. These three principal axes are selected to represent the vehicle's overall structure and direction of motion. These typically include the vehicle's longitudinal center axis (along the length of the vehicle), the transverse axis (a horizontal axis perpendicular to the length of the vehicle), and the vertical axis (an axis perpendicular to the ground). The sensor data coordinate system refers to the coordinate system used by each sensor's output data. Different sensors may use different coordinate systems (for example, GPS positioning sensors use a geographic coordinate system, while IMU angle sensors use a vehicle coordinate system). The azimuth deviation angle is the angular difference between the actual and theoretical orientation of a principal axis in the sensor data coordinate system. For example, the vehicle's longitudinal center axis should theoretically align with the positive x-axis of the sensor data coordinate system. If the actual detected center axis direction is at an angle θ with the positive x-axis, then θ is the azimuth deviation angle of the longitudinal center axis. By calculating the azimuth deviation angles for the three principal axes, the angular difference between the sensor data coordinate system and the vehicle's actual coordinate system can be fully reflected.

[0055] The sensor data is calibrated to a spatial coordinate system based on the weighted average of the azimuth deviation angles. This weighted average is calculated by assigning different weights to the azimuth deviation angles of the three principal axes. These weights are determined based on the importance of each principal axis in vehicle motion. For example, the longitudinal center axis has the greatest impact on the vehicle's reversing direction and can be assigned a higher weight, while the lateral and vertical axes have relatively lower weights. To calculate the weighted average, the azimuth deviation angles of each principal axis are multiplied by their corresponding weights and summed to obtain a composite azimuth deviation angle. This composite deviation angle is then used to transform the sensor data. For example, a rotation matrix is ​​used to convert the sensor data from its original coordinate system to a coordinate system based on the vehicle's actual principal axes. This ensures that the calibrated sensor data more accurately reflects the vehicle's actual spatial position and attitude. For example, after coordinate calibration, GPS positioning data can fully correspond to the vehicle's position in real geographic space. For IMU angle sensor data, calibration eliminates angular deviations caused by differences in sensor installation position or coordinate system, ensuring that the angular data truly reflects the actual attitude changes of each vehicle component. Through this coordinate transformation process, the data from different sensors are fused and analyzed in a unified coordinate system, which improves the consistency and reliability of the data and lays the foundation for the subsequent generation of fusion state vectors and folding state identification based on the vehicle geometric model.

[0056] Example 3: This example details the process of detecting motion feature spans and determining folding status in real time. When detecting the longitudinal or transverse distribution span of motion features, the data region is traversed based on the detection direction, either longitudinal or transverse. A longitudinal section refers to a vertical cross-section along the length of the vehicle (e.g., from front to rear of a semitrailer), while a transverse section refers to a vertical cross-section along the width of the vehicle (e.g., from left to right). By comprehensively traversing the data region along these two dimensions, the angular variations of all vehicle parts can be captured. The angular variation values ​​for each detected data set are statistically analyzed, with the angle value for potential folding areas being 180° (corresponding to the theoretical angle of a fully folded vehicle) and the angle value for normal driving areas being 0° (corresponding to the theoretical angle of a straight-line, unfolded vehicle). This serves as a baseline threshold for determining whether the angle is abnormal. In actual testing, the threshold can be fine-tuned based on the vehicle structure and driving characteristics.

[0057] The first detection group whose angle value exceeds the normal driving threshold is selected and marked as the risk initiation group. The normal driving threshold can be set to a small range close to 0 (such as ±5°). When the angle value of a detection group exceeds this range, it indicates that the vehicle has begun to deviate from an angle that may cause collapse. For example, in a longitudinal profile test, if the angle value of a detection group suddenly increases from 0° to 10°, this group is marked as the risk initiation group and is considered a potential starting point for collapse risk.

[0058] Continue traversing the data region. When a detection group is detected whose angle value returns to the normal driving threshold, it is marked as a pending group. At this point, the angle value has temporarily returned to the normal range, but further determination is needed to determine whether this recovery is a true termination point, rather than a temporary fluctuation. Obtain the spatial interval D1 between the pending group and the starting group. The calculation of spatial interval D1 is based on the coordinate system of the detection direction. For example, in a longitudinal section, the longitudinal distance between the two groups is calculated based on the longitudinal axis of the vehicle, and in a transverse section, the transverse distance is calculated based on the transverse axis of the vehicle. If spatial interval D1 is greater than or equal to the preset span P (the preset span P is set based on vehicle size and safety standards, such as 2 meters), and no angle anomalies are detected during continued detection along the detection direction, the pending group is marked as the ending group. This indicates that the angle change in this section was completed within a reasonable spatial range and is uniformly distributed. That is, the angle change is continuous and gradually recovers, without forming sudden changes or abnormal clusters.

[0059] If the spatial interval D1 is less than the preset span P, it indicates that the angle change has recovered in a relatively short period of time, potentially indicating a risk of abnormal fluctuations, and further traversal of this data region is required. When another detection group is detected where the angle exceeds the normal driving threshold, it is marked as the second starting group, and the interval D2 between the second starting group and the pending group is obtained. If the interval D2 is greater than the preset span Q (which can be set based on actual conditions and is typically less than the preset span P, such as 1 meter), the motion distribution is considered abnormal, indicating multiple abnormal angle breakthroughs within a short distance, which may indicate an unstable vehicle trajectory and a high risk of collapse.

[0060] When determining whether the folding state meets the real-time performance standard, for any selected monitoring unit, the reference axis at the unit's center point is selected. This axis is usually the geometric center axis of the monitoring unit (such as the longitudinal axis at the hinge point) as the reference line for measuring the unit's angle change. The offset of all angle points within the data area relative to this axis is obtained. The offset is calculated as the absolute value of the angular difference between the angle point and the reference axis. For example, if the angle value of a certain angle point is 30° and the reference axis angle is 0°, the offset is 30°.

[0061] The offset of each angle point is adjusted by a preset zoom factor. The zoom factor is used to magnify or reduce the scale of the offset to facilitate observation of data associations between adjacent monitoring units. The value of the zoom factor is continuously increased until adjacent monitoring units form a data overlap area. The data overlap area refers to the overlap of the distribution ranges of the angle offsets of adjacent monitoring units, indicating that the angle changes of the two are continuous and correlated. For example, when the zoom factor is gradually increased from 1 to 3, the distribution ranges of the angle offsets of the two originally separated monitoring units begin to overlap. At this time, a continuous trajectory of state changes can be obtained, which intuitively shows the consistency and trend of the vehicle's angle changes during reversing.

[0062] The trajectory is filtered using median filtering to obtain the corresponding fitting curve equation. Median filtering is a nonlinear filtering method that effectively removes noise, especially impulse noise, by sorting the data values ​​within a window and taking the median. For example, for trajectory data containing abnormal jump angle values, median filtering can replace them with the median of the adjacent data, making the trajectory smoother. The fitting curve equation can be obtained using methods such as polynomial fitting and linear fitting. The appropriate fitting method is selected based on the shape of the trajectory. For example, linear fitting is used when the trajectory exhibits a near-linear trend, while quadratic or higher-order polynomial fitting is used when the trajectory exhibits significant curvature.

[0063] Get the slope change rate of the curve equation. The slope change rate reflects the trend of the angle change rate. For example, if the fitting curve is a linear function , whose slope is a constant, and the slope change rate is 0, indicating that the angle change rate is constant; if the fitting curve is a quadratic function , whose slope is The slope change rate is , indicating that the rate of angle change varies linearly over time or space. When the slope rate of change deviates from the standard value by more than a preset range, the real-time performance of the folding state in that dimension is considered substandard. This indicates that the real-time monitoring data of the vehicle's angle changes does not accurately reflect the actual state, and may be subject to data lag or abnormal fluctuations, necessitating timely warnings or adjustments to monitoring parameters. Through this process, the system accurately extracts the dynamic characteristics of angle changes from real-time monitoring data, providing a scientific basis for real-time assessment of folding risk.

[0064] Example 4: This example describes further details of the path trajectory generation and data acquisition module. When generating the path trajectory, first select the three reference points closest to the center point of the monitoring unit. The center point of the monitoring unit is the center position determined according to the geometric shape of the monitoring unit (such as the intersection of the diagonals of a rectangular monitoring unit), and the reference point is the point marked with the starting angle value and the ending angle value of each monitoring point calculated based on the fusion state vector. By calculating the Euclidean distance (i.e., the spatial straight-line distance) from each reference point to the center point, the three reference points with the smallest distance are screened out. These three points serve as the initial key monitoring points, which can most directly reflect the angle change characteristics of the central area of ​​the monitoring unit and provide a basic reference for the path trajectory.

[0065] Adjacent reference points are associated to generate an initial path structure. Adjacent reference points are determined based on spatial proximity. That is, after sorting by distance, the two closest of the three reference points are connected first to form a line segment of the initial path. The third point is then connected to the closer of the two connected points to form an initial triangle or polyline structure containing three line segments, which serves as the basic framework of the path trajectory.

[0066] Then, the closest remaining reference points are selected one by one and added to the path structure. For each reference point not yet included in the path structure, its distance to all endpoints in the current path structure is calculated. The endpoint with the smallest distance is selected for connection and incorporated into the path structure. For example, if the current path structure has endpoints A and B, and the remaining reference point C is closer to A than to B, then C is connected to A, forming a new path segment AC. The path structure is now expanded to a polyline containing A, B, and C.

[0067] When adding new reference points, the system continuously checks whether the length of the connection paths corresponding to the existing reference points in the associated paths formed by the newly added reference points has increased. Specifically, when a new reference point D is added to the path structure and connected to endpoint C, the system checks whether the length of the path from endpoint B to C in the original path, which is connected to endpoint C, has changed due to the addition of D (for example, whether the length of the BC segment has changed due to recalculation of the path order). If the length of the connection path corresponding to the existing reference point has increased, it indicates that the addition of the new reference point has caused the path structure to deviate from the optimal connection method, possibly introducing redundant or unreasonable turns. In this case, the newly added point will be removed, maintaining the original path structure. If the path length has not increased or shortened, the newly added point will be retained and the path will continue to be expanded.

[0068] After this screening and adjustment process, the resulting connected path formed by the original reference points becomes the monitoring unit's path trajectory. By gradually optimizing the connection sequence and eliminating invalid points, this trajectory accurately reflects the spatial correlation and angular variation trends between reference points within the monitoring unit, presenting a smooth curve or regular broken line. This provides precise path data for subsequent calculations of the density of the path trajectory covering the preset safe angle range.

[0069] In the data acquisition module, GPS positioning sensors are deployed in a grid-like structure at key locations on the vehicle. This grid design is based on the layout of a semi-trailer vehicle. Grid nodes are formed at key locations, such as the roof of the cab at the front, the middle of the chassis, and the four corners of the rear cargo box. Each node is equipped with a GPS positioning sensor. For example, a 5×5 grid is formed at 20 cm intervals in the front area, a 10×3 grid is formed at 50 cm intervals in the middle area, and an 8×4 grid is formed at 30 cm intervals in the rear area. This ensures that position data from all vehicle parts is effectively collected. Each sensor transmits data to the central processing unit in real time via wireless communication modules, forming a vehicle-wide location monitoring network that accurately captures the coordinates and displacement changes of each vehicle part in geographic space.

[0070] IMU angle sensors are evenly spaced axially along the vehicle's longitudinal centerline (i.e., the centerline of the frame), starting from the front hinge point and ending at 30-centimeter intervals toward the rear. For the tractor and trailer parts of a semi-trailer, sensors are evenly distributed along the centerlines of the tractor and trailer frames, respectively—for example, five sensors are deployed on the tractor and eight on the trailer. Each IMU angle sensor collects real-time pitch, roll, and yaw angle data at its location. By working in conjunction with a three-axis gyroscope and a three-axis accelerometer, it accurately measures changes in the vehicle's posture, particularly angular fluctuations near the hinge points, providing critical angular data for identifying folding states.

[0071] Wheel speed sensors cover all moving joints of the vehicle, including the front and rear axles of the tractor, the axles of the trailer, and the articulated joints between the tractor and trailer. A wheel speed sensor is installed at each wheel axle, monitoring the wheel speed and number of revolutions through electromagnetic induction or photoelectric encoding to calculate the vehicle's travel speed and distance. Wheel speed sensors at articulated joints utilize a rotary encoder structure, mounted on the articulated shaft, to monitor the joint's rotational speed and angle changes in real time, ensuring comprehensive acquisition of the vehicle's movement speed. In particular, any speed anomalies at the joints (such as sudden acceleration or deceleration) can be captured promptly, providing speed-related data support for determining the vehicle's motion status and the risk of collapse.

[0072] Through the collaborative work of GPS positioning sensors, IMU angle sensors, and wheel speed sensors, the data acquisition module can synchronously acquire the vehicle's position, angle, and speed data, forming a multi-dimensional real-time monitoring data set. For example, when a semi-trailer is reversing, the GPS positioning sensor tracks the movement of various vehicle body parts in real time, the IMU angle sensor monitors the angle changes at the hinge points, and the wheel speed sensor records the movement speed of the wheels and joints. After time synchronization, this data provides comprehensive and accurate raw data for the data fusion module to generate the fusion state vector, ensuring that the entire monitoring system can perform folding state identification and real-time monitoring based on a reliable data foundation.

[0073] Example 5: This embodiment involves the specific contents of dynamic range adjustment, filtering processing and real-time monitoring modules. During the dynamic range adjustment process, the synchronous data is processed by linear transformation. This processing is based on the numerical distribution characteristics of the raw data of the sensor. First, the maximum and minimum values ​​of the raw data are determined to form the original range of the data. For example, the raw angle data range output by a certain IMU angle sensor is [-30°, 50°], with a maximum value of 50°, a minimum value of -30°, and an original range of 80°. Subsequently, the target interval is set according to the needs of subsequent data processing, such as setting the target interval to [0,1], and mapping the raw data to this interval through a linear transformation formula. The specific transformation formula is: Substituting the above example data, an original value of -30° corresponds to a normalized value of 0, 50° corresponds to a normalized value of 1, and an intermediate value, such as 0°, corresponds to a normalized value of 0.375. This linear transformation method preserves the relative size and distribution of the original data, compressing only the data's numerical range. This avoids deviations during fusion caused by large differences in the dimensionality of data from different sensors.

[0074] The moving average filter's window length is dynamically adjusted based on vehicle speed. Vehicle speed is calculated using real-time wheel speed data collected by the wheel speed sensor. For example, if the wheel speed sensor detects a wheel speed of 100 rpm, the actual vehicle speed can be calculated as 5 m / s based on the wheel diameter. When the vehicle is moving at high speeds (e.g., speeds greater than or equal to 3 m / s), sensor data is susceptible to high-frequency noise. In this case, the moving average filter's window length is automatically increased, for example, to include the most recent 15 sampling points. This noise is smoothed by averaging more data points. When the vehicle is moving at low speeds (e.g., speeds less than 3 m / s), data changes are relatively slow and the impact of noise is less. In this case, the window length is reduced to include the most recent 5 sampling points to improve real-time data processing and ensure that subtle changes in angle and speed are promptly reflected. The specific calculation method of the moving average filter is: for the current sampling point, the arithmetic mean of the data values ​​of several points before and after it (the number of points is determined by the window length) is taken as the filtered value for that point. For example, when the window length is 5, the filtered value of the nth point is the average of the original data of the n-2th, n-1th, nth, n+1th, and n+2th points.

[0075] The real-time monitoring module has a built-in convolutional neural network (CNN) model, which is trained using historical vehicle folding data. The historical data contains multi-sensor data of the vehicle in different reversing scenarios, as well as the corresponding annotation results of whether folding occurs. During the training process, the slope change rate of the state change curve is used as the input feature. The state change curve is generated by the real-time monitoring module, specifically by selecting any one-dimensional fusion state vector sequence (such as a vector sequence in the angle dimension), and longitudinally expanding the data area of ​​each monitoring unit based on the axis where the center point of the unit is located, until the data areas of adjacent monitoring units form a continuous motion trajectory, thereby generating a state change curve of this dimension. The slope change rate is obtained by calculating the derivative of the curve fitting equation. For example, for the quadratic curve fitting equation , whose slope is The slope change rate is , reflecting the changing trend of the angle change rate.

[0076] The model uses the probability of folding as its output, with the output value range being [0,1]. A larger value indicates a higher probability of folding. During model training, an adaptive learning rate strategy is used to optimize the loss function. Common adaptive learning rate algorithms, such as the Adam algorithm, have the core idea of ​​automatically adjusting the learning rate based on the gradient changes of the parameters during training. In the early stages of training, the gradient is large, and the learning rate is set high to speed up convergence. As training progresses, the gradient gradually decreases, and the learning rate automatically decays to avoid oscillations and improve training accuracy. The loss function uses the cross-entropy loss function, and the calculation formula is:

[0077] ,

[0078] in is the true label (0 means unfolded, 1 means folded), is the fold probability predicted by the model.

[0079] In terms of model architecture design, convolutional neural networks typically consist of several convolutional layers, pooling layers, and fully connected layers. Convolutional layers use convolution kernels to extract spatially local correlations in input features, such as extracting local trend features from the slope rate of change sequence of a state change curve. Pooling layers downsample feature maps to reduce the number of parameters and improve model robustness. Fully connected layers integrate the extracted features and output the final probability of fold occurrence. For example, a typical CNN architecture may include three convolutional layers (each with a convolution kernel size of 3×1 and a stride of 1), two max pooling layers (with a pooling window size of 2×1), and two fully connected layers (with 128 neurons and 1 neuron, respectively).

[0080] The real-time monitoring module's workflow is as follows: First, it obtains the fused state vector sequence from the data fusion module and selects data from one dimension (such as angle or velocity). Next, the data area of ​​each monitoring unit is expanded longitudinally along the axis of the unit's center point. By adjusting the expansion amplitude (e.g., expanding the angle range 5° on either side of the axis), the data areas of adjacent monitoring units overlap to form a continuous motion trajectory. Next, a curve fit (e.g., cubic spline fitting) is performed on the trajectory, and the slope change rate of the fitted curve is calculated. Finally, this slope change rate is input into the convolutional neural network model, which outputs the probability of collapse in the current state. If the probability value exceeds a preset threshold (e.g., 0.8), a warning mechanism is triggered, alerting the driver to the risk of collapse.

[0081] Preprocessing through dynamic range adjustment and moving average filtering ensures uniform dimension and low noise levels in the input model data. The convolutional neural network model, through automatic feature extraction and adaptive learning, captures implicit patterns related to folding from complex multi-sensor data. This provides real-time, accurate predictions of folding risks during semi-trailer reversing, assisting drivers in timely maneuvering and reducing accident rates. This entire process does not rely on manually set rules, but rather achieves intelligent monitoring through a data-driven approach, improving the system's generalization and adaptability.

[0082] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0083] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A real-time monitoring system for the reversing and folding of semi-trailers based on multi-sensor fusion, characterized in that: include: A data acquisition module is used to obtain multiple sensor data of the semi-trailer, including position data, angle data and speed data, and perform time synchronization processing on the sensor data to divide the vehicle into a normal driving area and a potential collapse area; a data fusion module, configured to generate a fusion state vector based on the sensor data and perform coordinate transformation according to a vehicle geometric model; The folding state recognition module is used to select any monitoring unit of the vehicle hinge point, calculate the starting angle value and ending angle value of each monitoring point based on the fusion state vector, mark them as reference points, obtain several reference points closest to the geometric center point of the monitoring unit, define them as key monitoring points, associate adjacent key monitoring points in sequence to generate a path trajectory, obtain the density of the path trajectory covering the preset safety angle range, and when the density is less than a preset threshold, identify whether it is a linear distribution or a curved distribution motion feature. If not, it is determined that the angle change of the monitoring unit is abnormal; if so, the span of the longitudinal or lateral distribution of the motion feature is detected. If the span is greater than the preset span value, it is determined that the monitoring unit has a high risk of folding; The real-time monitoring module is used to select a fusion state vector sequence of any dimension, expand the data area of ​​each monitoring unit longitudinally based on the axis where the center point of the monitoring unit is located, until the data areas in adjacent monitoring units form a continuous motion trajectory, generate a continuous state change curve, and calculate the slope change rate of the curve fitting equation. When the deviation between the change rate and the standard value exceeds a preset range, it is judged that the real-time performance of the folding state of this dimension does not meet the standard.

2. The real-time monitoring system for semi-trailer reversing and folding based on multi-sensor fusion according to claim 1 is characterized in that: The process of time synchronization is as follows: The multiple sensor data are timestamp aligned to generate synchronized data, the synchronized data are dynamically range adjusted, and the synchronized data are converted into normalized data by setting a data threshold. The normalized data are smoothed based on a moving average filter, and feature decomposition is performed on the smoothed normalized data to separate the normal driving area of ​​the vehicle from the potential folding area.

3. The real-time monitoring system for reverse folding of semi-trailer vehicles based on multi-sensor fusion according to claim 1 is characterized in that: The preset safety angle interval is composed of 6 reference anchor points, of which 2 anchor points are set in the front end area of ​​the monitoring unit, and the other 4 anchor points are set in the rear end area of ​​the monitoring unit. The two points in the front end area are positioned at the angle peak points of the axial distribution; the two anchor points in the rear end area are positioned at the intersection of the monitoring unit boundary line and the motion path, that is, the angle change point, and the remaining two points in the rear end area are positioned at the secondary uniformly distributed nodes of the line connecting the angle change points.

4. The real-time monitoring system for semi-trailer reversing and folding based on multi-sensor fusion according to claim 1 is characterized in that: The process of coordinate transformation of sensor data is: The boundary contour line of the normal driving area of ​​the vehicle is obtained, the azimuth deviation angles between the three main axes of the boundary and the sensor data coordinate system are calculated respectively, and the sensor data is calibrated in the spatial coordinate system according to the weighted average of the azimuth deviation angles.

5. The real-time monitoring system for semi-trailer reversing and folding based on multi-sensor fusion according to claim 1 is characterized in that: The process of detecting the vertical or horizontal distribution span of motion features is: The data area is traversed according to the detection direction, which is a longitudinal section or a transverse section. The angle change value of each group of data detected is counted. The angle value of the potential folding area is 180, and the angle value of the normal driving area is 0. The first detection group whose angle value exceeds the normal driving threshold is selected and marked as the risk starting group. The data area is continued to be traversed. When a detection group whose angle value returns to the normal driving threshold is detected, it is marked as a pending group. The spatial interval D1 between the pending group and the starting group is obtained. If the spatial interval D1 is greater than or equal to the preset span P, and no angle abnormality is found when continuing to detect along the detection direction, the pending group is marked as the termination group, and the motion feature is a uniformly distributed feature; if the spatial interval is less than D1, the data area is continued to be traversed. When a detection group whose angle exceeds the normal driving threshold is detected again, it is marked as the second starting group. The interval D2 between the second starting group and the pending group is obtained. If the interval D2 is greater than the preset span Q, it is determined that the motion feature distribution is abnormal.

6. The real-time monitoring system for semi-trailer reversing and folding based on multi-sensor fusion according to claim 5 is characterized in that: The specific process of judging whether the folding state is not up to standard in real time is as follows: For any selected monitoring unit, select the reference axis where the center point of the monitoring unit is located, obtain the offset of all angle points in the data area relative to the axis, adjust the offset of each angle point by a preset scaling factor, and continuously increase the value of the scaling factor until adjacent monitoring units form a data superposition area, obtain a continuous trajectory of state changes, obtain the corresponding fitting curve equation for the trajectory through median filtering, and obtain the slope change rate of the curve equation.

7. The real-time monitoring system for semi-trailer reversing and folding based on multi-sensor fusion according to claim 1 is characterized in that: The process of generating path trajectory is: First, the three reference points closest to the center point of the monitoring unit are selected, and the adjacent reference points are associated to generate an initial path structure. Then, the nearest points among the remaining reference points are selected one by one and added to the association of the path structure. When the length of the connection path corresponding to the original reference point increases in the associated path formed by the newly added reference point, the newly added reference point is removed, and the associated path formed by the original reference point is the path trajectory of the monitoring unit.

8. The real-time monitoring system for semi-trailer reversing and folding based on multi-sensor fusion according to claim 1 is characterized in that: The data acquisition module includes a GPS positioning sensor, an IMU angle sensor and a wheel speed sensor, wherein the GPS positioning sensor is arranged in a grid structure at key positions of the vehicle, the IMU angle sensor is distributed axially at equal intervals, and the wheel speed sensor covers all moving joint areas of the vehicle.

9. The real-time monitoring system for semi-trailer reversing and folding based on multi-sensor fusion according to claim 2 is characterized in that: The dynamic range adjustment adopts a linear transformation method to compress the range of the original data to a preset range while retaining the data distribution characteristics. The window length of the moving average filter is dynamically adjusted according to the vehicle movement speed.

10. The real-time monitoring system for semi-trailer reversing and folding based on multi-sensor fusion according to claim 1 is characterized in that: The real-time monitoring module has a built-in convolutional neural network model, which is trained using historical vehicle folding data, using the slope change rate of the state change curve as an input feature and the probability of folding as an output result. During model training, an adaptive learning rate strategy is used to optimize the loss function.

Images