System for detecting motion state of vehicle behind mobile maintenance vehicle based on Leiyu fusion

Abstract

A system for detecting the motion state of a vehicle behind a mobile maintenance vehicle based on thunder-vision fusion belongs to the technical field of traffic intelligent control and comprises a multi-source sensing module, a target detection module, a trajectory tracking module and a vehicle motion state modeling module. A structured state output module performs standardized integration on vehicle target trajectory information provided by a detection and tracking module based on a unified data expression model to form a multi-dimensional structured data format including fields such as a relative distance, a relative speed, an acceleration, a vehicle type, a lane number, a frame timestamp and the like; consistency analysis and downstream calling can be conveniently carried out among different systems, the problem that a traditional detection system is difficult to integrate due to lack of a structured interface is avoided, the system supports flexible butt joint with a road sensing platform, a vehicle-mounted control system and an upper-layer data analysis module, good expansibility and cross-platform adaptation capacity are achieved, and the system is suitable for being popularized and applied. And the engineering availability and deployment efficiency of the traffic flow state information are obviously improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent traffic control, and in particular relates to a moving maintenance vehicle rear vehicle motion state detection system based on radar and vision fusion. BACKGROUND

[0002] Currently, the traffic organization mode of "traffic on one side and operation on the other side" is usually adopted for highway maintenance construction in China. Non-all-weather maintenance work represented by greening pruning, watering and irrigation and road sweeping is usually carried out on a mobile maintenance operation vehicle as an operation platform, and is widely deployed in high-speed road traffic for short-time road occupation operation. However, the mobile maintenance operation vehicle lacks targeted active safety protection means, and there is no hard separation between the vehicle and the main traffic flow and a large speed difference, which leads to frequent safety risk events such as traffic congestion, sight distance obstruction and rear-end collision, and challenges the life and property safety of the operation personnel.

[0003] The traditional operation vehicle mainly relies on LED screen prompts and static reflective signs as warning mechanisms, and cannot realize the perception and judgment of the rear traffic flow, and cannot dynamically quantify the motion state of each vehicle in the traffic flow. Therefore, a multi-modal system with multi-target recognition ability, trajectory tracking ability and real-time traffic state modeling ability needs to be deployed on the operation vehicle to realize real-time and accurate perception of the rear traffic flow and structured semantic information output, and to improve the effectiveness and pertinence of traffic risk perception. In recent years, with the rapid development of multi-modal sensor technology, perception modules such as millimeter wave radar, binocular camera system, inertial navigation unit (IMU) and Beidou satellite navigation system (BDS) have gradually matured, and have the ability to cooperatively perceive targets from multiple dimensions such as image, distance, attitude and spatial coordinates. At the same time, the vigorous development of stereo ranging and computer vision technology, and the widely verified inference stability and target recognition accuracy of the new generation of deep neural network detection models represented by YOLO (You Look Only Once) in dynamic environment; at the same time, the multi-target association mechanism such as ByteTrack tracking algorithm also has industrial-level low-latency capability. The existing achievements in the above core technical fields provide a feasible technical basis for the realization of the application. SUMMARY

[0004] The application provides a moving maintenance vehicle rear vehicle motion state detection system based on radar and vision fusion to solve the defects in the prior art.

[0005] The application is implemented by the following technical solutions: The moving maintenance vehicle rear vehicle motion state detection system based on radar and vision fusion comprises a multi-source perception module, a target detection module, a trajectory tracking module and a vehicle motion state modeling module. The multi-source perception module is responsible for collecting information on vehicles and road alignment from behind. It uses a unified clock synchronization mechanism for time alignment and combines external parameter calibration to achieve spatial consistency, ensuring the fusion processing of multi-source data in the same coordinate system. The target detection module completes image distortion correction and feature enhancement through image preprocessing, extracts vehicle category, bounding box and confidence information in the image based on deep learning algorithm, and eliminates non-target area information by combining ROI region constraints and redundancy suppression logic, outputs effective vehicle detection results and uses them for subsequent tracking processing. The trajectory tracking module is based on the "detection-tracking" paradigm to perform cross-frame data association, and uses a trajectory matching algorithm to identify and number vehicle targets between frames, thereby realizing the continuous construction of multi-target trajectories and outputting target trajectory data with time labels and unique IDs, thus realizing target-level temporal tracking and trajectory structure construction. The vehicle motion state modeling module is based on binocular vision and millimeter-wave radar fusion to achieve high-precision calculation of the target's three-dimensional spatial position, velocity, and acceleration. It also uses the attitude angle provided by the IMU to perform coordinate compensation and stabilization processing on the target position. The state parameters of each target vehicle are organized into a standard format structure vector, including ID, timestamp, three-dimensional coordinates, velocity, and acceleration. It supports real-time push to the edge database and builds a vehicle flow historical trajectory database to support system integration and long-term data analysis tasks.

[0006] As described above, the mobile maintenance vehicle rear vehicle motion status detection system based on radar-visual fusion includes an integrated multi-source sensing module installed on the mobile maintenance vehicle, including a binocular camera, millimeter-wave radar, BDS positioning unit, IMU unit, 4G data antenna, and edge computing processor. The multi-source sensing devices are subject to unified power supply management and communication line layout to ensure stable data flow between various hardware components and the edge computing platform, and data storage or Ethernet interfaces are reserved for data storage, external transmission, and remote debugging functions.

[0007] As described above, the mobile maintenance vehicle rear vehicle motion status detection system based on radar-visual fusion uses a multi-source perception module to uniformly configure the sampling frequencies of the binocular camera, millimeter-wave radar, BDS positioning unit, and IMU unit through a custom clock synchronization, and calibrates the system's main clock to achieve time alignment of multi-modal sensor data. Suppose the multi-source sensing module has N sensors, and the sampling time vector is: T Unified reference clock The time synchronization error vector is then: in, It is a vector of dimension N consisting entirely of 1s; The error constraint is: That is, the maximum mean square difference between the sampling time of all sensors and the system master clock must be less than a set threshold. The sampling frequency is uniformly constrained as follows: T in, For uniformly configured synchronous sampling frequency; During long-term operation, the clocks of various sensors will drift; the drift rate is defined. = in, For the random jitter term, the system corrects it by minimizing the squared drift error: Under PTP or high-frequency pulse synchronization mechanisms, through multi-mode clock alignment optimization, the final alignment target is: in, This is a weighted compensation matrix used for delay correction and jitter smoothing for different sensors.

[0008] As described above, the mobile maintenance vehicle rear vehicle motion state detection system based on radar-visual fusion calibrates the intrinsic and extrinsic parameters of the binocular camera by capturing multiple checkerboard images from different angles and distances, obtains the corner feature positions, and solves the camera's intrinsic and extrinsic parameter matrices according to a system of linear equations to construct the geometric mapping relationship between the image coordinate system, pixel coordinate system, camera coordinate system and world coordinate system. The operation of the mathematical model with intrinsic and extrinsic parameters is as follows: After the internal and external parameters are calibrated, the reprojection error is calculated as an evaluation index of calibration accuracy. The reprojection error refers to the difference between the actual detected pixel and the estimated projection value after the three-dimensional world point is projected back to the image plane through the camera model. Its mean and variance reflect the geometric consistency of the calibration model in the actual projection scene. Let the world coordinate point be: T The camera projection model can then be written as: in, This is the intrinsic parameter matrix. ; This is the extrinsic parameter matrix of the camera, containing information such as rotation and translation. ∈ , Scale factor; Let the detected pixel coordinates be: T The predicted projection is: T The reprojection error of a single chessboard grid corner point can be expressed as: Introducing the Frobenius norm of the matrix, we further extend it to N feature points: in, ∈ This represents the observation matrix of all pixels; , represents the homogeneous coordinate matrix of all world points; The reprojection error is strictly controlled within 0.2 pixels to ensure the spatial accuracy of stereo matching and 3D modeling processes; To achieve spatiotemporal alignment of cross-modal sensing data, the method of manually calibrating objects and collecting images and radar echoes is adopted to construct a set of spatially corresponding point pairs observed by both. The transformation matrix between the radar and camera coordinate systems is finally obtained by minimizing the point cloud projection error. The installation pose of the IMU module is registered with the camera coordinate system by combining manual measurement and static attitude estimation, forming a unified vehicle coordinate framework. After power-on self-test, the system sequentially completes lens imaging test, radar beam response verification, inertial navigation initialization and positioning stability assessment, and dynamically configures parameters based on environmental parameters, providing a robust and stable sensing foundation for subsequent data acquisition processes.

[0009] As described above, the target detection module of the mobile maintenance vehicle rear vehicle motion state detection system based on radar-visual fusion is responsible for real-time semantic recognition processing of image data, including vehicle target bounding box detection, type classification and key position information extraction, providing an input basis for subsequent trajectory association and spatial traffic flow motion state modeling; The target detection module extracts image frames from the synchronously acquired binocular image stream and performs image preprocessing, including fisheye distortion correction, color normalization, motion blur suppression, and image enhancement, to enhance the edge sharpness and image contrast of the detected target.

[0010] As described above, the mobile maintenance vehicle rear vehicle motion state detection system based on radar-visual fusion includes a target detection module. To improve the robustness and scene adaptability of the target detection model in real-world operating environments, this system constructs a subset of operating scene images covering typical adverse conditions such as rain, fog, low light, backlight, and image blur, based on the standard COCO dataset. This forms an enhanced mobile operation detection dataset. The original dataset distribution is assumed to be... The enhanced distribution is Then we have: in, This represents an extended image sample generated through enhancement strategies such as brightness perturbation, contrast adjustment, rain and fog simulation, noise injection, and geometric transformation. During dataset construction, image enhancement techniques combined with field-collected samples are used to expand the training data distribution of the model in scenarios such as roadside work areas, nighttime operations, and low-contrast rainy days. This enhances its generalization ability in complex scenes. After introducing enhancement, the goal is to minimize the expected generalization model error, as shown in the formula: The above formula minimizes the expected generalization model error, where E is the expected value. For model generalization parameters, Indicates from augmented dataset Input samples obtained by random sampling and their corresponding tags , The parameter is The detection model for the input image The predicted output results; The loss function is used to measure the difference between the model's prediction and the true label. It consists of classification loss, bounding box regression loss, and target confidence loss. Image data streams are input into the detection model after transfer learning. The model extracts target semantic features through a backbone feature extraction module and a multi-scale fusion structure, and combines this with the detection head to generate multi-scale anchor boxes and target probability distributions. This represents the input image, with a size of [size missing]. , For Kronecker product, The formula for the RGB channel extension matrix in homogeneous pixel coordinates is as follows. , This represents the set of convolutional kernels in the backbone network, responsible for feature extraction. F is the extracted feature map, used in the downstream detection module. Indicates channel splicing iteration; The model uses a multi-scale feature block fusion mechanism to improve the ability to identify small targets at a distance; Block matrix: After feature fusion in blocks in, The bias matrix is ​​the backbone network, which stores the weights and bias information of the convolutional kernels. The value at each position represents the importance of that pixel in the detection. After non-maximum suppression processing, redundant bounding boxes are removed from the detection results, and the final detection list is output and passed to the downstream target tracking unit. The image frame number, timestamp, bounding box coordinates, and target category are encapsulated together to construct the initial state of the target in the feature map. Attention weights are applied to enhance the ability to focus on target region features and improve recognition stability under cluttered backgrounds and partial occlusion conditions. A 3×3 spatial attention weight matrix is ​​defined as follows: Simultaneously define the channel attention weight matrix: Construct the joint attention block matrix: Enhanced feature map By assigning higher weights to vehicle regions and lower weights to background regions, interference is suppressed and feature writing is enhanced. in, To obtain the diagonal enhancement operator, used to ensure local feature weights, To adjust the parameters, It is a non-linear activation function; The detection output is also used to establish lane region filtering logic. By fusing image segmentation results or ROI information obtained from perspective transformation, only valid targets within the lane are retained, and the class probability and bounding box parameters are output, thereby reducing the false detection rate and subsequent processing pressure, and improving the overall system efficiency. The combined linear transformation and nonlinear mapping operations are as follows: in, , The weights and biases for the classification branches; , representing the probability vector of the target belonging to each category; For activation functions; , The weights and biases of the bounding box regression branch; , representing the predicted bounding box parameters, and , representing the center point coordinates ( ),width ;high ; This represents the activation function for a block diagonal mapping.

[0011] As described above, in the mobile maintenance vehicle rear vehicle motion state detection system based on radar-visual fusion, the target detection module ensures numerical stability in real-time engineering scenarios, and the output adopts a normalized form: Final detection box confidence for: in, This indicates the probability that the bounding box contains the target. This represents the maximum classification probability of the most likely category of the box; To assess the overall confidence level and measure the reliability of the test results, this information is used for subsequent screening. In the candidate box set = , In the definition of intersection matrix Define area vector For two candidate boxes, this reflects the degree of overlap between them. matrix ,satisfy: Constructing the adjacency matrix : in, The threshold parameter is set. The nonmaximum suppression problem can be formalized as the maximum weighted independent set problem: in, , represents the confidence vector, where each element is the detection score of the corresponding candidate box; Indicates the selection vector. Indicates that candidate boxes are retained. ; Represented by the adjacency matrix The defined set of conflicting edges; For ease of solution, it can be relaxed into a convex optimization problem: in, This is the penalty coefficient, used to constrain the joint selection of overlapping candidate boxes, balancing confidence and overlap penalty; Introducing the Graph Laplacian Operator Then the optimization objective is: The boxes with lower confidence are then removed, resulting in a high-confidence target detection result.

[0012] As described above, in the mobile maintenance vehicle rear vehicle motion state detection system based on radar-visual fusion, the trajectory tracking module is responsible for associating the detected vehicle targets with data in consecutive frame images, constructing the cross-frame trajectory of each vehicle, maintaining the consistency of target ID, and forming a stable temporal tracking structure. A "two-stage matching + confidence level" association strategy is constructed. In the first stage, high-confidence detection boxes are used to perform main matching with historical trajectories to ensure the accuracy of matching to the greatest extent. A "motion prediction-assisted association" mechanism is introduced during the tracking process. A short-time velocity vector is constructed using the positions of the first two frames to predict the next possible location of the target, improving the matching robustness in occluded or weakly detected scenarios. The target's state vector at time t is defined as: in, It is a position vector; It is the velocity vector; This is the acceleration vector.

[0013] As described above, in the mobile maintenance vehicle rear vehicle motion state detection system based on radar-visual fusion, the trajectory tracking module constructs a state transition model: in, , It is a 2-order identity matrix. The process noise covariance matrix is... This represents the predicted current position, which is used as a candidate point to compensate for matching issues caused by occlusion or weak detection. , This indicates the target's position in the first two frames (center coordinates). The calculation formula is: To address the special working conditions of large changes in the field of view and drastic background dynamics of vehicle-mounted cameras, prior constraints on the target motion direction and acceleration are introduced to filter out target trajectories with obvious jumps or abnormal motion characteristics. This reduces ID drift and trajectory fragmentation problems caused by occlusion or detection interruptions. The prior constraint formula for motion parameters is as follows: in, This represents the velocity vector of the target at time t. The acceleration vector of the target at time t; Indicates the inter-frame time interval; The association constraint is optimized as follows: in, Indicates the maximum speed threshold; This represents the maximum acceleration threshold. If a solution violates the constraint, it is judged as an abnormal motion trajectory and is discarded. A spatial consistency verification mechanism is employed to compare the target's motion direction in the image coordinate system and the vehicle coordinate system, ensuring that the tracking results conform to the actual traffic flow direction and further improving the system's robustness. The formula for comparing the motion direction in the image coordinate system and the vehicle coordinate system is as follows: in, Let be the motion vector of the target in the image coordinate system; Let be the motion vector of the target in the vehicle coordinate system; Let the angle between the two be the angle when Less than At that time, the trajectories were considered to be consistent; Each target trajectory is bound to a unique identifier and global metadata, including: starting frame number, tracking length, ID stability confidence, etc., for trajectory data integrity assessment and subsequent structured output; The final output is the trajectory sequence data for each vehicle target, which includes timestamp, center position, category, confidence level and image frame number, providing a temporal basis for the 3D state modeling module.

[0014] As described above, the mobile maintenance vehicle rear vehicle motion status detection system based on radar-visual fusion uses the vehicle flow motion status module to calculate the core parameters such as the spatial position, relative speed and acceleration of the vehicle target in the vehicle reference frame through binocular ranging and radar-assisted fusion based on continuous image frame data of the tracked target, and organizes them into a standardized structured output. First, extract the ROI region of each target in the current frame image, obtain its corresponding position coordinates in the left and right camera images, and send them into the calibrated binocular stereo correction process; Epipolar correction is performed on the left and right image pairs to unify the image scanning direction, eliminate the vertical component of stereo parallax, and reduce the matching task to a one-dimensional horizontal direction, thereby improving the matching speed and accuracy. The corrected image is input into the RAFT-Stereo stereo matching network. The network generates a dense disparity map through context fusion and learnable cost modeling, which is suitable for depth recovery in areas with weak texture and varying illumination. By using the baseline length and intrinsic parameter matrix of the binocular camera, the pixel disparity values ​​are converted into three-dimensional spatial coordinates of the target point, thus establishing the spatial positioning result of the vehicle target relative to the vehicle itself.

[0015] The advantages of this invention are: This invention significantly improves the system's perception robustness in various extreme environments such as low visibility (e.g., rain and fog) and complex lighting (e.g., backlighting and nighttime) by integrating the strong anti-interference capability and all-weather ranging capability of millimeter-wave radar with the high-resolution image information and color feature extraction capability of binocular cameras, thus breaking through the application boundaries of traditional single sensor systems. This invention optimizes the detection scenario under the rearward perspective of a moving vehicle. By introducing a vehicle-scale adaptive anchor frame mechanism and a multi-scale feature enhancement network, it effectively solves the recognition difficulties such as large changes in target size and blurred edges when the vehicle approaches rapidly. At the same time, the improved ID preservation mechanism and Hungarian matching algorithm enhance the trajectory continuity of the target under conditions such as frequent occlusion and dense traffic. Compared with existing visual detection schemes, the detection and tracking chain proposed in this invention can stably identify and continuously track target vehicles under complex conditions such as large vehicle speed differences and changing perspectives, providing a reliable data source for subsequent state modeling.

[0016] This invention integrates high-precision positioning data from multiple sources, such as IMU and BDS, to achieve dynamic trajectory fitting and real-time updating of key state parameters for vehicles approaching from behind. Based on a relative coordinate system construction method, it replaces the traditional static modeling logic of "road as reference" with a dynamic modeling mechanism of "vehicle as reference," solving problems such as coordinate drift and speed error accumulation in traditional systems on mobile platforms, and laying a physical quantity foundation for dynamic risk identification.

[0017] The structured state output module of this invention is based on a unified data representation model. It standardizes and integrates the vehicle target trajectory information provided by the detection and tracking module to form a multi-dimensional structured data format containing fields such as relative distance, relative speed, acceleration, vehicle type, lane number, and frame timestamp. This facilitates consistent parsing and downstream calls between different systems, avoiding the system integration difficulties and real-time performance degradation caused by the lack of structured interfaces in traditional detection systems. The system supports flexible docking with road perception platforms, vehicle control systems, and upper-level data analysis modules, and has good scalability and cross-platform adaptability, significantly improving the engineering usability and deployment efficiency of traffic flow status information. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of the motion state of the present invention; Figure 2 This is a schematic diagram of the integrated deployment of a multi-source sensing device instance based on the present invention. Figure 3 This is a diagram illustrating the effect of the invention on moving multi-target detection and cross-frame trajectory tracking under typical road conditions; Figure 4 This is a schematic diagram of the multi-source fusion driven three-dimensional motion state modeling and structured output of the backward traffic flow according to the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] like Figure 1 As shown, the mobile maintenance vehicle rear vehicle motion status detection system based on radar-visual fusion is characterized by including a multi-source perception module, a target detection module, a trajectory tracking module, and a vehicle motion status modeling module. The multi-source perception module is responsible for collecting information on vehicles and road alignment from behind. It uses a unified clock synchronization mechanism for time alignment and combines external parameter calibration to achieve spatial consistency, ensuring the fusion processing of multi-source data in the same coordinate system. The target detection module completes image distortion correction and feature enhancement through image preprocessing, extracts vehicle category, bounding box and confidence information in the image based on deep learning algorithm, and eliminates non-target area information by combining ROI region constraints and redundancy suppression logic, outputs effective vehicle detection results and uses them for subsequent tracking processing. The trajectory tracking module is based on the "detection-tracking" paradigm to perform cross-frame data association, and uses a trajectory matching algorithm to identify and number vehicle targets between frames, thereby realizing the continuous construction of multi-target trajectories and outputting target trajectory data with time labels and unique IDs, thus realizing target-level temporal tracking and trajectory structure construction. The vehicle motion state modeling module is based on binocular vision and millimeter-wave radar fusion to achieve high-precision calculation of the target's three-dimensional spatial position, velocity, and acceleration. It also uses the attitude angle provided by the IMU to perform coordinate compensation and stabilization processing on the target position. The state parameters of each target vehicle are organized into a standard format structure vector, including ID, timestamp, three-dimensional coordinates, velocity, and acceleration. It supports real-time push to the edge database and builds a vehicle flow historical trajectory database to support system integration and long-term data analysis tasks.

[0022] like Figure 2 As shown, specifically, the multi-source sensing module described in this embodiment includes an integrated multi-source sensing device installed on a mobile maintenance vehicle (such as above the gantry behind the cab), including a binocular camera, millimeter-wave radar, BDS positioning unit, IMU unit, 4G data antenna and edge computing processor, to ensure that each sensor is installed firmly and the field of view is unobstructed, meeting the requirement of full coverage of the rear lane. The multi-source sensing devices are subject to unified power supply management and communication line layout to ensure stable data flow between various hardware components and the edge computing platform, and data storage or Ethernet interfaces are reserved for data storage, external transmission, and remote debugging functions.

[0023] Specifically, the multi-source sensing module described in this embodiment uses a custom clock to uniformly configure the sampling frequencies of the binocular camera, millimeter-wave radar, BDS positioning unit, and IMU unit, and calibrates the system master clock to achieve time alignment of multimodal sensor data. The synchronization mechanism supports PTP (Precision Time Protocol) or high-frequency pulse calibration signals to meet the data time-domain alignment requirements in high frame rate scenarios. The time series uses UTC time, and the time delay with Universal Time should be within 100 milliseconds. Suppose the multi-source sensing module has N sensors, and the sampling time vector is: T Unified reference clock The time synchronization error vector is then: in, It is a vector of dimension N consisting entirely of 1s; The error constraint is: That is, the maximum mean square difference between the sampling time of all sensors and the system master clock must be less than a set threshold. The sampling frequency is uniformly constrained as follows: T in, For uniformly configured synchronous sampling frequency; During long-term operation, the clocks of various sensors will drift; the drift rate is defined. = in, For the random jitter term, the system corrects it by minimizing the squared drift error: Under PTP or high-frequency pulse synchronization mechanisms, through multi-mode clock alignment optimization, the final alignment target is: in, This is a weighted compensation matrix used for delay correction and jitter smoothing for different sensors.

[0024] This invention employs a standard multi-sensor extrinsic parameter calibration process to obtain the spatial relative positions and attitude parameters of the binocular camera, millimeter-wave radar, and IMU inertial navigation module in the vehicle coordinate system, serving as a prerequisite for multi-source fusion, 3D reconstruction, and state calculation.

[0025] More specifically, the multi-source sensing module described in this embodiment calibrates the intrinsic and extrinsic parameters of the binocular camera. By capturing multiple checkerboard images at different angles and distances, it obtains the corner feature positions and solves the camera's intrinsic parameter matrix (focal length, principal point coordinates, radial and tangential distortion coefficients) and extrinsic parameter matrix (rotation and translation vectors of the camera relative to the world coordinate system) according to a system of linear equations, thus constructing the geometric mapping relationship between the image coordinate system, pixel coordinate system, camera coordinate system and world coordinate system. The operation of the mathematical model with intrinsic and extrinsic parameters is as follows: After the internal and external parameters are calibrated, the reprojection error is calculated as an evaluation index of calibration accuracy. The reprojection error refers to the difference between the actual detected pixel and the estimated projection value after the three-dimensional world point is projected back to the image plane through the camera model. Its mean and variance reflect the geometric consistency of the calibration model in the actual projection scene. Let the world coordinate point be: T The camera projection model can then be written as: in, This is the intrinsic parameter matrix. ; This is the extrinsic parameter matrix of the camera, containing information such as rotation and translation. ∈ , Scale factor; Let the detected pixel coordinates be: T The predicted projection is: T The reprojection error of a single chessboard grid corner point can be expressed as: Introducing the Frobenius norm of the matrix, we further extend it to N feature points: in, ∈ This represents the observation matrix of all pixels; , represents the homogeneous coordinate matrix of all world points; The reprojection error is strictly controlled within 0.2 pixels to ensure the spatial accuracy of stereo matching and 3D modeling processes; To achieve spatiotemporal alignment of cross-modal sensing data (such as radar point clouds and images), the method of manually calibrating objects and collecting images and radar echoes is adopted to construct a set of spatially corresponding point pairs observed by both. The optimization is carried out by minimizing the point cloud projection error, and finally the transformation matrix between the radar and camera coordinate systems is obtained. The installation pose of the IMU module is registered with the camera coordinate system by combining manual measurement and static attitude estimation method, forming a unified vehicle coordinate framework. After power-on self-test, the system sequentially completes lens imaging test, radar beam response verification, inertial navigation initialization and positioning stability assessment. Combined with environmental parameters (such as weather conditions and light levels), it completes dynamic configuration and parameter tuning, providing a robust and stable sensing foundation for subsequent data acquisition processes.

[0026] More specifically, the target detection module described in this embodiment is responsible for real-time semantic recognition processing of image data, including bounding box detection, type classification and key location information extraction of vehicle targets, providing an input basis for subsequent trajectory association and spatial traffic flow motion state modeling; The target detection module extracts image frames from the synchronously acquired binocular image stream and performs image preprocessing, including fisheye distortion correction, color normalization, motion blur suppression, and image enhancement, to enhance the edge sharpness and image contrast of the detected target.

[0027] More specifically, the target detection module described in this embodiment aims to improve the robustness and scene adaptability of the target detection model in real-world working environments. Based on the standard COCO dataset, this system constructs a subset of working scene images covering typical adverse conditions such as rain and fog, low light, backlight, and image blur, forming an enhanced mobile work detection dataset. Let the original dataset distribution be... The enhanced distribution is Then we have: in, This represents an extended image sample generated through enhancement strategies such as brightness perturbation, contrast adjustment, rain and fog simulation, noise injection, and geometric transformation. During dataset construction, image enhancement techniques combined with field-collected samples are used to expand the training data distribution of the model in scenarios such as roadside work areas, nighttime operations, and low-contrast rainy days. This enhances its generalization ability in complex scenes. After introducing enhancement, the goal is to minimize the expected generalization model error, as shown in the formula: The above formula minimizes the expected generalization model error, where E is the expected value. For model generalization parameters, Indicates from augmented dataset Input samples obtained by random sampling and their corresponding tags , The parameter is The detection model for the input image The predicted output results; The loss function is used to measure the difference between the model's prediction and the true label. It consists of classification loss, bounding box regression loss, and target confidence loss. Image data streams are input into the detection model after transfer learning. The model extracts target semantic features through a backbone feature extraction module and a multi-scale fusion structure, and combines this with the detection head to generate multi-scale anchor boxes and target probability distributions. This represents the input image, with a size of [size missing]. , For Kronecker product, The formula for the RGB channel extension matrix in homogeneous pixel coordinates is as follows. , This represents the set of convolutional kernels in the backbone network, responsible for feature extraction. F is the extracted feature map, used in the downstream detection module. Indicates channel splicing iteration; The model uses a multi-scale feature block fusion mechanism to improve the ability to identify small targets at a distance; Block matrix: After feature fusion in blocks in, The bias matrix of the backbone network stores the weights and bias information of the convolutional kernels. The value at each position represents the importance of that pixel in the detection. After non-maximum suppression processing, redundant bounding boxes are removed from the detection results, and the final detection list is output and passed to the downstream target tracking unit. The image frame number, timestamp, bounding box coordinates, and target category are encapsulated together to construct the initial state of the target in the feature map. Attention weights are applied to enhance the ability to focus on target region features and improve recognition stability under cluttered backgrounds and partial occlusion conditions. A 3×3 spatial attention weight matrix is ​​defined as follows: Simultaneously define the channel attention weight matrix: Construct the joint attention block matrix: Enhanced feature map By assigning higher weights to vehicle regions and lower weights to background regions, interference is suppressed and feature writing is enhanced. in, To obtain the diagonal enhancement operator, used to ensure local feature weights, To adjust the parameters, It is a non-linear activation function; The detection output is also used to establish lane region filtering logic. By fusing image segmentation results or ROI information obtained from perspective transformation, only valid targets within the lane are retained, and the class probability and bounding box parameters are output, thereby reducing the false detection rate and subsequent processing pressure, and improving the overall system efficiency. The combined linear transformation and nonlinear mapping operations are as follows: in, , The weights and biases for the classification branches; , represents the probability vector of the target belonging to each category (𝐶 is the number of categories, such as cars, trucks, buses, etc.); For activation functions; , The weights and biases of the bounding box regression branch; , representing the predicted bounding box parameters, and , representing the center point coordinates ( ),width ;high ; The activation function represents the block diagonal mapping (Softmax for classification and linear identity for regression).

[0028] Furthermore, to ensure numerical stability in real-time engineering scenarios, the target detection module described in this embodiment outputs a normalized form: Final detection box confidence for: in, This indicates the probability that the bounding box contains an object (Objectness score). This represents the maximum classification probability of the most likely category of the box; To assess the overall confidence level and measure the reliability of the test results, this information is used for subsequent screening. In the candidate box set = , In the definition of intersection matrix Define area vector For two candidate boxes, this reflects the degree of overlap between them. matrix ,satisfy: Constructing the adjacency matrix : in, The threshold parameter is set. The nonmaximum suppression problem can be formalized as the maximum weighted independent set problem: in, , represents the confidence vector, where each element is the detection score of the corresponding candidate box; Indicates the selection vector. Indicates that candidate boxes are retained. ; Represented by the adjacency matrix The defined set of conflicting edges; For ease of solution, it can be relaxed into a convex optimization problem: in, This is the penalty coefficient, used to constrain the joint selection of overlapping candidate boxes, balancing confidence and overlap penalty; Introducing the Graph Laplacian Operator Then the optimization objective is: The boxes with lower confidence are then removed, resulting in a high-confidence target detection result.

[0029] like Figure 3 As shown, further, the trajectory tracking module described in this embodiment is responsible for associating the detected vehicle targets with data in consecutive frame images, constructing the cross-frame trajectory of each vehicle, maintaining the consistency of target IDs, and forming a stable temporal tracking structure; A "two-stage matching + confidence level" association strategy is constructed. In the first stage, high-confidence detection boxes are used to perform main matching with historical trajectories to ensure the accuracy of matching to the greatest extent. A "motion prediction-assisted association" mechanism is introduced during the tracking process. A short-time velocity vector is constructed using the positions of the first two frames to predict the next possible location of the target, improving the matching robustness in occluded or weakly detected scenarios. The target's state vector at time t is defined as: in, It is a position vector; It is the velocity vector; This is the acceleration vector.

[0030] Furthermore, the trajectory tracking module described in this embodiment constructs a state transition model: in, , It is a 2-order identity matrix. The process noise covariance matrix is... This represents the predicted current position, which is used as a candidate point to compensate for matching issues caused by occlusion or weak detection. , This indicates the target's position in the first two frames (center coordinates). The calculation formula is: To address the special working conditions of large changes in the field of view and drastic background dynamics of vehicle-mounted cameras, prior constraints on the target motion direction and acceleration are introduced to filter out target trajectories with obvious jumps or abnormal motion characteristics. This reduces ID drift and trajectory fragmentation problems caused by occlusion or detection interruptions. The prior constraint formula for motion parameters is as follows: in, This represents the velocity vector of the target at time t. The acceleration vector of the target at time t; Indicates the inter-frame time interval; The association constraint is optimized as follows: in, Indicates the maximum speed threshold; This represents the maximum acceleration threshold. If a solution violates the constraint, it is judged as an abnormal motion trajectory and is discarded. A spatial consistency verification mechanism is employed to compare the target's motion direction in the image coordinate system and the vehicle coordinate system, ensuring that the tracking results conform to the actual traffic flow direction and further improving the system's robustness. The formula for comparing the motion direction in the image coordinate system and the vehicle coordinate system is as follows: in, Let be the motion vector of the target in the image coordinate system; Let be the motion vector of the target in the vehicle coordinate system; Let the angle between the two be the angle when Less than At that time, the trajectories were considered to be consistent; Each target trajectory is bound to a unique identifier and global metadata, including: starting frame number, tracking length, ID stability confidence, etc., for trajectory data integrity assessment and subsequent structured output; The final output is the trajectory sequence data for each vehicle target, which includes timestamp, center position, category, confidence level and image frame number, providing a temporal basis for the 3D state modeling module.

[0031] like Figure 4 As shown, further, the vehicle flow motion state module in this embodiment calculates the core parameters such as the spatial position, relative speed and acceleration of the vehicle target in the vehicle reference frame by using binocular ranging and radar-assisted fusion based on the continuous image frame data of the tracked target, and organizes them into a standardized structured output; First, extract the ROI region of each target in the current frame image, obtain its corresponding position coordinates in the left and right camera images, and send them into the calibrated binocular stereo correction process; Epipolar correction is performed on the left and right image pairs to unify the image scanning direction, eliminate the vertical component of stereo parallax, and reduce the matching task to a one-dimensional horizontal direction, thereby improving the matching speed and accuracy. The corrected image is input into the RAFT-Stereo stereo matching network. The network generates a dense disparity map through context fusion and learnable cost modeling, which is suitable for depth recovery in areas with weak texture and varying illumination. By using the baseline length and intrinsic parameter matrix of the binocular camera, the pixel disparity values ​​are converted into three-dimensional spatial coordinates of the target point, thus establishing the spatial positioning result of the vehicle target relative to the vehicle itself.

[0032] This invention constructs a continuous frame three-dimensional position trajectory, uses a sliding window difference and regression algorithm to calculate the target velocity and acceleration, and automatically removes abnormal data segments to maintain the smoothness of the state solution.

[0033] To overcome the ranging instability caused by pitch, vibration, and yaw during vehicle operation, this invention introduces IMU inertial navigation data for attitude compensation, mapping the target's three-dimensional coordinates to a stable vehicle reference frame.

[0034] This invention performs a weighted fusion of target range provided by millimeter-wave radar and binocular estimation results. The fusion weights are dynamically adjusted based on radar signal strength, target range, and visual confidence. Finally, a Kalman filter is used to smooth the output.

[0035] The multi-source fusion strategy of this invention significantly improves the ranging stability and depth recognition capability of the system in actual working environments, especially in high reflectivity, weak texture or oblique viewing angles, it can still maintain estimation accuracy.

[0036] For each tracked target, the system generates a structured state vector at each moment, with fields including: target ID, timestamp, relative three-dimensional coordinates, velocity vector, acceleration vector, target type, tracking stability, etc.

[0037] The structured status data of this invention supports high-frequency (≥10Hz) refresh and continuous output, and can be transmitted to the upper-level platform, road monitoring center or vehicle control system via CAN bus, Ethernet or wireless link.

[0038] This invention establishes a trajectory cache buffer in an edge computing platform, constructs a historical vehicle flow trajectory database, and configures a high-performance non-relational database or a lightweight time-series database for real-time writing of structured vehicle status data. The database uses vehicle ID + timestamp as a composite primary key to enhance the system's vertical accumulative capability.

[0039] The trajectory library of this invention supports data partitioning and compressed storage in hours / days, allows for flexible setting of retention duration strategies, and provides tool interfaces such as trajectory playback, target recovery, and label annotation, which are convenient for later use in training risk identification models or optimizing detection strategies.

[0040] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A mobile maintenance vehicle rear vehicle motion status detection system based on radar-visual fusion, characterized in that: It includes a multi-source perception module, a target detection module, a trajectory tracking module, and a vehicle motion state modeling module; The multi-source perception module is responsible for collecting information on vehicles and road alignment from behind. It uses a unified clock synchronization mechanism for time alignment and combines external parameter calibration to achieve spatial consistency, ensuring the fusion processing of multi-source data in the same coordinate system. The target detection module completes image distortion correction and feature enhancement through image preprocessing, extracts vehicle category, bounding box and confidence information in the image based on deep learning algorithm, and eliminates non-target area information by combining ROI region constraints and redundancy suppression logic, outputs effective vehicle detection results and uses them for subsequent tracking processing. The trajectory tracking module is based on the "detection-tracking" paradigm to perform cross-frame data association, and uses a trajectory matching algorithm to identify and number vehicle targets between frames, thereby realizing the continuous construction of multi-target trajectories and outputting target trajectory data with time labels and unique IDs, thus realizing target-level temporal tracking and trajectory structure construction. The vehicle motion state modeling module is based on binocular vision and millimeter-wave radar fusion to achieve high-precision calculation of the target's three-dimensional spatial position, velocity, and acceleration. It also uses the attitude angle provided by the IMU to perform coordinate compensation and stabilization processing on the target position. The state parameters of each target vehicle are organized into a standard format structure vector, including ID, timestamp, three-dimensional coordinates, velocity, and acceleration. It supports real-time push to the edge database and builds a vehicle flow historical trajectory database to support system integration and long-term data analysis tasks.

2. The mobile maintenance vehicle rear vehicle motion status detection system based on radar-visual fusion as described in claim 1, characterized in that: The multi-source sensing module includes an integrated multi-source sensing device installed on the mobile maintenance vehicle, including a binocular camera, millimeter-wave radar, BDS positioning unit, IMU unit, 4G data antenna, and edge computing processor; The multi-source sensing devices are subject to unified power supply management and communication line layout to ensure stable data flow between various hardware components and the edge computing platform, and data storage or Ethernet interfaces are reserved for data storage, external transmission, and remote debugging functions.

3. The mobile maintenance vehicle rear vehicle motion status detection system based on radar-visual fusion as described in claim 2, characterized in that: The multi-source sensing module uses a custom clock to uniformly configure the sampling frequencies of the binocular camera, millimeter-wave radar, BDS positioning unit, and IMU unit, and calibrates the system's main clock to achieve time alignment of multimodal sensor data. Suppose the multi-source sensing module has N sensors, and the sampling time vector is: T Unified reference clock The time synchronization error vector is then: in, It is a vector of dimension N consisting entirely of 1s; The error constraint is: That is, the maximum mean square difference between the sampling time of all sensors and the system master clock must be less than a set threshold. The sampling frequency is uniformly constrained as follows: T in, For uniformly configured synchronous sampling frequency; During long-term operation, the clocks of various sensors will drift; the drift rate is defined. = in, For the random jitter term, the system corrects it by minimizing the squared drift error: Under PTP or high-frequency pulse synchronization mechanisms, through multi-mode clock alignment optimization, the final alignment target is: in, This is a weighted compensation matrix used for delay correction and jitter smoothing for different sensors.

4. The mobile maintenance vehicle rear vehicle motion status detection system based on radar-visual fusion as described in claim 2, characterized in that: The multi-source sensing module calibrates the intrinsic and extrinsic parameters of the binocular camera. By capturing multiple chessboard images from different angles and distances, it obtains the corner feature positions and solves the camera's intrinsic and extrinsic parameter matrices according to a system of linear equations to construct the geometric mapping relationship between the image coordinate system, pixel coordinate system, camera coordinate system and world coordinate system. The operation of the mathematical model with intrinsic and extrinsic parameters is as follows: After the internal and external parameters are calibrated, the reprojection error is calculated as an evaluation index of calibration accuracy. The reprojection error refers to the difference between the actual detected pixel and the estimated projection value after the three-dimensional world point is projected back to the image plane through the camera model. Its mean and variance reflect the geometric consistency of the calibration model in the actual projection scene. Let the world coordinate point be: T The camera projection model can then be written as: in, This is the intrinsic parameter matrix. ; This is the extrinsic parameter matrix of the camera, containing information such as rotation and translation. ∈ , Scale factor; Let the detected pixel coordinates be: T The predicted projection is: T The reprojection error of a single chessboard grid corner point can be expressed as: Introducing the Frobenius norm of the matrix, we further extend it to N feature points: in, ∈ This represents the observation matrix of all pixels; , represents the homogeneous coordinate matrix of all world points; The reprojection error is strictly controlled within 0.2 pixels to ensure the spatial accuracy of stereo matching and 3D modeling processes; To achieve spatiotemporal alignment of cross-modal sensing data, the method of manually calibrating objects and collecting images and radar echoes is adopted to construct a set of spatially corresponding point pairs observed by both. The transformation matrix between the radar and camera coordinate systems is finally obtained by minimizing the point cloud projection error. The installation pose of the IMU module is registered with the camera coordinate system by combining manual measurement and static attitude estimation, forming a unified vehicle coordinate framework. After power-on self-test, the system sequentially completes lens imaging test, radar beam response verification, inertial navigation initialization and positioning stability assessment, and dynamically configures parameters based on environmental parameters, providing a robust and stable sensing foundation for subsequent data acquisition processes.

5. The mobile maintenance vehicle rear vehicle motion status detection system based on radar-visual fusion as described in claim 1, characterized in that: The target detection module is responsible for real-time semantic recognition processing of image data, including bounding box detection, type classification and key location information extraction of vehicle targets, providing an input basis for subsequent trajectory association and spatial traffic flow motion state modeling; The target detection module extracts image frames from the synchronously acquired binocular image stream and performs image preprocessing, including fisheye distortion correction, color normalization, motion blur suppression, and image enhancement, to enhance the edge sharpness and image contrast of the detected target.

6. The mobile maintenance vehicle rear vehicle motion status detection system based on radar-visual fusion as described in claim 5, characterized in that: To improve the robustness and scene adaptability of the target detection model in real-world working environments, the system constructs a subset of working scene images covering typical adverse conditions such as rain, fog, low light, backlight, and image blur, based on the standard COCO dataset. This forms an enhanced mobile work detection dataset. The original dataset is distributed as follows: The enhanced distribution is Then we have: in, This represents an extended image sample generated through enhancement strategies such as brightness perturbation, contrast adjustment, rain and fog simulation, noise injection, and geometric transformation. During dataset construction, image enhancement techniques combined with field-collected samples are used to expand the training data distribution of the model in scenarios such as roadside work areas, nighttime operations, and low-contrast rainy days. This enhances its generalization ability in complex scenes. After introducing enhancement, the goal is to minimize the expected generalization model error, as shown in the formula: The above formula minimizes the expected generalization model error, where E is the expected value. For model generalization parameters, Indicates from augmented dataset Input samples obtained by random sampling and their corresponding tags , The parameter is The detection model for the input image The predicted output results; The loss function is used to measure the difference between the model's prediction and the true label. It consists of classification loss, bounding box regression loss, and target confidence loss. Image data streams are input into the detection model after transfer learning. The model extracts target semantic features through a backbone feature extraction module and a multi-scale fusion structure, and combines this with the detection head to generate multi-scale anchor boxes and target probability distributions. This represents the input image, with a size of [size missing]. , For Kronecker product, The formula for the RGB channel extension matrix in homogeneous pixel coordinates is as follows. , This represents the set of convolutional kernels in the backbone network, responsible for feature extraction. F is the extracted feature map, used in the downstream detection module. Indicates channel splicing iteration; The model uses a multi-scale feature block fusion mechanism to improve the ability to identify small targets at a distance; Block matrix: After feature fusion in blocks in, The bias matrix is ​​the backbone network, which stores the weights and bias information of the convolutional kernels. The value at each position represents the importance of that pixel in the detection. After non-maximum suppression processing, redundant bounding boxes are removed from the detection results, and the final detection list is output and passed to the downstream target tracking unit. The image frame number, timestamp, bounding box coordinates, and target category are encapsulated together to construct the initial state of the target in the feature map. Attention weights are applied to enhance the ability to focus on target region features and improve recognition stability under cluttered backgrounds and partial occlusion conditions. A 3×3 spatial attention weight matrix is ​​defined as follows: Simultaneously define the channel attention weight matrix: Construct the joint attention block matrix: Enhanced feature map By assigning higher weights to vehicle regions and lower weights to background regions, interference is suppressed and feature writing is enhanced. in, To obtain the diagonal enhancement operator, used to ensure local feature weights, To adjust the parameters, It is a non-linear activation function; The detection output is also used to establish lane region filtering logic. By fusing image segmentation results or ROI information obtained from perspective transformation, only valid targets within the lane are retained, and the class probability and bounding box parameters are output, thereby reducing the false detection rate and subsequent processing pressure, and improving the overall system efficiency. The combined linear transformation and nonlinear mapping operations are as follows: in, , The weights and biases for the classification branches; , representing the probability vector of the target belonging to each category; For activation functions; , The weights and biases of the bounding box regression branch; , representing the predicted bounding box parameters, and , representing the center point coordinates ( ),width ;high ; This represents the activation function for a block diagonal mapping.

7. The mobile maintenance vehicle rear vehicle motion status detection system based on radar-visual fusion as described in claim 6, characterized in that: To ensure numerical stability in real-time engineering scenarios, the target detection module outputs in a normalized form. Final detection box confidence for: in, This indicates the probability that the bounding box contains the target. This represents the maximum classification probability of the most likely category of the box; To assess the overall confidence level and measure the reliability of the test results, this information is used for subsequent screening. In the candidate box set = , In the definition of intersection matrix Define area vector For two candidate boxes, this reflects the degree of overlap between them. matrix ,satisfy: Constructing the adjacency matrix : in, The threshold parameter is set. The nonmaximum suppression problem can be formalized as the maximum weighted independent set problem: in, , represents the confidence vector, where each element is the detection score of the corresponding candidate box; Indicates the selection vector. Indicates that candidate boxes are retained. ; Represented by the adjacency matrix The defined set of conflicting edges; For ease of solution, it can be relaxed into a convex optimization problem: in, This is the penalty coefficient, used to constrain the joint selection of overlapping candidate boxes, balancing confidence and overlap penalty; Introducing the Graph Laplacian Operator Then the optimization objective is: The boxes with lower confidence are then removed, resulting in a high-confidence target detection result.

8. The mobile maintenance vehicle rear vehicle motion status detection system based on radar-visual fusion according to claim 1, characterized in that: The trajectory tracking module is responsible for associating the detected vehicle targets with data in consecutive frame images, constructing the cross-frame trajectory of each vehicle, maintaining the consistency of target IDs, and forming a stable temporal tracking structure. A "two-stage matching + confidence level" association strategy is constructed. In the first stage, high-confidence detection boxes are used to perform main matching with historical trajectories to maximize matching accuracy. A "motion prediction-assisted association" mechanism is introduced during the tracking process. This mechanism uses the positions of the first two frames to construct a short-time velocity vector, predicting the next possible location of the target. This improves the matching robustness in occluded or weakly detected scenarios. The target's state vector at time t is defined as follows: in, It is a position vector; It is the velocity vector; This is the acceleration vector.

9. The mobile maintenance vehicle rear vehicle motion status detection system based on radar-visual fusion as described in claim 8, characterized in that: The trajectory tracking module constructs a state transition model: in, , It is a 2-order identity matrix. The process noise covariance matrix is... This represents the predicted current position, which is used as a candidate point to compensate for matching issues caused by occlusion or weak detection. , This indicates the target's position in the first two frames (center coordinates). The calculation formula is: To address the special working conditions of large changes in the field of view and drastic background dynamics of vehicle-mounted cameras, prior constraints on the target motion direction and acceleration are introduced to filter out target trajectories with obvious jumps or abnormal motion characteristics. This reduces ID drift and trajectory fragmentation problems caused by occlusion or detection interruptions. The prior constraint formula for motion parameters is as follows: in, This represents the velocity vector of the target at time t. The acceleration vector of the target at time t; Indicates the inter-frame time interval; The association constraint is optimized as follows: in, Indicates the maximum speed threshold; This represents the maximum acceleration threshold. If a solution violates the constraint, it is judged as an abnormal motion trajectory and is discarded. A spatial consistency verification mechanism is employed to compare the target's motion direction in the image coordinate system and the vehicle coordinate system, ensuring that the tracking results conform to the actual traffic flow direction and further improving the system's robustness. The formula for comparing the motion direction in the image coordinate system and the vehicle coordinate system is as follows: in, Let be the motion vector of the target in the image coordinate system; Let be the motion vector of the target in the vehicle coordinate system; Let the angle between the two be the angle when Less than At that time, the trajectories were considered to be consistent; Each target trajectory is bound to a unique identifier and global metadata, including: starting frame number, tracking length, ID stability confidence, etc., for trajectory data integrity assessment and subsequent structured output; The final output is the trajectory sequence data for each vehicle target, which includes timestamp, center position, category, confidence level and image frame number, providing a temporal basis for the 3D state modeling module.

10. The mobile maintenance vehicle rear vehicle motion status detection system based on radar-visual fusion according to claim 1, characterized in that: The vehicle flow motion state module calculates the core parameters of the vehicle target, such as its spatial position, relative velocity, and acceleration in the vehicle reference frame, based on continuous image frame data of the tracked target through binocular ranging and radar-assisted fusion, and organizes them into a standardized structured output. First, extract the ROI region of each target in the current frame image, obtain its corresponding position coordinates in the left and right camera images, and send them into the calibrated binocular stereo correction process; Epipolar correction is performed on the left and right image pairs to unify the image scanning direction, eliminate the vertical component of stereo parallax, and reduce the matching task to a one-dimensional horizontal direction, thereby improving the matching speed and accuracy. The corrected image is input into the RAFT-Stereo stereo matching network. The network generates a dense disparity map through context fusion and learnable cost modeling, which is suitable for depth recovery in areas with weak texture and varying illumination. By using the baseline length and intrinsic parameter matrix of the binocular camera, the pixel disparity values ​​are converted into three-dimensional spatial coordinates of the target point, thus establishing the spatial positioning result of the vehicle target relative to the vehicle itself.

Images