An online road condition detection system and method based on multi-source information fusion

Abstract

The present invention discloses an online road condition detection system and method based on multi-source information fusion, the system includes a radar sensor module for measuring the position, speed information and sampling period of the target to be measured; a video sensor module for acquiring image data of various scenes; a data processing module for fusing the information of the radar sensor module and the video sensor module, and calculating the traffic flow information; a parameter configuration module for configuring parameters such as lane configuration, radar calibration and camera calibration; a visual display module for realizing the output and storage display of data results and information of each module. The present invention overcomes the problems of information inaccuracy and susceptibility to environmental interference caused by a single information method and system, and increases the comprehensiveness of road condition detection information through multi-source information fusion.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent transportation technology, and in particular to an online road condition detection system and method based on the fusion of information from multiple sensors. Background Art

[0002] Driven by national policies, the construction of "smart cities" has become a core strategic force in leading urban development. As traffic monitoring methods and systems become increasingly widespread, practical demands and challenges are also increasing. Existing road condition detection methods and systems primarily fall into two categories: those based on video sensing and those based on radar sensing.

[0003] Road condition detection methods and systems based on video sensing process and analyze road video footage of the monitored area to determine the presence and location of vehicles in the image. Based on changes in image position, they can roughly estimate the vehicle's distance and speed. Video-based road condition detectors provide images of the road surface and have excellent detection performance for both large vehicles and densely packed small vehicles. They also offer advantages in identifying different vehicle types and pedestrians. However, video-based detection methods and systems are significantly affected by lighting and weather conditions, and their detection range is limited, generally exceeding 100 meters. Road condition detection methods and systems based on radar sensing transmit and receive modulated continuous waves within the detection area. Based on point cloud data analysis, they can quickly calculate the precise location and speed of targets. Radar-based detection methods and systems are less affected by weather conditions and the change of day and night. However, radar-based vehicle detectors lack on-site road image information, cannot detect stopped objects, have poor detection performance for large vehicles, and cannot effectively distinguish between large vehicles, small cars, non-motor vehicles, and pedestrians.

[0004] In today's smart transportation sector, the fusion of radar and video data can be categorized into early fusion, mid-term fusion, and tracking result fusion. Early fusion involves performing logical operations on radar data and target detection bounding boxes, combined with the probability of target detection bounding boxes, before target tracking. This approach can result in a large number of missed detections. Mid-term fusion involves using radar data to verify video data, or using video detection bounding boxes to verify radar data, during the target tracking process. This approach offers better accuracy than early fusion, but is overly dependent on the radar point's own signal or the accuracy of visual target detection. Tracking result fusion involves combining vision-based and radar-based tracking within multiple timeframes. This approach also relies on the accuracy of either method. Therefore, it is essential to develop an efficient fusion method. In recent years, with the development of deep learning technology, algorithms based on various sensor data have been widely applied. Therefore, given the outstanding performance of deep learning technology in other fields, its application to the fusion of radar and video data is inevitable. In recent years, some excellent related work has also emerged, but it lacks robustness and accuracy in real-world complex scenario applications. Therefore, there is still much room for improvement in data fusion for project implementation. Summary of the Invention

[0005] The purpose of the present invention is to address the shortcomings of the above-mentioned single-sensor online road condition detection method and system, and propose an online road condition detection method and system based on multi-source information fusion, which realizes the accurate measurement of traffic flow information of road condition targets such as pedestrians, motor vehicles, and non-motor vehicles.

[0006] The technical solution to achieve the purpose of the present invention is: an online road condition detection system based on the fusion of information from multiple sensors, the system including a radar sensor module, a video sensor module, a data processing module, a parameter configuration module and a visual display module;

[0007] The radar sensor module is used to measure the position, speed and sampling period of the target to be measured;

[0008] The video sensing module is used to obtain image data of various scenes;

[0009] Here, the radar sensor module and the video sensor module collect information for the same detection area;

[0010] The data processing module is used to fuse the information of the radar sensor module and the video sensor module, and calculate the traffic flow information;

[0011] The parameter configuration module is used to configure parameters such as lane configuration, radar calibration, and camera calibration;

[0012] The visual display module is used to realize the output, storage and display of data results and information of each module.

[0013] Preferably, the target detection in the data processing module is based on a deep learning model, and extracts the center position of the target in the image coordinate system, the height and width of the target in the image coordinate system, and the ReID category features of the target.

[0014] Preferably, the data processing module regresses a perspective transformation model from the radar coordinate system to the image coordinate system based on the radar target information, thereby achieving the function of intelligent registration.

[0015] Preferably, the data processing module uses a multi-target ReID feature extracted after multi-source information fusion for target comparison in multi-target tracking, and performs shallow feature extraction and deep feature extraction based on the fused features respectively, and can simultaneously output perspective transformation information and target detection information, covering the spatial distance and speed information of the target point, while overcoming the errors introduced by sensor noise.

[0016] Preferably, the deep neural network in the data processing module uses an end-to-end training method, and simultaneously trains the shallow perspective transformation matrix fitting module and the deep object detection and ReID feature extraction module.

[0017] Preferably, the sample label generation method used in the training shallow perspective transformation matrix fitting module in the data processing module, in addition to using the method of manually calibrating corresponding points to calculate the perspective transformation matrix, considers transforming the image and then calculating a new perspective transformation matrix based on the corresponding points to achieve the effect of data augmentation; in order to achieve the diversity of vehicle positions in the scene, the correspondence between multiple frames of radar points and pixel points is calculated considering each camera point, and used as the true value in the training process.

[0018] Preferably, the sample label generation method used in the training deep target detection and ReID feature extraction module in the data processing module uses a manual calibration method to calibrate the tracking ID of multiple targets in multiple scenes, and performs frame skipping in the process of ID labeling, while also meeting the need to label different vehicles with different IDs; in the collection and labeling of training samples, different samples across scenes, weather conditions, and time are used for labeling.

[0019] Preferably, the data processing module uses the ID of multi-target tracking to count the traffic volume in the time period, determines whether the vehicle has crossed the line by calculating the relative position of the target and the lane line in the image, obtains the traffic light information of the road section from the signal machine, and determines whether there is a red light running behavior based on the relative position of the target and the lane line in the image.

[0020] Preferably, the parameter configuration module configures the lane lines, limits the position range of the radar information, and sets the radar data calibration parameters and the camera data calibration parameters; when the target leaves the lane range, the tracking target information deletion setting is performed, and the radar calibration and camera calibration parameters change with the change of the radar and camera positions.

[0021] Preferably, the visualization module includes: real-time image display, real-time detection result display, real-time traffic flow information display, violation statistics display, and information output and storage. The image display is collected and displayed by the video sensor module. The real-time detection results include single radar detection and tracking results, video target detection results, and the fusion tracking results of the two. The traffic flow information includes the location and movement status of pedestrians, motor vehicles, and non-motor vehicles. The violations include lane crossing information and red light running information.

[0022] Furthermore, the data processing module includes:

[0023] The scene recognition submodule is used to assess the video quality of the scene based on a deep neural network;

[0024] The target detection and ReID feature extraction submodule is used to fuse radar data with video data to generate target detection frames and the target's ReID features for target tracking;

[0025] The post-processing submodule is used to obtain traffic flow information and illegal behaviors based on the tracking results and update the parameter configuration module.

[0026] Furthermore, the scene recognition submodule includes:

[0027] The preprocessing submodule is used to preprocess the video frame image. The preprocessing includes: first, transforming the image size to 64x64, then subtracting the mean of the image dataset and dividing it by the variance of the image dataset to obtain a normalized distribution;

[0028] The recognition submodule is used to pass the preprocessed image through the four stages and linear layer of the ResNet-9 deep neural network in sequence, and output the probability value of the video quality being normal / abnormal. If the normal probability value is greater than or equal to 0.5, the object detection and ReID feature extraction submodules are executed.

[0029] Furthermore, the target detection and ReID feature extraction submodule includes:

[0030] The multi-source input information acquisition submodule is used to collect radar data at multiple times and single-frame image data. The sampling set is shown in the following formula:

[0031] S={F n Rn-1 R n R n+1}

[0032] Among them, F n T n The video frame at time R n-1 、R n 、R n+1 T n-1 、T n 、T n+1 Radar data at three moments;

[0033] The first shallow feature extraction submodule is used to extract the shallow features R of multi-time radar data. s ;

[0034] The second shallow feature extraction submodule is used to extract the shallow features F of single-frame image data. s ;

[0035] Multi-source feature fusion submodule is used to fusion shallow features R of multi-time radar data s Compared with the shallow feature F of single frame video s to integrate;

[0036] A third shallow feature extraction submodule is used to extract shallow features of the features fused by the multi-source feature fusion submodule, obtain perspective transformation matrix regression values, and regress a perspective transformation matrix model from the radar coordinate system to the image coordinate system;

[0037] The deep feature extraction submodule is used to extract the deep features of the features fused by the multi-source feature fusion submodule, and regress the center position of the target in the image coordinate system, the height and width of the target in the image coordinate system, and the ReID features of the target.

[0038] Furthermore, the first shallow feature extraction submodule extracts shallow features R of multi-time radar data. s The specific process includes:

[0039] The radar data at a single moment contains 4 attribute values Where X represents the horizontal coordinate in the radar coordinate system, Y represents the vertical coordinate in the radar coordinate system, and V x Indicates the velocity in the horizontal direction, V y Indicates the speed in the vertical direction;

[0040] (1) The radar data at a single moment is divided into two channels, channel 1 represents the velocity in the X direction, and channel 2 represents the velocity in the Y direction;

[0041] (2) In the channel dimension, the radar data of three moments are superimposed to obtain the 6-channel multi-time radar data R input ;

[0042] (3) Transform the plane of the radar coordinate system into a size of 224x224, and then find the maximum value for normalization;

[0043] (4) The input is sent to the convolution layer with a convolution kernel size of 7x7 and a step size of 2. The output result is then batch normalized and processed by ReLu. Finally, the shallow feature R of the multi-time radar data with a channel number of 64 is obtained. s , the calculation formula is:

[0044] R s =relu(bn(Conv(R input ))).

[0045] Furthermore, the second shallow feature extraction submodule extracts shallow features F of single-frame image data. s The specific process includes:

[0046] A single frame of video data consists of three channels, denoted as F input ;

[0047] (1) Transform the image data to 224x224 size and divide each channel by 255 to obtain the normalized distribution;

[0048] (2) Input the convolution layer with a convolution kernel size of 7x7 and a step size of 2, and perform BatchNormalization on the output result and then process it through ReLu to finally obtain the shallow feature F of a single-frame video with 64 channels. s , the calculation formula is:

[0049] F s =relu(bn(Conv(F input ))).

[0050] Furthermore, the third shallow feature extraction submodule is used to extract shallow features of the features fused by the multi-source feature fusion submodule, obtain perspective transformation matrix regression values, and regress a perspective transformation matrix model from the radar coordinate system to the image coordinate system. The specific process includes:

[0051] The fused features are subjected to three layers of convolution and BatchNormalization operations. The convolution kernel size of the three layers of convolution operations is 1x1, and the number of output channels is 32, 16, and 9 respectively. Finally, the mean is taken in the spatial dimension, and 9 regression values ​​of the perspective transformation matrix are output.

[0052] Furthermore, the deep feature extraction submodule is used to extract the deep features of the features fused by the multi-source feature fusion submodule, and regress the center position of the target in the image coordinate system, the height and width of the target in the image coordinate system, and the ReID features of the target. The specific process includes:

[0053] (1) The fused features are passed through the second, third, and fourth stages of the ResNet-18 deep neural network to obtain deep features;

[0054] (2) Input the deep features into the target center point sub-branch heads center , target size regression sub-branch heads wh , target center point error sub-branch heads offset , ReID sub-branch heads reid , after decoding operation, the predicted target detection box S is obtained bbox And the feature Feature at the center point of the target reid ; The network structure of the above sub-branch is in CenterNet form;

[0055] The center point of the detection box is output as Center bbox , the detection frame width and height are output as WH bbox , the error of the center point of the detection frame is Offset bbox , calculated as:

[0056] Center bbox =heads center (Stage4(Stage3(Stage2(a×F s +b×R s ))))

[0057] WH bbox =heads wh (Stage4(Stage3(Stage2(a×F s +b×R s ))))

[0058] Offset bbox =heads offset (Stage4(Stage3(Stage2(a×F s +b×R s ))))

[0059] Feature reid =heads reid (Stage4(Stage3(Stage2(a×F s +b×Rs ))))

[0060] S bbox =decode(Center bbox ,WH bbox ,Offset bbox ).

[0061] Furthermore, the target detection and ReID feature extraction submodule performs target tracking processing based on the generated target detection frame and the ReID feature of the target, specifically including:

[0062] (1) The target detection box and ReID features are respectively calculated with the historical features maintained by the tracking sequence and the detection box predicted by the Kalman filter for distance cost matrix calculation; wherein, the ReID features are measured with the historical features of each ID using the cosine distance, and the target detection box and the detection box predicted by the Kalman filter are measured using 1-IoU;

[0063] (2) The weighted sum of the two distance cost matrices is used to obtain the overall distance cost matrix;

[0064] (3) performing Hungarian matching based on the overall distance cost matrix to obtain the tracking ID of each target in the current frame;

[0065] (4) Update the historical features and tracking sequence of the tracked ID.

[0066] Furthermore, the traffic flow information includes the position and movement status information of pedestrians, motor vehicles and non-motor vehicles, as well as traffic volume information; the illegal behaviors include crossing lane lines and running red lights; and the updating parameter configuration module includes updating the solved perspective transformation matrix into the parameter configuration module.

[0067] An online road condition detection method based on information fusion of multiple sensors, the method comprising the following steps:

[0068] Step 1: Configure the scene and sensors through the parameter configuration module; the sensors include radar sensors and video sensors;

[0069] Step 2: Obtain radar data and video data of the road;

[0070] Step 3: Use the data processing module to fuse the sensor data and generate the target detection frame and the ReID feature of the target for target tracking;

[0071] Step 4: Use the data processing module to obtain traffic flow information and illegal behaviors based on the tracking results, and update the parameter configuration module;

[0072] Step 5: Feedback the required information to the visualization display module.

[0073] Compared with the prior art, the present invention has the following significant advantages:

[0074] 1) The system integrates information from radar sensors and video sensors and performs trajectory tracking, ensuring a sufficiently large detection area, sufficiently accurate data, and ignoring the influence of climatic conditions. It also provides on-site image information, completes the detection of stationary targets and large vehicles, and distinguishes various types of vehicle models, achieving accurate measurement of road conditions and statistics on illegal vehicles.

[0075] 2) Based on the fused features, shallow feature extraction and deep feature extraction are performed separately, which can simultaneously output perspective transformation information and target detection information, covering the spatial distance and speed information of the target point, while overcoming the errors introduced by sensor noise;

[0076] 3) It overcomes the problems of information inaccuracy and susceptibility to environmental interference brought by single information methods and systems, and increases the comprehensiveness of road condition detection information through multi-source information fusion.

[0077] The present invention is further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 Schematic diagram of the framework of an online road condition detection system based on multi-source information fusion in one embodiment.

[0079] Figure 2 The figure is a flow chart of an online road condition detection method based on multi-source information fusion in one embodiment.

[0080] Figure 3 Schematic diagram of an application scenario of an online road condition detection system and method based on multi-source information fusion in one embodiment.

[0081] Figure 4 Schematic diagram of the data processing module flow in one embodiment.

[0082] Figure 5 The figure is a flow chart of the scene recognition submodule in the data processing module in one embodiment.

[0083] Figure 6 Schematic diagram of the flow of the target detection and ReID feature extraction submodules in the data processing module in one embodiment.

[0084] Figure 7 Schematic diagram of the multi-source information sampling process of the target detection and ReID feature extraction submodules in the data processing module in one embodiment.

[0085] Figure 8Schematic diagram of the shallow feature extraction process of multi-time radar data in the target detection and ReID feature extraction submodule in the data processing module in one embodiment.

[0086] Figure 9 Schematic diagram of the shallow feature extraction process of video frames in the target detection and ReID feature extraction submodule in the data processing module in one embodiment.

[0087] Figure 10 Schematic diagram of the multi-source feature fusion process of the target detection and ReID feature extraction submodules in the data processing module in one embodiment.

[0088] Figure 11 Schematic diagram of the shallow feature extraction process of the target detection and ReID feature extraction submodule in the data processing module in one embodiment.

[0089] Figure 12 Schematic diagram of the deep feature extraction process of the target detection and ReID feature extraction submodule in the data processing module in one embodiment.

[0090] Figure 13 Schematic diagram of the training process of the scene recognition submodule in the data processing module in one embodiment.

[0091] Figure 14 Schematic diagram of the training process of the target detection and ReID feature extraction submodules in the data processing module in one embodiment.

[0092] Figure 15 Schematic diagram of the target tracking process in the data processing module in one embodiment.

[0093] Figure 16 Schematic diagram of an online road condition detection method and system tracking effect based on multi-source information fusion in one embodiment. DETAILED DESCRIPTION

[0094] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific implementation methods. It should be understood that these implementation methods are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, modifications of various equivalent forms of the invention made by those skilled in the art all fall within the scope defined by the claims attached to the present invention.

[0095] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0096] refer to Figure 1The following is a framework diagram of an online road condition detection system based on multi-source information fusion, provided in accordance with one embodiment of the present invention. It can be seen that the system primarily comprises a radar sensor module, a machine vision sensor module, a data processing module, a parameter configuration module, and a visualization display module. The radar sensor is used to obtain three-dimensional position and velocity information of the target object, the machine vision sensor is used to acquire video data, the data processing module is used to fuse radar signals, video signals, and radar-vision data, and to calculate and process traffic flow statistics. The parameter configuration module performs lane configuration, camera calibration, and radar calibration. The visualization display module is used to display, output, and store real-time video images, real-time detection results, and traffic flow information.

[0097] The present invention overcomes the problems of information inaccuracy and susceptibility to environmental interference caused by a single information method and system, and increases the comprehensiveness of road condition detection information through multi-source information fusion.

[0098] refer to Figure 2 This is a flowchart of an online road condition detection method based on multi-source information fusion, provided by one embodiment of the present invention. Upon startup, the method and system of the present invention first configure parameters. After configuration is complete, radar and video data are acquired through radar and video sensors. The data processing module then processes these data, and the output results are post-processed for feedback to the visualization module. The following describes each step in detail.

[0099] Step 1: Parameter configuration module configures the scene and sensor

[0100] The parameter configuration module primarily involves configuring the lane lines in the scene, initial calibration of the radar sensor, and camera calibration. All of these calibrations require manual intervention. Lane line configuration involves calibrating the lines between lanes in the displayed scene.

[0101] The purpose of camera calibration is to obtain the camera's intrinsic and extrinsic parameter matrices. These coefficients can be used to correct subsequent images captured by the camera, resulting in images with minimal distortion. In this paper, camera calibration is performed using a calibration plate. First, corner information is extracted from the calibration images. Sub-pixel corner information is then extracted from each calibration image to complete the camera calibration.

[0102] The radar sensor calibration in this invention mainly involves obtaining the extrinsic parameters between the radar sensor and the camera. In this step, the radar sensor calibration is a rough initial calibration, which converts the radar coordinates into a Cartesian coordinate system. Similarly, the corresponding points in the radar Cartesian coordinate system and the image coordinate system are manually found. The number of point pairs found needs to be greater than four, and then the perspective transformation matrix between the two coordinate systems is calculated. At this point, the preliminary calibration of the radar data is completed.

[0103] In addition, the present invention performs synchronization processing of video data and radar data, and the synchronization frequency of the two is N frames / second. Specifically, the synchronization data are mutually calculated using the following formula:

[0104]

[0105] Where T 雷达 is the radar update time, N is the frequency of synchronization between the two, and INT represents the rounding operation. 雷达 The value is 70, and the calculated N value is 14.

[0106] The above information is stored in the method and system and visualized into the image coordinate system.

[0107] Step 2: Acquisition of radar data and video data

[0108] The application scenario of this invention refers to Figure 3 ,Radar sensors and video sensors are placed at traffic checkpoints to obtain road data in real time.

[0109] The present invention uses a millimeter-wave radar to collect data, which is used to obtain the coordinate data of the road target position relative to the ground, referred to as the target ground coordinates, and transmits the data to the data processing module. The present invention uses a monocular camera to collect data on the road scene and transmit the data to the data processing module.

[0110] The data acquisition module and data processing module in the present invention can transmit data via CSI, USB, and Ethernet interfaces. The millimeter-wave radar uses a 77GHz millimeter-wave radar and supports 24GHz millimeter-wave radar. The data acquisition module and data processing module support data transmission via RS485, RS232, SPI, UART, CAN, and Ethernet interfaces. The data processing module's external interface supports data transmission via RS485, RS232, SPI, UART, CAN, and Ethernet interfaces.

[0111] Step 3: The data processing module performs fusion processing on the sensor data

[0112] In the present invention, the data processing module adopts an edge computing module of model JETSON TX2, which continuously obtains video and radar data and synchronizes the data.

[0113] The data processing module is the key module of the invention. Figure 4The data processing module mainly processes the sensor data fusion, including the following modules: scene recognition submodule, object detection and ReID feature extraction submodule. The specific submodules and subprocesses of each process are described in detail below.

[0114] The main purpose of the scene recognition submodule in this invention is to evaluate the video quality in the scene. If the video quality is normal, it will enter the subsequent logic. If the video quality is abnormal, it will directly feedback to the visualization module and record the fault point to facilitate maintenance by construction personnel. Figure 5 The scene recognition submodule is implemented based on a deep neural network. Specifically, the video frames are preprocessed by first transforming the image size to 64x64, then subtracting the mean of the ImageNet dataset and dividing it by the variance of the ImageNet dataset to obtain a normalized distribution. The scene recognition submodule in this invention is implemented based on ResNet-9, a deep neural network with four stages. Finally, through a linear layer, it outputs the probability value of the normal / abnormal category. In actual deployment, if the normal probability value is greater than or equal to 0.5, the next step is to proceed to the target detection and ReID feature extraction submodule.

[0115] The target detection and ReID feature extraction submodules in this invention refer to Figure 6 The main function of this module is to fuse radar data with video data to generate a target detection frame and the target's ReID features for target tracking. Compared with the previous method of extracting target ReID features only through image information, the ReID features generated by fusing radar data have richer semantics. In a single image scene, it is difficult to obtain the depth of the target in the real world and the actual speed of the target through static textures. The depth of the target in the real world and the actual speed of the target are very distinguishable. Therefore, the ReID features generated by fusing radar data can obtain more accurate matching and improve the accuracy of multi-target tracking. Vision-based target detection often has many missed detections and false detections. When the actual depth and speed of the target are obtained, this situation can be avoided to a large extent. Therefore, by fusing radar data, the accuracy of target detection can also be indirectly improved.

[0116] Through experimental comparison and data verification, the present invention found that radar data at a single moment often has errors, so the present invention uses radar data at multiple moments to fuse with single-frame image data. Figure 7 After synchronizing the sensor data, the two data can be aligned in the time dimension. Consider three moments T n-1 、T n 、T n+1 Three moments, the present invention uses T n The video frame at time T n-1、T n 、T n+1 The radar data at three moments are used as the input for fusion, which alleviates the noise problem of the radar data and also considers that radar data at multiple moments can increase the recognition of the target. The sampling set is shown in the following formula:

[0117] S={F n R n-1 R n R n+1}

[0118] The radar raw data collected in this invention is the polar coordinate value in the radar coordinate system. In order to match it with the video data in space, this invention first converts the polar coordinates into Cartesian coordinates. However, the two are not in the same coordinate system, and a perspective transformation model is required to calibrate the radar data to the image coordinate system. Therefore, the two cannot be directly fused. This invention extracts shallow features R from multi-time radar data and single-frame video data respectively. s and F s .

[0119] Extracting shallow features from multi-time radar data s The process, reference Figure 8 , radar data at a single moment contains 4 attribute values Where X represents the horizontal coordinate in the radar coordinate system, Y represents the vertical coordinate in the radar coordinate system, and V x Indicates the velocity in the horizontal direction, V y Indicates the speed in the ordinate direction. Therefore, the radar data at a single moment can be divided into two channels, channel 1 represents the speed in the X direction, and channel 2 represents the speed in the Y direction. The radar data at three moments are superimposed in the channel dimension to obtain the 6-channel multi-moment radar data R input First, the plane of the radar coordinate system is transformed into a size of 224x224, and then the maximum value is found for normalization. Then it is input into the convolution layer with a convolution kernel size of 7x7 and a step size of 2. The output result is processed by BatchNormalization and ReLu, and finally the shallow feature R of the multi-time radar data with a channel number of 64 is obtained. s , the calculation process is as follows:

[0120] R s =relu(bn(Conv(R input )))

[0121] Extract shallow features F from single frame video data s The process, reference Figure 9 , a single frame of video data includes 3 channels, denoted as F inputSimilarly, the present invention first transforms the image data to 224x224 size, divides each channel by 255 to obtain the normalized distribution; then, it is input into the convolution layer with a convolution kernel size of 7x7 and a step size of 2, and the output result is processed by BatchNormalization and ReLu, finally obtaining the single-frame video shallow feature F with 64 channels. s , the calculation process is as follows:

[0122] F s =relu(bn(Conv(F input )))

[0123] Then, the shallow features R of multi-time radar data are s Compared with the shallow feature F of single frame video s For fusion, refer to Figure 10 The present invention converts the shallow features R of multi-time radar data into s Compared with the shallow feature F of single frame video s The weighted summation is used for fusion. Here, the coordinate alignment of the two must be considered, which leads to the next question.

[0124] The present invention performs a priori processing on the fused features, so that the network can autonomously learn the perspective transformation model between the two coordinate systems. Figure 11 The present invention performs 3-layer convolution and BatchNormalization on the fused features. The convolution kernel size of the 3-layer convolution is 1x1, and the number of output channels is 32, 16, and 9 respectively. Finally, the spatial mean is taken to output 9 regression values ​​of the perspective transformation matrix. By introducing a priori and training the regression value of the perspective transformation matrix at the same time, the present invention ensures that the network can spontaneously align the multi-time radar shallow features R s Compared with the shallow feature F of single frame video s .

[0125] After fusing multi-time radar data features with single-frame video features, refer to Figure 12 Based on ResNet-18, the fused features are respectively passed through the second, third and fourth stages of ResNet-18 to obtain deep features; the deep features are then input into the target center point sub-branch heads center , target size regression sub-branch heads wh , target center point error sub-branch heads offset , ReID sub-branch heads reid The network structure of these branches in the present invention is implemented in the form of CenterNet. After decoding operation decode, the predicted target detection box S can be obtained. bboxAnd the feature Feature of the target center point reid The present invention has been verified and found that when the feature dimension is 128, the tracking effect is better. The center point of the detection box is output as Center bbox , the detection frame width and height are output as WH bbox , the error of the center point of the detection frame is Offset bbox , the calculation process is as follows:

[0126] Center bbox =heads center (Stage4(Stage3(Stage2(a×F s +b×R s ))))

[0127] WH bbox =heads wh (Stage4(Stage3(Stage2(a×F s +b×R s ))))

[0128] Offset bbox =heads offset (Stage4(Stage3(Stage2(a×F s +b×R s ))))

[0129] Feature reid =heads reid (Stage4(Stage3(Stage2(a×F s +b×R s ))))

[0130] S bbox =decode(Center bbox ,WH bbox ,Offset bbox )

[0131] The multi-target tracking process of the present invention refers to Figure 15 Specifically, the output target detection box and ReID features are respectively used to calculate the cost matrix with the historical features maintained by the tracking sequence and the bbox predicted by the Kalman filter. The ReID features are measured with the historical features of each ID using the cosine distance, and the output target detection box and the bbox predicted by the Kalman filter are measured using 1-IoU. Finally, the two distance cost matrices are weighted and summed to obtain the overall cost matrix. Hungarian matching is performed based on this distance cost matrix to obtain the tracking ID of each target in the current frame. At the same time, the historical features of the tracked ID are updated.

[0132] For the training process of the scene recognition submodule, refer to Figure 13 The image data from the prepared training samples is preprocessed and input into the model. The obtained results are compared with the true classification labels of the samples to calculate the mean square error (MSE) loss. The Adam optimizer is used to optimize the parameters of the deep network. Training is stopped when the number of iterations is reached. In this invention, the number of training rounds is set to 25, the initial learning rate is set to 0.01, and an annealing strategy is used, with the learning rate multiplied by 0.1 after every 10 rounds.

[0133] For the training process of target detection and ReID feature extraction submodules, refer to Figure 14 , the training sample contains two label values, one is the perspective transformation matrix of radar data and video data, and the other is the annotation of target detection and ID label; it should be noted that the target detection in the present invention includes multiple categories, but the tracked ID is only labeled with the vehicle category, so during the training process, it is necessary to use a mask to filter the ReID branch Loss; if the category information of the target detection is not the vehicle category, its ReID feature is unreliable; the present invention inputs the single-frame video data and multi-time radar data in the sample into the model, calculates the overall loss value through each sub-branch, and then uses the SGD optimizer for optimization. In the present invention, the number of training rounds is set to 50, the initial learning rate is set to 0.01, and the cosine descent learning strategy is used for training.

[0134] Step 4: Post-processing of the output results of the data processing module

[0135] By statistically tracking the total amount of ID data, the present invention can estimate traffic volume during a specific time period. This information is then synchronized with the traffic light, allowing the signal phase at the intersection to be adjusted based on actual conditions. Simultaneously, the present invention also obtains information about the intersection from the signal light. If the light is red, the relative position of the target detection output and the lane marking is determined. If the light exceeds the lane line, it is considered a lane crossing or red light running violation. The vehicle is then captured and stored in a storage unit.

[0136] At the same time, the solved perspective transformation matrix is ​​updated to the parameter configuration module.

[0137] Step 5: Post-processing module feeds back to visualization module

[0138] refer to Figure 16 , is the visualization result of the tracking of the present invention, and the corresponding information can be fed back to the visualization module according to actual needs. The perspective transformation matrix used to transform the radar data coordinates into image coordinates in the visualization module of the present invention is retrieved from the parameter configuration module.

[0139] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the foregoing embodiments. The foregoing embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. An online road condition detection system based on information fusion of multiple sensors, characterized in that: The system includes a radar sensor module, a video sensor module, a data processing module, a parameter configuration module and a visual display module; The radar sensor module is used to measure the position, speed information and sampling period of the target to be measured; The video sensor module is used to obtain image data of various scenes; The data processing module is used to fuse the information of the radar sensor module and the video sensor module, and calculate the traffic flow information; The parameter configuration module is used to configure parameters such as lane configuration, radar calibration and camera calibration; The visual display module is used to realize the output, storage and display of data results and information of each module; The data processing module comprises: The scene recognition submodule is used to assess the video quality in the scene based on a deep neural network; The target detection and ReID feature extraction submodule is used to fuse radar data with video data to generate a target detection frame and the ReID feature of the target for target tracking; The post-processing submodule is used to obtain traffic flow information and illegal behaviors based on the tracking results and update the parameter configuration module; The target detection and ReID feature extraction submodule includes: The multi-source input information acquisition submodule is used to collect radar data at multiple times and single-frame image data. The sampling set is shown in the following formula: S={F n R n-1 R n R n+1 } Among them, F n T n The video frame at time R n-1 , R n , R n+1 T n-1 , T n , T n+1 Radar data at three moments; The first shallow feature extraction submodule is used to extract the shallow features R of multi-time radar data. s ; The second shallow feature extraction submodule is used to extract the shallow features F of single-frame image data. s ; Multi-source feature fusion submodule is used to fusion shallow feature R of multi-time radar data s Compared with the shallow feature F of a single frame video s to integrate; The third shallow feature extraction submodule is used to extract shallow features of the features fused by the multi-source feature fusion submodule, obtain perspective transformation matrix regression values, and regress a perspective transformation matrix model from the radar coordinate system to the image coordinate system; The deep feature extraction submodule is used to extract the deep features of the features fused by the multi-source feature fusion submodule, and regress the center position of the target in the image coordinate system, the height and width of the target in the image coordinate system, and the ReID features of the target.

2. The online road condition detection system based on multiple sensor information fusion according to claim 1 is characterized in that: The scene recognition submodule includes: The preprocessing submodule is used to preprocess the video frame image. The preprocessing includes: firstly transforming the image size to 64x64, then subtracting the mean of the image data set and dividing by the variance of the image data set to obtain a normalized distribution; The recognition submodule is used to pass the preprocessed image through the four stages and linear layer of the ResNet-9 deep neural network in sequence, and output the probability value of the video quality being normal / abnormal. If the normal probability value is greater than or equal to 0.5, the target detection and ReID feature extraction submodule is executed.

3. The online road condition detection system based on multiple sensor information fusion according to claim 1 is characterized in that: The first shallow feature extraction submodule extracts shallow features R of multi-time radar data. s The specific process includes: The radar data at a single moment contains 4 attribute values Where X represents the horizontal coordinate in the radar coordinate system, Y represents the vertical coordinate in the radar coordinate system, and V x Represents the velocity in the horizontal direction, V y Indicates the speed in the ordinate direction; (1) The radar data at a single moment is divided into two channels, channel 1 represents the speed in the X direction, and channel 2 represents the speed in the Y direction; (2) In the channel dimension, the radar data at three times are superimposed to obtain the 6-channel multi-time radar data R input ; (3) Transform the plane of the radar coordinate system into a size of 224x224, and then find the maximum value for normalization; (4) The data is input into a convolutional layer with a convolution kernel size of 7x7 and a step size of 2. The output is then batch normalized and processed by ReLu. Finally, the shallow feature R of multi-time radar data with 64 channels is obtained. s , the calculation formula is: R s =relu(bn(Conv(R input ))); The second shallow feature extraction submodule extracts shallow features F of single frame image data. s The specific process includes: A single frame of video data includes three channels, denoted as F input ; (1) Transform the image data to 224x224 size, and divide each channel by 255 to obtain the normalized distribution; (2) The output is input into a convolutional layer with a convolution kernel size of 7x7 and a step size of 2. The output result is processed by BatchNormalization and ReLu, and finally a single-frame video shallow feature F with 64 channels is obtained. s , the calculation formula is: F s =relu(bn(Conv(F input )))。 4. The online road condition detection system based on multiple sensor information fusion according to claim 1 is characterized in that: The third shallow feature extraction submodule is used to extract shallow features of the features fused by the multi-source feature fusion submodule, obtain perspective transformation matrix regression values, and regress the perspective transformation matrix model from the radar coordinate system to the image coordinate system. The specific process includes: The fused features are subjected to three layers of convolution operations and BatchNormalization operations. The convolution kernel size of the three layers of convolution operations is 1x1, and the number of output channels is 32, 16, and 9 respectively. Finally, the mean is taken in the spatial dimension, and 9 regression values ​​of the perspective transformation matrix are output.

5. The online road condition detection system based on multiple sensor information fusion according to claim 1 is characterized in that: The deep feature extraction submodule is used to extract the deep features of the features fused by the multi-source feature fusion submodule, and regress the center position of the target in the image coordinate system, the height and width of the target in the image coordinate system, and the ReID features of the target. The specific process includes: (1) The fused features are passed through the second, third, and fourth stages of the ResNet-18 deep neural network to obtain deep features; (2) Input the deep features into the target center point sub-branch heads center , target size regression sub-branch heads wh , target center point error sub-branch heads offset 、ReID sub-branch heads reid , after decoding operation, the predicted target detection box S is obtained bbox And the feature Feature at the center point of the target reid ; The network structure of the above sub-branch is in CenterNet form; The center point of the detection box is output as Center bbox , the detection box width and height are output as WH bbox , the error of the center point of the detection frame is Offset bbox , calculated as: Center bbox =heads center (Stage4(Stage3(Stage2(a×F s +b×R s )))) WH bbox =heads wh (Stage4(Stage3(Stage2(a×F s +b×R s )))) Offset bbox =heads offset (Stage4(Stage3(Stage2(a×F s +b×R s )))) Feature reid =heads reid (Stage4(Stage3(Stage2(a×F s +b×R s )))) S bbox =decode(Center bbox ,WH bbox ,Offset bbox ) In the formula, a and b are the shallow features R obtained by fitting training. s and F s The corresponding weight coefficient.

6. The online road condition detection system based on multiple sensor information fusion according to claim 1 is characterized in that: The target detection and ReID feature extraction submodule performs target tracking processing according to the generated target detection frame and the ReID feature of the target, specifically including: (1) The target detection box and ReID features are respectively calculated with the historical features maintained by the tracking sequence and the detection box predicted by the Kalman filter for distance cost matrix calculation; wherein the ReID feature is measured with the historical features of each ID using the cosine distance, and the target detection box is measured with the detection box predicted by the Kalman filter using 1-IoU; (2) The weighted sum of the two distance cost matrices is used to obtain the overall distance cost matrix; (3) performing Hungarian matching according to the overall distance cost matrix to obtain the tracking ID of each target in the current frame; (4) Update the historical features and tracking sequence of the tracked ID.

7. The online road condition detection system based on multiple sensor information fusion according to claim 6 is characterized in that: The traffic flow information includes the position and movement status information of pedestrians, motor vehicles and non-motor vehicles, as well as traffic volume information; the illegal behaviors include crossing lane lines and running red lights; the updating parameter configuration module includes updating the solved perspective transformation matrix into the parameter configuration module.

8. An online road condition detection method based on information fusion of multiple sensors based on the system according to any one of claims 1 to 7, characterized in that: The method comprises the following steps: Step 1, configuring the scene and the sensor through a parameter configuration module; the sensor includes a radar sensor and a video sensor; Step 2, obtaining radar data and video data of the road; Step 3: Use the data processing module to fuse the sensor data to generate a target detection frame and the ReID feature of the target for target tracking; Step 4: Using the data processing module to obtain traffic flow information and illegal behaviors according to the tracking results, and updating the parameter configuration module; Step 5: Feedback the required information to the visualization display module.

Images