Full-time global multi-vehicle tracking method based on multi-modal vehicle-road collaborative awareness

Abstract

The invention discloses a full-time global multi-vehicle tracking method based on multi-modal vehicle-road collaborative awareness. The method comprises the following steps: 1) acquiring multi-modal road observation image data; 2) extracting image features; 3) feature fusion detection; (4) when the target vehicle travels to any road section, full-time global multi-vehicle-road cooperative calibration is carried out by utilizing a three-dimensional detection result fused by the road side and the vehicle end; 5) performing full-time global multi-target motion state and observation prediction; the method comprises the steps of (1) carrying out data association on a target, (2) carrying out data association on the target, (3) carrying out multi-target observation data fusion, (4) carrying out weight adjustment on multi-source observation according to an observation noise covariance matrix after data association is carried out, and (5) carrying out probability fusion by combining model estimation of two motion models to obtain the final state of the target.

Description

Technical Field

[0001] The present invention relates to intelligent driving vehicle-road collaborative technology, and in particular to a full-time and full-domain multi-vehicle tracking method based on multimodal vehicle-road collaborative perception. Background Art

[0002] Deep learning-based target detection and tracking algorithms have been widely used in multi-vehicle tracking. Furthermore, multi-sensor data fusion technology, real-time communication protocol optimization, and distributed computing architectures provide strong support for the implementation of full-time, full-domain vehicle-infrastructure cooperative systems. In terms of applications, full-time, full-domain multi-vehicle tracking holds broad promise in intelligent traffic management, autonomous driving, and fleet collaboration. By improving target tracking accuracy and reliability, full-time, full-domain vehicle-infrastructure cooperative systems can not only significantly enhance traffic safety and efficiency but also lay the foundation for the development of future smart cities. In full-time, full-domain vehicle-infrastructure cooperative systems, multimodal perception data fusion technology integrates data from different sensor types (such as cameras, lidar, ultrasonic sensors, and GPS) to obtain more comprehensive and accurate environmental perception information, further enhancing the performance of multi-vehicle target tracking. This full-time, full-domain multi-vehicle tracking approach ensures continuous monitoring and efficient collaboration under various conditions. Although significant progress has been made in vehicle-centric 3D object detection, multi-vehicle target tracking during road driving is still hampered by blind spot occlusion and long-distance perception capabilities, due to the limited field of view of sensors mounted on top of vehicles and the insufficient observation range of sensors installed on the roadside. This makes it impossible to achieve continuous, uninterrupted, and effective tracking across the entire time and area without blind spots. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a full-time and full-domain multi-vehicle tracking method based on multimodal vehicle-road collaborative perception in response to the defects in the existing technology.

[0004] The technical solution adopted by the present invention to solve the technical problem is: a full-time and full-domain multi-vehicle tracking method based on multimodal vehicle-road cooperative perception, comprising the following steps: 1) Acquire multimodal road observation image data, including the target vehicle's vehicle-side observation image, other vehicles' vehicle-side observation images in the area during the current period, and roadside observation images in the area during the current period; 2) Image feature extraction; Extract local features and context features from the acquired road scene to obtain semantic segmentation results of environmental targets; 3) Feature fusion detection; Combined with the image semantic segmentation results, 3D target detection is performed; 4) When the target vehicle reaches any road section, full-time and full-area multi-vehicle-road collaborative calibration is performed using the fused 3D detection results from the roadside and vehicle sides. This includes: using the fused 3D target detection results from the roadside and vehicle sides to find the largest set of common detection frames in the intersection detection area, and obtaining the relative position relationship of the target on the vehicle side and the roadside side. 5) All-time and all-domain multi-target motion status and observation prediction; Based on the position and motion parameters obtained by 3D target detection, full-time and full-domain multi-target motion state prediction and observation prediction are carried out; The motion models are set as: uniform velocity motion model CV and constant turning rate and velocity model CTRV, and the unscented Kalman filter is used to predict the target state for both motion modes; 6) Fusion of multi-target observation data; 7) After data association, the weights of multi-source observations are adjusted based on the observation noise covariance matrix; 8) Combine the model estimates of the two motion models and perform probabilistic fusion to obtain the final state of the target.

[0005] According to the above scheme, in step 2), the CGNet network is used and up-sampled to obtain the semantic segmentation result of the environmental target, wherein the CGNet network structure includes three Convolution block, M CG convolution blocks, N CG convolution blocks and convolution.

[0006] According to the above scheme, in step 3), 3D target detection is performed in combination with the image semantic segmentation results, as follows: 3.1) Convert the captured image into the corresponding depth image, and redraw and enhance the depth image based on the segmentation target obtained by CGNet to obtain a new range depth image; 3.2) Use the fully convolutional semantic segmentation network (FCN) to further segment the 2D depth image, select a downsampling rate of 32 times, and use Darknet53 as the backbone network for feature extraction; 3.3) Upsampling the extracted feature maps to restore the image resolution, adding skip connections between different levels of output strides through convolutional layer operations and adding them to the corresponding feature volumes to restore the lost edge information; 3.4) Obtaining a (u, v) pair for each point in the processed image of step 3.3), where u represents the horizontal coordinate and v represents the vertical coordinate; using the image coordinates corresponding to each point to index the depth image and generate a semantic label for each point present in the input point cloud; 3.5) Use KNN search to find the point set for each point in the point cloud based on a certain threshold, further subdividing the semantic categories. At the same time, set the clustering threshold to perform Euclidean clustering and generate a 3D detection box.

[0007] According to the above solution, in step 4), the relative position relationship of the target at the vehicle end and the roadside end is obtained as follows: 4.1) Use 3D IoU to find the corresponding associated targets. The formula is as follows:

[0008] Where m and n represent the number of 3D detection frames detected by roadside perception and vehicle-side perception, respectively; and are the i-th and j-th 3D detection frames detected by roadside perception and vehicle-side perception respectively; It is a function that calculates the volume overlap of 3D detection boxes; represents the volume of the 3D detection box, and Represent the intersection and union of 3D detection boxes respectively; Each 3D detection box is considered as a node, and association and matching are performed through linear assignment based on the relationship matrix between corresponding vertices and corresponding edges. The edge represents the positional relationship and overlap between detection boxes, and the edge association degree represents the size of the overlapping part or the distance of the detection boxes in space. Correlation Matrix as follows:

[0009] in, Considering the nodes and all the edges associated with them, represents the set of all edges connected to node i, Represents the association between edges m and n; The relative position relationship between the target at the vehicle end and the roadside end is obtained according to the Hungarian matching method.

[0010] According to the above solution, in step 5), the unscented Kalman filter is used to perform target state prediction for both motion modes, including state prediction and observation prediction.

[0011] According to the above scheme, in step 6), the multi-target observation data is fused as follows: 6.1) Preliminary fusion of multi-source observations After calculating the state prediction results of the multiple targets sensed at the previous moment, data association is performed with the current multi-terminal actual observation results; Preliminary fusion of multi-source actual observations before data association as follows: ; in, is the weight value of each observation; 6.2) Association Matching By building the Euclidean distance between the center of mass of the target detection box and the prediction box, it is determined whether there is a potential matching relationship. If the match is not successful, a new missing target Tracks set is established, and the state is updated and the next prediction is made. If there is a matching relationship, the corresponding matching target is further screened through the three-dimensional volume intersection-over-union ratio of 3DIoU. For targets that are not successfully matched, they are placed in the disappeared target Tracks set.

[0012] According to the above scheme, in step 7), the observation noise covariance matrix is ​​constructed as follows: Construct the initialized observation noise covariance, then its observation noise covariance matrix is:

[0013] For the first frame of full-time global multi-target tracking, the observation noise covariance of each observation end is ; As tracking continues, the observation noise covariance is adjusted by the residual difference between the actual observation and the predicted observation. The formula is as follows:

[0014] in, is the residual at time t-1, is the adjustment coefficient, and its value range is ; The residual covariance matrix is:

[0015] Where H is the observation matrix, is the state prediction covariance.

[0016] The beneficial effects produced by the present invention are: 1. By fusing multimodal sensor data and collaboratively calibrating multi-vehicle and multi-roadside perception data for joint perception, effective perception for all-time, all-domain, and multi-target detection is achieved. 2. By combining the uniform motion model CV and the constant turn rate and velocity model CTRV, the unscented Kalman filter is used to predict the states of multiple targets in two motion conditions respectively, and finally the final state update is achieved by weighting the likelihood probability weights of each model. This can obtain a more adaptable and effective estimation model and more accurately estimate the motion of the target.

[0017] 3. The dynamic change formula of the observation noise covariance is constructed based on the observation noise covariance of each observation, and then the weights of the observations are assigned by averaging the weights of the observation noise covariances, thereby giving greater weights to more reasonable observations and improving the accuracy and robustness of system state updates. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which: Figure 1 is a flow chart of a method according to an embodiment of the present invention; Figure 2 Schematic diagram of the modal sensor data fusion principle of an embodiment of the present invention; Figure 3 2 is a schematic diagram of the CGNet network structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0020] like Figure 1 As shown, a full-time and full-area multi-vehicle tracking method based on multimodal vehicle-road cooperative perception includes the following steps: 1) Acquire multimodal road observation image data, including the target vehicle's vehicle-side observation image, other vehicles' vehicle-side observation images in the area during the current period, and roadside observation images in the area during the current period; In a full-time, full-domain vehicle-road cooperative system, multimodal perception data will come from different types of sensors (such as cameras, lidar, and ultrasonic sensors); 2) Image feature extraction; Extract local features and context features from the acquired road scene to obtain semantic segmentation results of environmental targets; In this embodiment, the CGNet network is used and up-sampled to obtain the semantic segmentation results of the environment target, wherein the CGNet network structure includes three Convolution block, M CG convolution blocks, N CG convolution blocks and Convolution. The entire CGNet structure is as follows Figure 3 As shown; Each CG module consists of First, the real-time road image feature map is Convolution, and then input to the local feature extractor and context feature extractor Among them and Both use channel convolution.

[0021] pass For the joint feature extraction operation after local feature extraction and context feature extraction, batch normalization BN and parameterized activation function PReLU operation are further performed Finally The extractor extracts global context features, performs global average pooling and multi-layer perceptron on the features, and then multiplies the obtained weights and input elements.

[0022] Each CG module is connected using two different residual learning processes, one is local residual learning LRL, and the other is global residual learning GRL.

[0023] 3) Feature fusion detection; This embodiment uses the RangeNet++ network combined with lightweight image semantic segmentation results for enhancement to achieve accurate 3D target detection and semantic segmentation of road scenes. Combined with the image semantic segmentation results, 3D target detection is performed, such as Figure 2 , as follows: 3.1) Convert the captured image into the corresponding depth image, and redraw and enhance the depth image based on the segmentation target obtained by CGNet to obtain a new range depth image; 3.2) Use the fully convolutional semantic segmentation network (FCN) to further segment the 2D depth image, select a downsampling rate of 32 times, and use Darknet53 as the backbone network for feature extraction; 3.3) Upsampling the extracted feature maps to restore the image resolution, adding skip connections between different levels of output strides through convolutional layer operations and adding them to the corresponding feature volumes to restore the lost edge information; 3.4) Obtaining a (u, v) pair for each point in the processed image of step 3.3), where u represents the horizontal coordinate and v represents the vertical coordinate; using the image coordinates corresponding to each point to index the depth image and generate a semantic label for each point present in the input point cloud; 3.5) Use KNN search to find the point set for each point in the point cloud based on a certain threshold, further subdividing the semantic categories. At the same time, set the clustering threshold to perform Euclidean clustering and generate a 3D detection box.

[0024] 4) When the target vehicle reaches any road section, full-time and full-area multi-vehicle-road collaborative calibration is performed using the fused 3D detection results from the roadside and vehicle terminals. This includes: using the fused 3D target detection results from the roadside and vehicle terminals to find the largest set of common detection frames in the intersection detection area and obtain the relative position relationship between the vehicle terminal and the roadside terminal; 4.1) Use 3D IoU to find the corresponding associated targets. The formula is as follows:

[0025] Where m and n represent the number of 3D detection frames detected by roadside perception and vehicle-side perception, respectively; and are the i-th and j-th 3D detection frames detected by roadside perception and vehicle-side perception respectively; It is a function that calculates the volume overlap of 3D detection boxes; represents the volume of the 3D detection box, and Represent the intersection and union of 3D detection boxes respectively; 4.2) Treat each 3D detection box as a node and perform association and matching via linear assignment based on the relationship matrix between corresponding vertices and edges. Edges represent the positional relationship and overlap between detection boxes, and the edge association degree indicates the size of the overlap or proximity of the detection boxes in space. Correlation Matrix as follows:

[0026] in, Considering the nodes and all the edges associated with them, represents the set of all edges connected to node i, Represents the association between edges m and n; 4.3) Find the relative position relationship between the target on the vehicle side and the road side according to the Hungarian matching method.

[0027] 5) Full-time and full-area multi-target motion status and observation prediction Based on the position and motion parameters obtained by 3D target detection, full-time and full-domain multi-target motion state prediction and observation prediction are carried out; The motion models are set as: uniform velocity model (CV) and constant turn rate and velocity model (CTRV), and the unscented Kalman filter is used to predict the target state for both motion modes; First, the state variables are described as , respectively, representing the target Moment Axis coordinates, Axis coordinates, Axis coordinates, speed, yaw angle and yaw angular velocity. When the CV motion state changes to the CTRV state.

[0028] The general formula of its state equation is:

[0029] Due to the existence of noise, the model cannot accurately predict the target state, and the process noise term ,in is the radial acceleration noise, is the yaw acceleration noise, which obeys the zero-mean Gaussian distribution.

[0030] Therefore, combined with process noise, when the motion state is CV, its complete state equation is as follows:

[0031] When the motion state is CTRV, the complete state equation is:

[0032] At the same time, the process noise covariance is as follows:

[0033] Where, is the standard deviation of noise acceleration, is the standard deviation of the noise angular acceleration.

[0034] Unscented Kalman filtering is used to perform target state prediction for both motion modes, including state prediction and observation prediction.

[0035] 5.1) State Prediction First, we need to give the initial probability distribution of the state quantity, as well as the mean and covariance, and calculate 2n+1 pairs The posterior probability distribution sampling point of the state quantity at the moment is as follows:

[0036] Where, express The i-th sampling point of the posterior state probability distribution at time , represents the mean of the posterior state at that moment; According to the relevant parameters set, the mean and covariance corresponding weights of all sigma points, i.e. sampling points, in the unscented transform are calculated as follows:

[0037] in, and Represent the state prediction mean weight and covariance weight respectively, is a constant, where Characterizes the degree to which sampling points other than the mean point deviate from the mean; Calculate the one-step prediction of 2n+1 sigma points, substitute each sampling point into the state equation of each motion mode, and obtain the predicted value of each sigma point under each motion model. , that is, each point in t Prior value of the moment; One-step prediction result of the overall state of the system It is obtained by weighted summation of the one-step prediction values ​​of each sigma point, and the system state covariance is calculated at the same time , as follows: ; in, express t Prior mean of the system state at the moment; 5.2) Observation and Prediction Apply UT transformation again to the system one-step prediction result to generate a new sigma point set; Substitute the newly generated sigma point set into the observation prediction equation, where the observation prediction equation is as follows:

[0038] To predict the observed value, for the state description variable And the three-dimensional detection results of multimodal sensor fusion perception Measurement matrix of relative transformation relationship , which are as follows:

[0039] The weighted sum of each predicted observation value is used to obtain the mean of the system predicted observation value and its covariance as follows:

[0040] is the mean of the predicted observations; Cross-covariance between the simultaneous state predictions and the predicted observations as follows:

[0041] The cross-covariance and the covariance of the predicted observations are used to calculate t Momentary Kalman gain as follows: .

[0042] 6) Fusion of multi-target observation data; 6.1) Preliminary fusion of multi-source observations After calculating the state prediction results of the multiple targets sensed at the previous moment, data association is performed with the current multi-terminal actual observation results; For multiple observation terminals, first perform preliminary observation fusion weight assignment, and then calculate the mean of the system prediction observation value. and actual observation data at each end Observation residuals ( i Indicates the i observations), and then use the inverse square of the residual value as the weight corresponding to the observation ,as follows:

[0043] Finally, normalize the observation weights to get the final weight ;

[0044] Preliminary fusion of multi-source actual observations before data association as follows:

[0045] 6.2) Association Matching The cost matrix of multi-target corresponding matching is constructed by constructing the centroid Euclidean distance and 3DIoU between the target detection box and the prediction box, and the Hungarian algorithm is used to optimize the matching.

[0046] The target centroid is used to calculate the target Euclidean distance between two frames. The calculation formula is as follows:

[0047] in Indicates the current frame target With the previous frame target The Euclidean distance between predictions.

[0048] The size of the cost matrix is ,in and The number of targets detected in the previous and next frames are respectively, and then they are put into the Hungarian network for multi-target matching.

[0049] At the same time, a threshold is set. If the Euclidean distance between the targets is less than the threshold, it means that there is a potential matching relationship between the targets in the two frames. If the match is not successful, a new missing target Tracks set is established, and the state is updated and the next prediction is made. If there is a matching relationship, the corresponding matching target is further screened through the three-dimensional volume intersection-over-union ratio of 3DIoU; the calculation method is as follows:

[0050] Since the target vehicle between two consecutive frames on the road can be regarded as translating and rotating parallel to the ground, the z-axis coordinate can be assumed to be unchanged. Therefore, for the targets A and B corresponding to the previous and next frames, the overlapping volume can be obtained by calculating the area of ​​the intersecting polygons projected onto the ground and multiplying it by the height.

[0051] The polygon area is calculated using the Gaussian area formula as follows:

[0052] The 3DIoU values ​​between all tracked targets and all detection results are used to construct the 3DIoU matrix ,in Indicates the Tracking target, Indicates the The cost matrix is ​​constructed as follows:

[0053] When the 3DIoU value is larger, the corresponding cost is smaller. For matches with extremely small 3DIoU values, the cost is set to infinity. For targets that are not successfully matched, put them into the disappeared target Tracks collection.

[0054] 7) Adjust the weights of multi-source observations based on the observation noise covariance matrix; 7.1) Construct the initial observation noise covariance, then its observation noise covariance matrix is:

[0055] For the first frame of full-time global multi-target tracking, the observation noise covariance of each observation end is ; As tracking continues, the observation noise covariance is adjusted by the residual difference between the actual observation and the predicted observation. The formula is as follows:

[0056] in, is the residual at time t-1, is the adjustment coefficient, and its value range is ; At this time, the residual covariance matrix is:

[0057] Where H is the observation matrix, is the state prediction covariance; Set the residual covariance matrix threshold. When the threshold is exceeded, the coefficient is adjusted. Increase, when the residual covariance matrix is ​​small, decrease To keep the observation noise covariance stable.

[0058] 7.2) Update of multi-source actual observation weights Assume that at present t The number of observations at a given moment is m, and the unnormalized weight of each observation is:

[0059] in For the i The determinant of the observation noise covariance matrix of the observations at the current moment.

[0060] The normalized weights of the respective observations are as follows:

[0061] 7.3) System Status Update The predicted value of the system state at the previous moment obtained by untraceable transformation , system state covariance 、The current moment Kalman gain , predicted observations , covariance of predicted observations And the actual observation after weight update can calculate the state update and covariance update of the system as follows:

[0062] 8) Combine the model estimates of the two motion models and perform probabilistic fusion to obtain the final state of the target.

[0063] Since two different motion modes are considered, two filters are maintained for each target and independent state estimation is performed for each target. Finally, the fusion estimation results of the two models are used. It is assumed that the CV motion state is model 1 and the CTRV motion state is model 2.

[0064] The transition probability matrix between models is defined as follows:

[0065] Where, Indicates the state transition probability from model 1 to model 2. Initially, the prior probability of the model needs to be set, and then the conditional probability of model 1 is calculated as follows:

[0066] express The posterior probability of model 1 at time t, It represents the prior probability of model 2 at the current moment, which is obtained by the posterior probability at the previous moment and the model transition probability.

[0067] Since each target has two filters, the posterior state and covariance of each filter at the previous moment are fused as follows:

[0068] in, express The estimation results are integrated at the moment. represents the result obtained by model 1 at the previous moment, and Represent the estimated covariance of a single model and the fusion covariance, respectively.

[0069] The model probability is updated based on the error between the actual multi-observation fusion result and the predicted observation, and the likelihood probability of each model is calculated as follows:

[0070] Where, Represents the error between the prediction and observation of model 1 Furthermore, the model probability at the current moment can be Updates as follows:

[0071] Finally, the final state and covariance are obtained by fusing the model probability at the current moment and the output results of each filter. The calculation method is as follows:

[0072] The state is used as the new tracks input for the next tracking cycle to continue iteration.

[0073] The present invention performs joint data association of targets in previous and next frames by fusing multi-source observation data, and proposes an adaptive weight transformation method for multiple observations at the same time, so that target vehicles on the road can perform reasonable, accurate and effective data matching and status updates through multi-party joint observations with different weights in all time and domain conditions, ultimately achieving stable and robust target tracking.

[0074] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all such improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A full-time and full-area multi-vehicle tracking method based on multimodal vehicle-road cooperative perception, characterized by: The following steps are involved: 1) Acquire multimodal road observation image data, including the target vehicle's vehicle-side observation image, other vehicles' vehicle-side observation images in the area during the current period, and roadside observation images in the area during the current period; 2) Image feature extraction; Extract local features and context features from the acquired road scene to obtain semantic segmentation results of environmental targets; 3) Feature fusion detection; Combined with the image semantic segmentation results, 3D target detection is performed; 4) When the target vehicle reaches any road section, full-time and full-area multi-vehicle-road collaborative calibration is performed using the fused 3D detection results from the roadside and vehicle sides. This includes: using the fused 3D target detection results from the roadside and vehicle sides to find the largest set of common detection frames in the intersection detection area, and obtaining the relative position relationship of the target on the vehicle side and the roadside side. 5) Full-time and full-area multi-target motion status and observation prediction Based on the position and motion parameters obtained by 3D target detection, full-time and full-domain multi-target motion state prediction and observation prediction are carried out; The motion models are set as: uniform velocity motion model CV and constant turning rate and velocity model CTRV, and the unscented Kalman filter is used to predict the target state for both motion modes; 6) Fusion of multi-target observation data; 7) After data association, the weights of multi-source observations are adjusted based on the observation noise covariance matrix; 8) Combine the model estimates of the two motion models and perform probabilistic fusion to obtain the final state of the target.

2. The full-time and full-area multi-vehicle tracking method based on multimodal vehicle-road cooperative perception according to claim 1 is characterized in that: In step 2), the CGNet network is used and up-sampled to obtain the semantic segmentation result of the environment target, wherein the CGNet network structure includes three Convolution block, M CG convolution blocks, N CG convolution blocks and convolution.

3. The full-time and full-area multi-vehicle tracking method based on multimodal vehicle-road cooperative perception according to claim 1 is characterized in that: In step 3), three-dimensional object detection is performed in combination with the image semantic segmentation results, as follows: 3.1) Convert the captured image into the corresponding depth image, and redraw and enhance the depth image based on the segmentation target obtained by CGNet to obtain a new range depth image; 3.2) Use the fully convolutional semantic segmentation network FCN to further perform semantic segmentation on the two-dimensional depth image; 3.3) Upsample the extracted feature map to restore the image resolution; 3.4) Obtaining a (u, v) pair for each point in the processed image of step 3.3), where u represents the horizontal coordinate and v represents the vertical coordinate; using the image coordinates corresponding to each point to index the depth image and generate a semantic label for each point present in the input point cloud; 3.5) Use KNN search to find the point set for each point in the point cloud based on a certain threshold, further subdividing the semantic categories. At the same time, set the clustering threshold to perform Euclidean clustering and generate a 3D detection box.

4. The full-time and full-area multi-vehicle tracking method based on multimodal vehicle-road cooperative perception according to claim 1 is characterized in that: In step 4), the relative position relationship of the target at the vehicle end and the roadside end is obtained as follows: 4.1) Use 3D IoU to find the corresponding associated targets. The formula is as follows: Where m and n represent the number of 3D detection frames detected by roadside perception and vehicle-side perception, respectively; and are the i-th and j-th 3D detection frames detected by roadside perception and vehicle-side perception respectively; It is a function that calculates the volume overlap of 3D detection boxes; represents the volume of the 3D detection box, and Represent the intersection and union of 3D detection boxes respectively; Each 3D detection box is considered as a node, and association and matching are performed through linear assignment based on the relationship matrix between corresponding vertices and corresponding edges. The edge represents the positional relationship and overlap between detection boxes, and the edge association degree represents the size of the overlapping part or the distance of the detection boxes in space. Correlation Matrix as follows: in, Considering the nodes and all the edges associated with them, represents the set of all edges connected to node i, Represents the association between edges m and n; The relative position relationship between the target at the vehicle end and the roadside end is obtained according to the Hungarian matching method.

5. The full-time and full-area multi-vehicle tracking method based on multimodal vehicle-road cooperative perception according to claim 1 is characterized in that: In step 5), an unscented Kalman filter is used to perform target state prediction for both motion modes, including state prediction and observation prediction.

6. The full-time and full-area multi-vehicle tracking method based on multimodal vehicle-road cooperative perception according to claim 1 is characterized in that: In step 6), the multi-target observation data is fused as follows: 6.1) Preliminary fusion of multi-source observations After calculating the state prediction results of the multiple targets sensed at the previous moment, data association is performed with the current multi-terminal actual observation results; Preliminary fusion of multi-source actual observations before data association as follows: ； in, is the weight value of each observation; 6.2) Association Matching By building the Euclidean distance between the center of mass of the target detection box and the prediction box, it is determined whether there is a potential matching relationship. If the match is not successful, a new missing target Tracks set is established, and the state is updated and the next prediction is made. If there is a matching relationship, the corresponding matching target is further screened through the three-dimensional volume intersection-over-union ratio of 3DIoU. For targets that are not successfully matched, they are placed in the disappeared target Tracks set.

7. The full-time and full-area multi-vehicle tracking method based on multimodal vehicle-road cooperative perception according to claim 1 is characterized in that: In step 7), the observation noise covariance matrix is ​​constructed as follows: Construct the initialized observation noise covariance, then its observation noise covariance matrix is: For the first frame of full-time global multi-target tracking, the observation noise covariance of each observation end is ; As tracking continues, the observation noise covariance is adjusted by the residual difference between the actual observation and the predicted observation. The formula is as follows: in, is the residual at time t-1, is the adjustment coefficient, and its value range is ; The residual covariance matrix is: Where H is the observation matrix, is the state prediction covariance.

8. An electronic device, characterized in that: include: one or more processors; as well as a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to perform the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images