A Raid-visual target association method with adaptive coupling of measurement calibration model and cost matrix

Abstract

This invention discloses a radar-visual target association method that adaptively couples a measurement calibration model with a cost matrix. First, the collected multi-sensor data is preprocessed to obtain radar-visual measurements and undergoes spatiotemporal transformation. Second, visual measurement offset calibration is performed to establish a parameterized measurement calibration model. Then, mask filtering is applied to the radar-visual measurements to separate unassociated measurements. Next, the measurement calibration model is coupled to adaptively select and correct the cost matrix. Finally, an allocation method is used to establish target association, and secondary filtering is performed to obtain the association result. This method solves the offset association problem under the same sensor configuration with a single measurement calibration, exhibiting low complexity and high reliability. It addresses the instability of radar-visual target association during multi-sensor fusion, improves the reliability of multi-source sensor environmental perception in traffic scenarios, and enhances the accuracy of radar-visual measurement matching across multiple scenarios. It can be applied to scenarios requiring radar-visual sensor fusion, such as autonomous driving and traffic monitoring.

Description

Technical Field

[0001] This invention belongs to the field of multi-sensor fusion technology, specifically relating to a radar-visual target association method that adaptively couples a measurement calibration model with a cost matrix. Background Technology

[0002] Environmental perception is the cornerstone of autonomous driving and intelligent transportation development. Currently, the mainstream environmental perception sensors in the transportation field include cameras (also known as visual sensors), lidar, and millimeter-wave radar (simply referred to as radar). Cameras have advantages such as rich semantic information, low cost, and mature technology, facilitating target detection and recognition. However, they are highly susceptible to special environmental conditions such as strong light, nighttime, rain, snow, and fog, and their range and velocity measurement capabilities are weak. Millimeter-wave radar has advantages such as all-weather robustness, accurate detection speed, and long detection range, but it lacks semantic texture information, making accurate target identification and classification difficult. LiDAR, on the other hand, is currently expensive and still some distance from widespread application. Considering the shortcomings of single sensors, developing multi-sensor fusion perception, especially the fusion of radar and visual sensors (referred to as radar-visual fusion), is crucial and represents a trend towards achieving reliable and robust environmental perception.

[0003] Current radar-visual fusion can be categorized into target-level fusion, feature-level fusion, data-level fusion, and hybrid fusion. Among these, target-level fusion, characterized by rapid deployment and high reliability, is a highly valuable fusion method for widespread application. The key to target-level fusion is solving the target association problem among different sensors. The paper "Accurate Radar Measurements Association with Visual Targets in Traffic Scene. 2022 IEEE Radar Conference (RadarConf22). New York City, NY, USA: IEEE, 2022: 1-6" uses visual recognition results as prior information and combines Euclidean distance to conduct radar-visual (radar and vision) target association research. However, it does not consider the significant differences in measurements between different sensors due to variations in sensor environmental perception methods. This method only has some effect in the near-field region and cannot encompass radar-visual target association across the entire joint sensor perception range. The paper "Research on Intelligent Vehicle Environmental Perception Technology Based on Multi-Source Information Fusion. Doctoral Thesis, Jiangsu University, 2018" conducted a unified covariance analysis on radar-visual measurement and constructed a radar-visual target association method based on Mahalanobis distance. This method directly integrates radar-visual measurement, ignores the differences between millimeter-wave radar and visual sensors, and does not consider the measurement instability bias phenomenon generated during visual target detection (i.e., visual detection has some reliable results and some results with large deviations), which makes it difficult for the method to have high reliability and scalability. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention proposes a radar-visual target association method that adaptively couples a measurement calibration model with a cost matrix. This method solves the problem of measurement differences between radar and visual sensors that are ignored by existing association methods, thereby achieving reliable radar-visual fusion perception.

[0005] The technical solution adopted in this invention is: a radar target association method adaptively coupled with a measurement calibration model and a cost matrix, the specific steps of which are as follows:

[0006] Step 1: Radar visual data preprocessing to generate radar visual measurements;

[0007] Install radar and vision sensors, calibrate the sensors, obtain intrinsic and extrinsic parameter data, and collect data from radar and vision sensors respectively.

[0008] The radar echo sampling data is processed, and strong background clutter is removed using moving target indication and moving target detection. The radar measurements under radar coordinates are obtained using a constant false alarm rate detector and a direction of arrival estimation method.

[0009] Based on the original images captured by the camera, deep learning is used to perform object detection on the visual images and obtain visual measurements in the pixel coordinate system.

[0010] Step 2: Measure the spatiotemporal transformation to obtain the same-domain radar measurement;

[0011] The external parameters obtained during sensor calibration are used to transform the radar measurements into a unified coordinate system. The pixel coordinate transformation is shown in Equation (1), and the radar coordinate transformation is shown in Equation (2). The expressions are as follows:

[0012]

[0013]

[0014] Where (x,y,z) represents a unified coordinate system; z v K represents the depth in the camera coordinate system; R represents the camera intrinsic parameters. v and R r T represents the extrinsic rotation matrices of the camera and radar coordinate systems relative to a unified coordinate system, respectively. v and T r (u,v) represents the extrinsic translation matrices of the camera and radar coordinate systems relative to a unified coordinate system; (u,v) represents the pixel coordinate system; (x) represents the extrinsic translation matrix of the camera and radar coordinate systems relative to a unified coordinate system. r ,y r ,z r ) represents the radar coordinate system.

[0015] Then, based on the timestamps of the radar sensors, the measurements are extrapolated to the same moment to obtain a radar measurement set in the same spatial and temporal domains. and visual measurement set The expression is as follows:

[0016]

[0017]

[0018] Where, r i This represents the i-th radar point in the radar measurement set, whose coordinates in the unified coordinate system are: v j Let j be the j-th visual measurement point in the visual measurement set, and let its coordinates be in the unified coordinate system. N r N represents the number of radar measurement points. v Indicates the number of visual measurement points.

[0019] Step 3: Establish a visual measurement offset calibration model using joint radar;

[0020] Within the joint sensing range of the sensors: 0~R max Dense radar-view data of a single target is continuously acquired in a straight-line driving scenario, and the radar-view measurements are subjected to spatiotemporal transformation to unify them to the same spatial and temporal domains. The expression is as follows:

[0021]

[0022] in, This represents accumulated measurement and calibration data from multiple measurements; k r i , k v i ) represents the i-th radar measurement pair during the k-th data acquisition process; N cal N represents the number of measurement and calibration data collected in a single session. k This indicates the cumulative number of experimental data.

[0023] Using radar measurements as the reference true value, the instability offset between visual and radar measurements in the forward-looking direction (x-direction) is calculated, and a radar-based visual measurement offset calibration model is established based on sensor characteristics.

[0024]

[0025] Where F represents the calibration model; δ x δ represents the minimum threshold at which visual measurements deviate in the x-direction. y This represents the maximum distance threshold for radar measurements in the y-direction that belong to the same target.

[0026] Finally, by accumulating multiple calibration data, the measurement calibration model is solved using the least squares algorithm.

[0027]

[0028] Among them, F * This represents the solved parameterized measurement calibration model. and This represents the coordinate value in the x-direction of the i-th radar measurement pair during the k-th data acquisition process.

[0029] Step 4: Use a mask filter to separate discrete single-sensor measurements;

[0030] Based on the same spatial and temporal radar measurement sets, construct the initial cost matrix C and the mask matrix M respectively:

[0031]

[0032] Among them, c ij Let represent the cost between the i-th radar measurement and the j-th visual measurement; H represents the cost calculation function between the radar and visual measurements (including Euclidean distance and Mahalanobis distance); m ij Let represent the vector of the element in the i-th row and j-th column of the mask matrix, and

[0033] Then, a multi-dimensional differential mask is established based on the sensor measurement characteristics, as shown in the following expression:

[0034]

[0035] Among them, G M G represents a mask constructed based on the cost matrix M. ij This represents the mask value in the i-th row and j-th column of the mask; G(·) represents the mask construction function; T x T y These represent the differential mask thresholds in the x and y directions, respectively.

[0036] Then, a mask filter is used to separate the discrete single sensor measurement points, and the cost matrix after filtering is updated to C′. At this time, the number of effective associative radar target points is N′. r and N′ v The expression for C' is as follows:

[0037]

[0038] Where, c′ i c' represents all elements in the i-th row of the cost matrix C'; j c' represents all elements in the j-th column of the cost matrix C'; ijThis represents the element in the i-th row and j-th column of the cost matrix C′; g ij Represents mask G M The mask value in the i-th row and j-th column.

[0039] Step 5: Construct the selectively corrected cost matrix;

[0040] Using each radar measurement point as a reference, the visual measurements are divided into stable measurement sets. and offset measurement set The visual stability measurement set obtained by dividing the i-th radar measurement is as follows Visual offset measurement set is The expression is as follows:

[0041]

[0042]

[0043] By combining the radar-visual measurement calibration model and the visual measurement set, the initial cost matrix is ​​selectively modified to obtain the modified cost matrix C″, as shown in the following expression:

[0044]

[0045]

[0046] Where, c″ ij Let H represent the cost of the corrected i-th radar measurement and j-th visual measurement; H represents the cost calculation function between radar and visual measurements. The visual measurement represents the cost of calculating the i-th radar measurement.

[0047] Step Six: Target Association Assignment;

[0048] The selectively corrected cost matrix is ​​correlated using an allocation method to obtain the radar target correlation result set.

[0049]

[0050] in, n 'a' represents the nth group of radar-visual association pairs. n r and n v represents the radar and visual measurements of the nth correlation pair, respectively; N n This indicates the total number of measurements; Asso(·) indicates the associated allocation algorithm.

[0051] Step 7: Use a quadratic filter to remove erroneous association pairs and obtain the association results;

[0052] The association allocation result set obtained in step six is ​​further subjected to secondary mask filtering to remove erroneous association points generated by the allocation method. The secondary mask cost matrix is ​​constructed from the preliminary association pairs in step six, and the expression is as follows:

[0053] M′={m′ n |m′ n ={Δx n ,Δy n}, n=1,2,...,N n} (16)

[0054] Where, N n M' represents the total number of association pairs, and M′ represents the quadratic mask cost matrix with dimension N. n ×1, m′ n Let Δx represent the mask cost of the nth pair of associations, and Δx n =| n x (r) - n x (v) |,Δy n =| n y (r) - n y (v) |, n x (r) and n x (v) Let represent the coordinates of the radar and vision in the x-direction for the nth correlation pair, respectively. n y (r) and n y (v) Let represent the y-coordinates of the radar and vision pairs in the nth association pair, respectively.

[0055] Finally, based on the secondary mask cost matrix, the mask filter from step four is used for secondary mask filtering to obtain the final association pairs.

[0056]

[0057] Among them, G M′ This indicates that a mask is constructed based on the quadratic mask cost matrix M′, and g′ n This represents the nth mask value in the mask.

[0058] Furthermore, in step six, the allocation method includes: Hungarian algorithm, nearest neighbor algorithm, global nearest neighbor association algorithm, probabilistic interconnection algorithm, and joint probabilistic interconnection algorithm.

[0059] The beneficial effects of this invention are as follows: The method of this invention first preprocesses the collected multi-sensor data to obtain radar-visual measurements and performs spatiotemporal transformation. Next, it performs visual measurement offset calibration, establishing a parameterized measurement calibration model. Then, it performs mask filtering on the radar-visual measurements to separate uncorrelated measurements. The cost matrix is ​​then adaptively selected and corrected by coupling the measurement calibration model. Finally, an allocation method is used to establish target association, and secondary filtering is performed to obtain the association result. This method can solve the offset association problem under the same sensor configuration with a single measurement calibration, exhibiting low complexity and high reliability. It can solve the problem of unstable radar-visual target association during multi-sensor fusion, promoting the reliability of multi-source sensor environmental perception in traffic scenarios, and improving the accuracy of radar-visual measurement matching throughout the entire process in multiple scenarios. It can be applied to scenarios requiring radar-visual sensor fusion, such as autonomous driving and traffic monitoring. Attached Figure Description

[0060] Figure 1 This is a flowchart of a radar target association method that adaptively couples a measurement calibration model with a cost matrix according to the present invention.

[0061] Figure 2 This is a schematic diagram of the coordinate system used in the embodiments of the present invention.

[0062] Figure 3 This is a simulation scene diagram in an embodiment of the present invention.

[0063] Figure 4 This is a schematic diagram of the simulation results in an embodiment of the present invention.

[0064] Figure 5 This is a schematic diagram illustrating the parameters related to the accuracy evaluation index in an embodiment of the present invention.

[0065] Figure 6 This is a comparison chart of the accuracy of the method of the present invention and existing methods in the embodiments of the present invention.

[0066] Figure 7 This is a graph showing the correlation accuracy of 100 Monte Carlo simulations in an embodiment of the present invention.

[0067] Figure 8 This is a schematic diagram of the measured results of the method of the present invention in an embodiment of the present invention. Detailed Implementation

[0068] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0069] This invention mainly uses simulation experiments for verification and evaluation, and all steps and results are verified for effectiveness in Python.

[0070] like Figure 1The flowchart shown is a method for adaptively coupling a measurement calibration model and a cost matrix to associate radar-visual targets according to the present invention. The specific steps are as follows:

[0071] Step 1: Radar visual data preprocessing to generate radar visual measurements;

[0072] Install radar and vision sensors, calibrate the sensors, obtain intrinsic and extrinsic parameter data, and collect data from radar and vision sensors respectively.

[0073] The radar echo sampling data (radar sensor data) is processed, and strong background clutter is removed using moving target indication and moving target detection. The radar measurements under radar coordinates are detected and obtained using a constant false alarm rate detector and a direction of arrival estimation method.

[0074] Based on the raw images (visual sensor data) captured by the camera, deep learning is used to perform object detection on the visual images and obtain visual measurements in the pixel coordinate system.

[0075] Step 2: Measure the spatiotemporal transformation to obtain the same-domain radar measurement;

[0076] The intrinsic and extrinsic parameters obtained during sensor calibration are used to transform the radar measurement into a unified coordinate system (see diagram). Figure 2 As shown), the pixel coordinate transformation is as shown in Equation (1), and the radar coordinate transformation is as shown in Equation (2).

[0077] Then, based on the timestamps of the radar sensors, the measurements are extrapolated to the same moment to obtain a radar measurement set in the same spatial and temporal domains. and visual measurement set As in equations (3) and (4).

[0078] Step 3: Establish a visual measurement offset calibration model using joint radar;

[0079] Within the joint sensing range of the sensors: 0~R max Dense radar vision data of a single target scene is continuously collected, and the radar vision measurement is transformed in time and space to be unified to the same spatial and temporal domains, as shown in equation (5).

[0080] Then, using radar measurements as the reference true value, the instability offset between visual measurements and radar measurements in the forward-looking direction (x direction, i.e., the sensor sensing direction) under specific conditions is calculated, and a radar-based visual measurement offset calibration model is established in combination with sensor characteristics, as shown in equation (6).

[0081] Finally, the calibration data is accumulated from multiple tests, and the measurement calibration model is solved using the least squares algorithm as shown in equation (7). The more times the data is collected and the smaller the collection interval during the measurement calibration process, the more reliable the obtained measurement calibration model will be.

[0082] Depending on the characteristics of the visual sensor and the different target detection methods, the measurement calibration model includes, but is not limited to, linear functions, quadratic functions, and cubic functions.

[0083] Step 4: Use a mask filter to separate discrete single-sensor measurements;

[0084] The initial cost matrix C and the mask matrix M are constructed based on the same spatial domain and the same temporal domain radar measurement sets, respectively, as shown in equation (8).

[0085] Then, a multi-dimensional differentiated mask is established based on the sensor measurement characteristics, as shown in Equation (9). The mask filter is then used to separate the discrete single sensor measurement points, and the cost matrix after filtering is updated to C′, as shown in Equation (10). At this time, the number of effective (non-empty) associative radar target points is N′. r and N′ v .

[0086] Single sensor measurement points do not participate in subsequent correlation; candidate targets with potential multi-sensor measurements are included in the subsequent correlation steps.

[0087] Step 5: Construct the selectively corrected cost matrix;

[0088] Due to the imaging characteristics of visual sensors, the measurements obtained by visual target detection algorithms are unstable and biased. However, not all measurements are biased. Therefore, the cost matrix is ​​adaptively selected and corrected.

[0089] Using each radar measurement point as a reference, the visual measurements are divided into stable measurement sets. and offset measurement set By combining the Ravis measurement calibration model and the visual measurement set, the initial cost matrix is ​​selectively modified to obtain the modified cost matrix C″, as shown in equation (13).

[0090] Step Six: Target Association Assignment;

[0091] The selectively corrected cost matrix is ​​correlated using an allocation method to obtain the radar target correlation result set. (i.e., preliminary correlation results).

[0092] The allocation methods include: Hungarian algorithm, nearest neighbor algorithm, global nearest neighbor association algorithm, probabilistic interconnection algorithm, and joint probabilistic interconnection algorithm.

[0093] Step 7: Use a quadratic filter to remove erroneous association pairs and obtain the association results;

[0094] The association allocation result set obtained in step six is ​​further subjected to secondary mask filtering to remove erroneous association points generated by the allocation method. The secondary mask cost matrix is ​​constructed from the preliminary association pairs in step six, as shown in equation (16). Then, the mask filter in step four is used for secondary mask filtering to obtain the final association pairs. As shown in equation (17).

[0095] like Figure 3 As shown in the diagram, the simulation scene in this embodiment illustrates the motion process of the simulated target. The target motion attribute information is shown in Table 1, where " / " indicates that the motion state remains unchanged. The error distribution of the radar and visual simulation target is shown in Table 2.

[0096] Table 1

[0097] Target number T1 T2 T3 T4 T5 T6 T7 T8 Time of occurrence (s) 0 0 1 0 2 1 1 0 Starting position (x, y) (8.75,0) (12.25,0) (47,41.75) (47,46.25) (5.25,80) (1.75,80) (-33,38.25) (-33,34.75) Initial motion model CV CV CV CV CV CV CV CV Speed ​​of movement 22 30 -30 -30 -30 -18 60 60 Waiting time (s) (4.8,5) (3.6,5) / / (5.6,7) / / / Turning time (s) (5,8) / / / / (7,8.5) / / Turning model CTRV / / / / CTRV / / Turning speed (°) -30 / / / / 60 / / Termination conditions y＝-33 x＝80 y＝-33 y＝-33 x＝0 y＝-33 y＝47 y＝47

[0098] Table 2

[0099] x direction y direction Radar (m) 0.2 0.4 Visual (m) 0.8 0.2

[0100] During the simulation, some target points are randomly discarded to simulate measurement loss, with a loss probability of 20%. At the same time, an unstable offset is randomly added to the visual simulation target with a probability of 33%. The offset follows the measurement calibration model, and the visual measurement calibration model used in the simulation is as shown in equation (18):

[0101] F * =2.58×10 -6 x 3 -4.84×10 -5 x 2 +4.78×10 -2 x + 4.32 × 10 -1 (18)

[0102] To evaluate the simulation accuracy, this embodiment also designed a method for calculating evaluation indicators to measure the reliability of the correlation, as shown in Table 3.

[0103] Table 3

[0104]

[0105] In Table 3, positive samples represent the set of radar and visual point pairs that can be correctly associated, with the number of point pairs being P. Negative samples represent the union of radar and visual measurements that cannot be correctly associated, with the number of measurement points in the set being the number of negative samples, denoted as N. The total number of samples is T. CP represents the number of correctly associated point pairs in the positive samples; WP represents the number of incorrectly associated point pairs in the positive samples; CN represents the number of correctly associated measurement points in the negative samples; WN represents the number of incorrectly associated measurement points in the negative samples; CRP represents the percentage of correctly associated positive samples; CRN represents the percentage of correctly associated negative samples; and CRT represents the percentage of correctly associated samples in the overall sample.

[0106] Figure 4 This is a schematic diagram of the simulation results in this embodiment, where the solid line represents the association result of the present invention, the dashed line represents the association result of the existing method, and the association indicated by the arrow is an incorrect association. Figure 5 This section explains the relevant parameters of the accuracy evaluation index of the method of the present invention in this embodiment. Figure 6 To compare the accuracy of the method of this invention with existing methods, Figure 7 The figure shows the overall association accuracy curve of the method of the present invention in 100 Monte Carlo simulations in this embodiment. Proposed represents the association accuracy of the method of the present invention, Classic represents the association accuracy of the existing method, and the dashed lines represent the average overall accuracy of the corresponding methods. Figure 8 The diagram illustrates the measured results of the method of the present invention in this embodiment. In this diagram, a-1, a-2, and a-3 are three consecutive frames of association results. The solid line represents the association result of the method of the present invention, the dashed line represents the association result of the existing method, and the association indicated by the arrow is an incorrect association. b is the corresponding reference visual image.

[0107] In summary, the method of this invention first addresses the issue of offset in visual measurements by proposing a visually unstable measurement calibration model that combines radar target information. It then analyzes a parameterized measurement calibration model based on the least squares method. Next, it applies mask filtering to differentiated measurements using prior information, and then adaptively selects and corrects the radar-visual target association cost matrix by coupling the measurement calibration model. Finally, it completes the association of radar-visual targets and performs secondary filtering based on the corrected cost matrix to obtain the association result. This method fully considers the differences between millimeter-wave radar and visual sensors in the target measurement process. It obtains a parameterized visual measurement calibration model through an offset calibration method and adaptively selects and corrects the association cost matrix by coupling the measurement calibration model. This achieves an association method that takes into account the differences in radar-visual sensor measurements and the visually unstable offset phenomenon, greatly improving the accuracy of radar and visual measurement matching across multiple scenarios and the entire process. This method can solve the offset association problem under the same sensor configuration with a single measurement calibration, exhibiting low complexity and high reliability. It can be applied to scenarios requiring radar-visual sensor fusion, such as autonomous driving and traffic monitoring.

[0108] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A method for radar-visual target association by adaptive coupling of measurement calibration model and cost matrix, comprising the following steps: Step one, radar-visual data preprocessing to generate radar-visual measurements; Install radar and visual sensors, calibrate the sensors to obtain internal and external parameter data, and collect radar and visual sensor data respectively; Process radar echo sampling data, use moving target indication and moving target detection to remove strong background clutter, use constant false alarm detector and direction of arrival estimation method to detect radar measurements in radar coordinates; Based on the original image collected by the camera, use deep learning to detect the visual image and obtain visual measurements in the pixel coordinate system; Step two, measurement space-time transformation to obtain the same domain radar-visual measurements; Convert radar-visual measurements to a unified coordinate system using the external parameters obtained during sensor calibration; Step three, establish a visual measurement offset calibration model with joint radar; In the sensor joint perception range The single target dense radar vision data in the straight driving scene is continuously collected, and the space-time transformation of the radar vision measurement is unified to the same space domain and time domain, and the expression is as follows: （1）； wherein, represents cumulative multiple measurement calibration data; represents the first data acquisition process in the first radar measurement pair; represents the number of single acquisition measurement calibration data; represents the number of experimental data accumulation times; Then, the radar measurement is taken as the reference true value, and the front direction is calculated, i.e. The instability offset of the visual measurement and the radar measurement in the front direction is combined with the sensor characteristics to establish a radar-based visual measurement offset calibration model: （2）； wherein, denotes a calibration model; denotes a minimum threshold of a shift of visual measurement in the direction, denotes a maximum distance threshold of visual measurement belonging to the same target in the direction. Finally, accumulate multiple calibration data and use the least squares algorithm to solve the measurement calibration model: （3）； wherein, denotes the solved parameterized measurement calibration model, and denotes the first data acquisition process the radar vision measurement of the coordinate value in the direction; Step four, separate discrete single sensor measurements using a mask filter; According to the same space domain, the same time domain, the initial cost matrix is constructed respectively by the radar visual measurement set and the mask matrix : （4）； wherein, represents the cost of the th radar measurement and the th vision measurement; represents a cost computation function between radar and vision measurements; represents the vector of elements in the th row and the th column of the mask matrix, and ;​ Step five, construct a selective correction cost matrix; Using each radar measurement point as a reference, the visual measurements are divided into stable measurement sets. and offset measurement set ; Divide the first The set of visual stability measurements obtained from radar measurements is as follows: The visual offset measurement set is The expression is as follows: （5）； （6）； The initial cost matrix is selectively corrected by combining the LIDAR measurement calibration model and the visual measurement set to obtain a corrected cost matrix The expression is as follows: （7）； （8）； in, Indicates the corrected number The first radar measurement and the first The cost of visual measurement; A function representing the cost calculation between radar and visual measurements; Indicates the calculation of the first Visual measurement at the cost of radar measurement. Step six, target association and assignment; The cost matrix after selective correction is associated and distributed using a distribution method to obtain a set of radar-visual target association results : （9）； wherein, , denotes the set of radar- vision associated pairs, with denotes the set of radar and vision measurements of associated pairs; denotes the total number of measurement pairs; denotes the association assignment algorithm; Step seven, use secondary filtering to remove false association pairs to obtain the association result; Step six, perform secondary mask filtering on the association assignment result set obtained in step six to remove false association points generated by the assignment method.

2. The method of claim 1, wherein the method further comprises: In step two, pixel coordinate conversion is as formula (10) and radar coordinate conversion is as formula (11), and the expressions are as follows: （10）； （11）； wherein, represents a unified coordinate system; represents a depth in a camera coordinate system; represents a camera intrinsic parameter; and respectively represent an extrinsic rotation matrix of a camera and a radar coordinate system relative to a unified coordinate; and respectively represent an extrinsic translation matrix of a camera and a radar coordinate system relative to a unified coordinate; represents a pixel coordinate system; represents a radar coordinate system; According to the radar visual sensor timestamp, the measurement is extrapolated to the same time, and the radar measurement set in the same space domain and time domain is obtained and the visual measurement set , the expression is as follows: （12）； （13）； wherein, represents the i-th radar point in the radar measurements, whose coordinate in the unified coordinate system is ; ; represents the i-th vision measurement point in the vision measurements, whose coordinate in the unified coordinate system is ; ; represents the number of radar points; represents the number of vision measurement points; In step four, a multi-dimensional differentiated mask is established based on the characteristics of sensor measurements, and the expression is as follows: （14）； wherein, represents a cost matrix based on mask construction, represents a mask value of the mask at the row and the column; represents a mask construction function; , respectively represent a direction and a direction differentiated mask threshold value; The discrete single sensor measurement points are separated by using a mask filter, and the updated filtered cost matrix is At this time, the effective number of associated radar target points is and , The expression is as follows: （15）； in, Representing the cost matrix The Middle All elements of the row; Representing the cost matrix The Middle All elements in the column; Representing the cost matrix The Middle Line number Column elements; Indicates mask The Middle Line number The mask value of the column; In step six, the assignment method includes: Hungarian algorithm, nearest neighbor algorithm, global nearest neighbor association algorithm, probability interconnection algorithm, and joint probability interconnection algorithm; In step seven, a secondary mask cost matrix is constructed from the preliminary association pairs in step six, and the expression is as follows: （16）； wherein, represents the total number of associated pairs, represents a quadratic mask cost matrix, the matrix dimension is , represents the mask cost of the group of associated pairs, and , , and respectively represent the coordinates of the group of associated pairs in the radar and vision in the direction, and respectively represent the coordinates of the group of associated pairs in the radar and vision in the direction. Finally, based on the secondary mask cost matrix, the mask filter in step four is used to do secondary mask filtering to obtain the final correlation pair : （17）； wherein, represents a quadratic mask cost matrix constructing a mask, represents the i-th mask value in the mask mask value.

Images