Cross-modal cross-level target association method and device, electronic equipment and medium

Abstract

The application relates to a cross-modal cross-level target association method and device, electronic equipment and medium, and belongs to the field of sensor data fusion. The method comprises the following steps: collecting visual data and radar data and performing pretreatment; converting the position coordinates of radar observation targets in the pretreated radar data into corresponding pixel coordinates; obtaining first radar observation targets capable of being projected into target detection boxes in the visual data according to the pixel coordinates of the radar observation targets; calculating the pixel distances between the pixel coordinates of each first radar observation target and the target detection box; associating the radar observation targets in the radar data and the visual observation targets in the visual data with tracking targets; screening out first tracking targets of associated visual observation targets and unassociated radar observation targets; calculating the relative distances of each first radar observation target to the vehicle; and determining the radar observation target with the minimum relative distance and pixel distance as the associated target of the corresponding first tracking target.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sensor data fusion, in particular to the field of multi-source heterogeneous sensor target matching, and especially to a cross-modal cross-level target association method, a cross-modal cross-level target association device, an electronic device and a computer readable storage medium. BACKGROUND

[0002] Target data association technology is one of the key technologies in a multi-sensor target tracking and information fusion system, and its main purpose is to realize the matching of multiple targets between frames, which includes the birth of new targets, the death of old targets, and the matching of target IDs between historical frames and current frames.

[0003] At present, the commonly used data association algorithms include nearest neighbor algorithm, probability data association algorithm, joint probability data association algorithm, etc. Among them, the target association method based on target attribute information is the most mature association method. However, the association method based on target attribute information has high requirements for the similarity of target attribute output of heterogeneous sensors. If this condition cannot be met, the association method based on target attribute information will not be able to realize the association of the same target attribute with large deviation output by heterogeneous sensors. For an intelligent driving car system, since different types of sensors such as vision, long-millimeter wave radar, short-millimeter wave radar, and ultrasonic radar are deployed, and the working principles of these sensors are not the same under the condition of external environmental interference, it is difficult to ensure the similarity of the attributes of the same target observed by different types of sensors.

[0004] In the traditional method, in order to realize the matching of multi-source heterogeneous sensors, complex operations are needed, and the overall efficiency and accuracy are low, which affects the accuracy and timeliness of target tracking. SUMMARY

[0005] One of the purposes of the present application is to provide a cross-modal cross-level target association method to solve the problem of low overall efficiency and accuracy of target association in the prior art. The second purpose is to provide a cross-modal cross-level target association device, the third purpose is to provide an electronic device, and the fourth purpose is to provide a computer readable storage medium.

[0006] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0007] A cross-modal cross-level target association method, the method comprising:

[0008] Collecting visual data and radar data in a target scene;

[0009] Preprocessing the visual data and the radar data;

[0010] convert the position coordinates of the radar observation targets in the preprocessed radar data into corresponding pixel coordinates in an image coordinate system;

[0011] acquire, according to the pixel coordinates corresponding to the radar observation targets, first radar observation targets capable of being projected into the target detection frame in the visual data;

[0012] calculate the pixel distance between the pixel coordinates corresponding to each first radar observation target and the target detection frame in the visual data;

[0013] associate the radar observation targets in the preprocessed radar data and the visual observation targets in the preprocessed visual data with the tracking targets;

[0014] screen out the first tracking targets of which the visual observation targets have been associated and the radar observation targets have not been associated;

[0015] calculate the relative distance of each first radar observation target to the vehicle;

[0016] determine the radar observation target with the minimum relative distance and the minimum pixel distance as the associated target of the corresponding first tracking target.

[0017] According to the above technical means, the interference data is removed through preprocessing of the visual data and the radar data, so that the subsequent target association data is more accurate and efficient. By projecting the radar observation targets into the target detection frame in the visual data, the corresponding associated target can be quickly determined when processing the first tracking targets that have not been associated, and the association efficiency is further improved.

[0018] In the embodiments of the present application, the visual data and the radar data are preprocessed, including:

[0019] filtering the targets located outside the field of view of the visual data acquisition device and the targets with a target code of 0 in the visual data;

[0020] filtering the targets located outside the field of view of the radar data acquisition device, and the stationary targets with a height greater than a preset height and an obstacle probability less than a preset threshold in the radar data.

[0021] The target with a target code of 0 does not meet the tracking target condition and is not an observation target of the tracking target. Through the above technical means, the part of the actual collected data that does not conform to the standard field of view of the data acquisition device can be removed, and at the same time, the targets that do not need to be tracked can be removed through processing of the visual data, and the targets that do not affect intelligent driving can be removed through processing of the radar data; effectively preventing the ramp guardrail target or the radar observation target in the tunnel from being projected into the target detection frame in the subsequent processing, causing misassociation of the target, and improving the subsequent association efficiency.

[0022] In the embodiment of the present application, the position coordinates of the radar observation target in the pre-processed radar data are converted into corresponding pixel coordinates in the image coordinate system, which includes:

[0023] According to the pinhole camera model, the position coordinates of the radar observation target in the radar coordinate system are converted into the camera coordinate system;

[0024] According to the camera imaging principle, the position coordinates of the radar observation target converted into the camera coordinate system are converted into the image coordinate system, to obtain the corresponding pixel coordinates of the position coordinates of the radar observation target in the image coordinate system.

[0025] The above technical means project the collected data in different coordinate systems into the same coordinate system, which is beneficial to improve the processing efficiency of subsequent target association.

[0026] In the embodiment of the present application, the first radar observation target capable of being projected into the target detection box in the visual data is obtained according to the pixel coordinates corresponding to the radar observation target, which includes:

[0027] The projection range corresponding to the radar observation target is determined according to the target detection box in the visual data and the preset range parameter;

[0028] If the pixel coordinates corresponding to the radar observation target are within the projection range, the radar observation target belongs to the first radar observation target.

[0029] Through the above technical means, the first radar observation target corresponding to the target detection box in the visual data is determined, and the radar observation targets that will affect the subsequent association effect are screened out, so as to improve the target association accuracy.

[0030] In the embodiment of the present application, the preset range parameter includes a horizontal axis preset range parameter X and a vertical axis preset range parameter Y;

[0031] Whether the pixel coordinates corresponding to the radar observation target are within the projection range is determined according to the following formula:

[0032]

[0033] Wherein, u r is the horizontal axis pixel coordinate corresponding to the radar observation target; v r is the vertical axis pixel coordinate corresponding to the radar observation target; u lt is the horizontal axis pixel coordinate of the top left vertex of the target detection box; v lt is the vertical axis pixel coordinate of the top left vertex of the target detection box; u rb is the horizontal axis pixel coordinate of the bottom right vertex of the target detection box; v rb is the vertical axis pixel coordinate of the bottom right vertex of the target detection box.

[0034] According to the inherent parameters of the projection conversion of the visual data acquisition device and the radar data acquisition device, the radar observation target outside the projection range is removed, the data amount of the subsequent correlation processing is reduced, and the efficiency and accuracy are improved.

[0035] In the embodiment of the application, the radar observation target in the preprocessed radar data and the visual observation target in the preprocessed visual data are associated with the tracking target, including:

[0036] The first Mahalanobis distance between the longitudinal and lateral distances of each visual observation target in the visual data and the longitudinal and lateral distances of the tracking target is calculated.

[0037] The second Mahalanobis distance between the longitudinal and lateral speeds of each visual observation target in the visual data and the longitudinal and lateral speeds of the tracking target is calculated.

[0038] The third Mahalanobis distance between the longitudinal and lateral distances of the radar observation target in the radar data and the longitudinal and lateral distances of the tracking target is calculated.

[0039] The fourth Mahalanobis distance between the longitudinal and lateral speeds of the radar observation target in the radar data and the longitudinal and lateral speeds of the tracking target is calculated.

[0040] Based on the elliptical correlation gate, the correlation between the first Mahalanobis distance and the second Mahalanobis distance and the correlation between the third Mahalanobis distance and the fourth Mahalanobis distance are used to respectively associate the visual observation target with the tracking target and the radar observation target with the tracking target.

[0041] According to the above technical means, the Mahalanobis distance is calculated based on the attribute parameters of the visual observation target and the radar observation target, so as to realize the association of the radar observation target and the visual observation target with the tracking target.

[0042] In the embodiment of the application, after the radar observation target in the preprocessed radar data and the visual observation target in the preprocessed visual data are associated with the tracking target, the target number of the association is recorded.

[0043] When the target number in the visual data and the radar data is unchanged, the tracking association is performed based on the target number.

[0044] According to the above technical means, after the first association is performed, the tracking association is directly performed according to the target number, so that the calculation time is shortened in the process of target tracking, and the target association efficiency is improved.

[0045] In the embodiment of the application, the radar observation target with the minimum relative distance and pixel distance is determined as the associated target of the corresponding first tracking target, including:

[0046] The pixel distances of the first radar observation targets are sorted in ascending order.

[0047] sort the relative distances of each first radar observation target to the vehicle from small to large;

[0048] According to the sorting, determine the radar observation target with the same relative distance and the smallest pixel distance as the associated target of the corresponding first tracking target.

[0049] Through the above technical means, the radar observation target with the same relative distance and the smallest pixel distance is determined as the associated target of the corresponding first tracking target through sorting, which can speed up the target association efficiency.

[0050] The second aspect of the application provides a cross-modal cross-level target association device, the device comprising:

[0051] A data acquisition unit is configured to collect visual data and radar data in a target scene.

[0052] A data preprocessing unit is configured to preprocess the visual data and the radar data.

[0053] A coordinate conversion unit is configured to convert the position coordinates of the radar observation targets in the preprocessed radar data into corresponding pixel coordinates in an image coordinate system.

[0054] An observation target projection screening unit is configured to obtain first radar observation targets that can be projected into the target detection frame in the visual data according to the pixel coordinates of the radar observation targets.

[0055] A pixel distance calculation unit is configured to calculate the pixel distance between the pixel coordinates of each first radar observation target and the target detection frame in the visual data.

[0056] A target association unit is configured to associate the radar observation targets in the preprocessed radar data and the visual observation targets in the preprocessed visual data with tracking targets.

[0057] An unassociated observation target screening unit is configured to screen the first tracking targets of the associated visual observation targets and the unassociated radar observation targets.

[0058] A relative distance calculation unit is configured to calculate the relative distance of each first radar observation target to the vehicle.

[0059] An associated target determination unit is configured to determine the radar observation target with the smallest relative distance and pixel distance as the associated target of the corresponding first tracking target.

[0060] Through the technical means, the device pre-processes the visual data and the radar data through the data preprocessing unit to eliminate interference data, so that the subsequent target association data is more accurate and efficient. The radar observation target is projected into the target detection frame in the visual data through the observation target projection screening unit. When the first tracking target of the unassociated radar observation target is processed, the corresponding associated target can be quickly determined, further improving the association efficiency.

[0061] In the embodiment of the application, the data preprocessing unit comprises:

[0062] The visual data preprocessing module is configured to filter out targets in the visual data that are outside the field of view of the visual data acquisition device and targets with a target code of 0.

[0063] The radar data preprocessing module is configured to filter out targets in the radar data that are outside the field of view of the radar data acquisition device, and stationary targets with a height above ground greater than a preset height and an obstacle probability less than a preset threshold.

[0064] Through the technical means, the part of the actual collected data that does not match the standard field of view of the data acquisition device can be eliminated. At the same time, the visual data preprocessing module can eliminate targets that do not need to be tracked, and the radar data preprocessing module can eliminate targets that do not affect intelligent driving. This effectively prevents the barrier target of the ramp or the radar observation target in the tunnel from being projected into the target detection frame in the subsequent processing, causing misassociation of the target, while improving the subsequent association efficiency.

[0065] In the embodiment of the application, the coordinate conversion unit comprises:

[0066] The radar coordinate system conversion module is configured to convert the position coordinates of the radar observation target in the radar coordinate system to the camera coordinate system according to the pinhole camera model.

[0067] The camera coordinate system conversion module is configured to convert the position coordinates of the radar observation target converted to the camera coordinate system to the image coordinate system according to the camera imaging principle, to obtain the corresponding pixel coordinates of the position coordinates of the radar observation target in the image coordinate system.

[0068] Through the technical means, the data collected in different coordinate systems is projected into the same coordinate system through the radar coordinate system conversion module and the camera coordinate system conversion module, which is beneficial to improving the processing efficiency of the subsequent target association.

[0069] The third aspect of the application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the cross-modal and cross-level target association method.

[0070] The fourth aspect of the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the cross-modal cross-level target association method.

[0071] The beneficial effects of the present application are:

[0072] (1) The cross-modal cross-level target association method provided by the present application can realize the association of multi-sensor output targets, and can better realize the association of targets with large attribute deviations of the same target detected by different sensors.

[0073] (2) The cross-modal cross-level target association method provided by the present application eliminates interference data by preprocessing visual data and radar data, so that the subsequent target association data is more accurate and efficient. By projecting the radar observation target into the target detection frame in the visual data, the corresponding associated target can be quickly determined when processing the unassociated first tracking target, further improving the association efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0074] Figure 1 is a flow chart of the cross-modal cross-level target association method provided by an embodiment of the present application;

[0075] Figure 2 is a technical roadmap for optimizing the efficiency of target association of the cross-modal cross-level target association method in the present application;

[0076] Figure 3a is an association effect diagram of the prior art based on target attribute information association in the case of road congestion and sudden stop of the front vehicle;

[0077] Figure 3b is an association effect diagram of the cross-modal cross-level target association method of the present application in the case of road congestion and sudden stop of the front vehicle;

[0078] Figure 4a is an association effect diagram of the prior art based on target attribute information association in the case of parking a vehicle on the side of the road;

[0079] Figure 4b is an association effect diagram of the cross-modal cross-level target association method of the present application in the case of parking a vehicle on the side of the road;

[0080] Figure 5a is an association effect diagram of the prior art based on target attribute information association in the case of cutting out the front vehicle and the front-front vehicle being stationary;

[0081] Figure 5bThe effect diagram of the target association method of the application in the case of front vehicle cut-out and the front vehicle being stationary;

[0082] Figure 6 The device block diagram of the target association method of one embodiment of the application is provided. DETAILED DESCRIPTION

[0083] The embodiments of the application will be described in detail with reference to the drawings and preferred embodiments, and those skilled in the art can easily understand other advantages and effects of the application from the content disclosed in the specification. The application can also be implemented or applied by different specific embodiments, and various modifications or changes can be made to the details in the specification based on different views and applications without departing from the spirit of the application. It should be understood that the preferred embodiments are only for illustrating the application, and are not intended to limit the protection scope of the application.

[0084] It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the application in a schematic manner, and only the components related to the application are shown in the diagrams, not the number, shape and size of the components when actually implemented. The actual implementation of each component may be a random change, and the component layout pattern may be more complex.

[0085] The terms "first", "second", "third" and the like in the embodiments of the application are only used for differentiation and description, and cannot be understood as indicating or implying relative importance.

[0086] Figure 1 The target association method of one embodiment of the application is provided, as shown in Figure 1 The method comprises:

[0087] S1: Collecting visual data and radar data in a target scene. In the embodiments of the application, the visual data is collected by a visual data collection device, and the radar data is collected by a radar data collection device. In the embodiments of the application, the CANFD data collection card is used to collect the output signals of the visual data collection device and the radar data collection device according to the CANFD signal transmission protocol of the visual data collection device and the radar data collection device, and the visual data and the radar data are obtained by analysis. In the embodiments of the application, the target scene is divided into a front view scene, a left view scene, a rear view scene and a right view scene according to the arrangement of the vehicle visual data collection device and the radar data collection device. In one embodiment of the application, the visual data is collected by a camera, and the radar data is collected by a millimeter wave radar, a laser radar or the like.

[0088] S2: Preprocessing the visual data and the radar data. In the embodiments of the application, the visual data and the radar data are preprocessed by using the following technical means:

[0089] S201: filtering the target located outside the field of view of the vision data acquisition device in the vision data and the target coded as 0.

[0090] S202: filtering the target located outside the field of view of the radar data acquisition device in the radar data, and the static target with the height above ground greater than the preset height and the obstacle probability less than the preset threshold.

[0091] The target coded as 0 belongs to the observation target not meeting the tracking target condition and not belonging to the tracking target. Through the above technical means, part of the data not meeting the standard field of view range of the data acquisition device in the actual collected data can be removed, at the same time, the target not needing to be tracked can be removed through the processing of the vision data, and the target not causing the influence on the intelligent driving can be removed through the processing of the radar data; the ramp guardrail target or the radar observation target in the tunnel projected into the target detection frame in the subsequent processing is effectively prevented, the mis-association of the target is caused, and the subsequent association efficiency is improved.

[0092] In an embodiment of the present application, the preset height is the height above ground of 4 meters, and the obstacle probability preset threshold is 60%, that is, when the radar data is preprocessed, the target located outside the field of view of the radar data acquisition device and the static target with the height above ground greater than 4 meters and the obstacle probability less than 60% are removed.

[0093] In other embodiments, the vision data and the radar data can be preprocessed according to the observation attribute of the observation target, for example, whether to filter is judged according to the jump of the observation attribute. The observation attribute can be the speed, the position, the acceleration and the like. The jump of the observation attribute refers to that the observation attribute of the same observation target in the previous frame data does not meet the kinematics law of the vehicle with the observation attribute in the next frame data. For example, taking the speed as an example, the time interval of each frame data is t, the maximum acceleration of the vehicle is a max , the speed of the observation target in the previous frame data is v1, the speed of the observation target in the next frame data is v2, if v2 > v1 + a max t, the observation attribute of the observation target jumps and needs to be filtered out. The position and the acceleration can also be filtered based on the vehicle parameters and the kinematics law.

[0094] In other embodiments, the observation target can be tracked for a short time, the target appearing for a short time is filtered out, and the vision data and the radar data are preprocessed. The appearing for a short time refers to that the frame number of the observation target appearing in the collected data is less than the preset frame number.

[0095] S3: converting the position coordinates of the radar observation target in the preprocessed radar data into the corresponding pixel coordinates in the image coordinate system, specifically including:

[0096] Based on the pinhole camera model, the position coordinates of the radar observation target located in the radar coordinate system are transformed to the camera coordinate system;

[0097] Based on the camera imaging principle, the position coordinates of the radar-observed target, which are transformed from the camera coordinate system, are transformed back to the image coordinate system to obtain the pixel coordinates corresponding to the position coordinates of the radar-observed target in the image coordinate system.

[0098] The aforementioned visual data acquired by the CANFD data acquisition card includes visually observed targets, target detection boxes corresponding to each visually observed target, position coordinates, longitudinal and lateral velocities, longitudinal and lateral distances, and target numbers for each visually observed target; the radar data includes radar-observed targets, position coordinates, longitudinal and lateral velocities, longitudinal and lateral distances, and target numbers for each radar-observed target.

[0099] To facilitate coordinate system transformation, let P be the set of position coordinates of visually observed targets in visual data, and Q be the set of position coordinates of radar-observed targets in radar data. Then P and Q can be expressed by the following formula:

[0100]

[0101] In the formula, P i q represents the position coordinates of the visually observed target in the i-th frame of the visual data; i This represents the position coordinates of the radar-observed target in the i-th frame of the radar data.

[0102] The principle of the pinhole camera model is that the relative positional relationship between any two coordinate systems can be described by the rotation matrix R and the translation vector T.

[0103] When performing coordinate system transformation, firstly, based on the pinhole camera model, the position coordinates of the radar observation target located in the radar coordinate system are transformed to the camera coordinate system. This transformation can be performed using the following formula:

[0104] P i =Rq i +T;

[0105] In the formula, P i R represents the coordinates of the radar-observed target in the camera coordinate system, R is the rotation matrix between the radar coordinate system and the camera coordinate system, and T is the translation vector between the radar coordinate system and the camera coordinate system.

[0106] Then, based on the camera imaging principle, the position coordinates of the radar-observed target, transformed from the camera coordinate system, are converted to the image coordinate system. This conversion can be performed using the following formula:

[0107] p i =KP i;

[0108] In the formula, K represents the intrinsic parameter matrix of the camera, p i This represents the pixel coordinates of the radar-observed target in the image coordinate system. Projecting data from different coordinate systems into the same coordinate system using the aforementioned techniques improves the efficiency of subsequent target association processing.

[0109] S4: Obtain the first radar-observed target within the target detection box that can be projected into the visual data based on the pixel coordinates corresponding to the radar-observed target. In this embodiment, it specifically includes:

[0110] The projection range corresponding to the radar-observed target is determined based on the target detection box in the visual data and preset range parameters. In this embodiment, the preset range parameters are determined according to the parameters of the visual data acquisition device, and include a horizontal preset range parameter X and a vertical preset range parameter Y. In one embodiment of this application, the horizontal preset range parameter X = 25, and the vertical preset range parameter Y = 100. The upper left vertex of the projection range determined according to the preset range parameters is: (u lt -25, v lt -100), the bottom right vertex is: (u rb+ 25, v rb +100).

[0111] Then, it is determined whether the pixel coordinates corresponding to the radar-observed target are within the projection range, specifically based on the following formula:

[0112]

[0113] Among them, u r It represents the horizontal pixel coordinates of the radar-observed target; v r It represents the vertical pixel coordinates of the radar-observed target; u lt It is the horizontal pixel coordinate of the top-left vertex of the object detection box; v lt It is the pixel coordinate of the top-left vertex of the target detection box along the vertical axis; u rb It is the horizontal pixel coordinate of the bottom right vertex of the object detection box; v rb This is the pixel coordinate of the bottom right vertex of the target detection bounding box along the vertical axis. It should be noted that the image coordinate system has a different orientation than the conventional coordinate system; in the image coordinate system, the vertical axis is positively oriented downwards.

[0114] By employing the aforementioned technical means, and based on the inherent parameters of projection conversion performed by visual data acquisition equipment and radar data acquisition equipment, radar observation targets outside the projection range can be eliminated, reducing the amount of data required for subsequent correlation processing and improving efficiency and accuracy.

[0115] If the pixel coordinates of the radar-observed target satisfy the above formula, then the radar-observed target can be projected into the target detection box, and the corresponding radar-observed target belongs to the first radar-observed target; otherwise, the corresponding radar-observed target is discarded.

[0116] By using the above technical means, the first radar observation target corresponding to the target detection box in the visual data is determined, radar observation targets that will affect the subsequent association effect are screened out, the pixel coordinates corresponding to each first radar observation target and the pixel distance between the target detection box in the visual data are calculated, and sorting is performed to improve the accuracy of target association.

[0117] In other embodiments, back projection can be performed, transforming the pixel coordinates of the visually observed target in the preprocessed video data to the radar coordinate system based on the estimated depth information of the visual data acquisition device. The transformed visually observed target is then projected into the detection box in the radar data. The coordinate system transformation method and the projection method of the transformed visually observed target are consistent with S3 and S4.

[0118] It should be noted that there may be one or more visual observation targets in the visual data, and each visual observation target corresponds to a target detection box. Therefore, each target detection box will be filtered to obtain its corresponding first radar observation target set.

[0119] S5: Calculate the pixel distance between the pixel coordinates of each first radar-observed target and the target detection box in the visual data. In this embodiment, the pixel distance is calculated based on the pixel coordinates of the first radar-observed target and the pixel coordinates of the center point of the target detection box. The center point coordinates of the target detection box can be determined based on the coordinates of the four vertices of the target detection box. In this embodiment, calculating the pixel distance only requires calculation using the first radar-observed target corresponding to the corresponding target detection box, and does not require calculation for all first radar-observed targets, thus saving computational resources. In this embodiment, Euclidean distance, Mahalanobis distance, cosine similarity, etc., can be used to calculate the pixel distance.

[0120] S6: Associating the radar observation targets in the preprocessed radar data and the visual observation targets in the preprocessed visual data with the tracking targets. In this embodiment, the specific steps include the following:

[0121] Calculate the first Mahalanobis distance between the longitudinal and lateral distances of each visually observed target in the visual data and the longitudinal and lateral distances of the tracked target;

[0122] Calculate the second Mahalanobis distance between the longitudinal and lateral velocities of each visually observed target and the longitudinal and lateral velocities of the tracked target in the visual data;

[0123] Calculate the third Mahalanobis distance between the longitudinal and lateral distances of the radar-observed target and the longitudinal and lateral distances of the tracked target in the radar data;

[0124] Calculate the fourth Mahalanobis distance between the longitudinal and lateral velocities of the radar-observed target and the longitudinal and lateral velocities of the tracked target in the radar data;

[0125] Based on the elliptic correlation gate, the correlation between the first and second Mahalanobis distances, as well as the correlation between the third and fourth Mahalanobis distances, are used to correlate visually observed targets with tracked targets and radar-observed targets with tracked targets, respectively. The specific correlation relationships can be expressed by the following formula:

[0126]

[0127] Where z represents the current observation value, Let S represent the predicted value of the tracked target, S represent the covariance matrix, and G represent the correlation gate constant.

[0128] Using the aforementioned technical means, the Mahalanobis distance is calculated based on the attribute parameters of visually observed targets and radar observed targets, thereby realizing the association between radar observed targets, visually observed targets, and tracked targets.

[0129] In other embodiments, the association between visually observed targets, radar observed targets, and tracked targets can be achieved through nearest neighbor algorithms or probabilistic data association algorithms.

[0130] It should be noted that the tracking target in this application comes from visual data. After the vehicle starts, the visual observation target in the first frame of visual data collected by the visual data acquisition device is used as the tracking target. As time changes, the tracking target in the collected visual data disappears or is added. The condition for adding a new tracking target is that the same observation target appears in three consecutive frames.

[0131] Associating radar observation targets from preprocessed radar data and visual observation targets from preprocessed visual data with the tracking target separately is beneficial for the decomposition of vehicle-mounted multi-sensor systems. If radar data and visual data are fused first, and then the fused data is associated with the tracking target, due to tolerance and accuracy, there may be problems such as the radar and visual sensors not being able to be fused in the early stage or the fused data not being able to be well associated with the tracking target.

[0132] S7: Select the first tracking target that is associated with visual observation targets but not with radar observation targets. In this embodiment, the first tracking target that is associated with visual observation targets but not with radar observation targets is selected by traversing all tracking targets.

[0133] S8: Calculate the relative distance from each first radar-observed target to the vehicle. In this embodiment, the polar radius distance is used to represent the relative distance from each first radar-observed target to the vehicle. In this embodiment, Euclidean distance, Mahalanobis distance, cosine similarity, etc., can be used to calculate the relative distance.

[0134] S9: Determine the radar observation target with the smallest relative distance and pixel distance as the associated target corresponding to the first tracking target. The tracking target originates from the visual observation target; therefore, the target detection box in the visual data is the target detection box of the corresponding tracking target. That is, the pixel distance calculated in step S5 is equivalent to the pixel distance from each first radar observation target to the corresponding tracking target. On the other hand, since the first radar observation target is a selected radar observation target that can be projected into the corresponding target detection box, a correspondence is formed between the first radar observation target and the tracking target corresponding to the target detection box. Therefore, the calculated relative distance from each first radar observation target to the vehicle is correlated with the corresponding tracking target; the smaller the relative distance, the more correlated it is with the corresponding tracking target.

[0135] In one embodiment of this application, the radar observation target with the smallest relative distance and pixel distance is determined by a sorting method, specifically including:

[0136] Sort the pixel distances of the targets observed by each first radar in ascending order;

[0137] Sort the relative distances from each target observed by the first radar to the vehicle in ascending order;

[0138] Based on the sorting, the radar observation target with the smallest relative distance and pixel distance sorting value is identified as the associated target of the corresponding first tracking target.

[0139] In other embodiments, the radar observation target with the smallest relative distance and pixel distance can be determined by the minimum value method. First, the radar observation target with the smallest pixel distance is determined using the pixel distance of all first radar observation targets, denoted as A1. The radar observation target with the smallest relative distance is determined using the relative distance of each first radar observation target to the vehicle, denoted as A2. A1 and A2 are compared to see if they are the same radar observation target. If they are, the radar observation target is the associated target of the corresponding first tracking target. Otherwise, the radar observation target with the smallest pixel distance is determined using the pixel distance of all first radar observation targets after removing A1, denoted as B1. The radar observation target with the smallest relative distance is determined using the relative distance of each first radar observation target after removing A2 to the vehicle, denoted as B2. If they are, the radar observation target is the associated target of the corresponding first tracking target. Otherwise, after removing B1 and B2, the minimum value is determined again, and so on.

[0140] Based on the above technical means, visual data and radar data are preprocessed to remove interference data, so that the accuracy and efficiency of subsequent target association data are higher; by projecting radar-observed targets into the target detection boxes in visual data, when processing the unassociated first tracking target, the corresponding associated target can be quickly identified, further improving the association efficiency.

[0141] In other embodiments of this application, after associating the radar observation target in the preprocessed radar data and the visual observation target in the preprocessed visual data with the tracking target, the associated target number is recorded.

[0142] When the target number remains unchanged in both visual and radar data, tracking and association are performed based on the target number.

[0143] Using the above technical means, after the initial association, tracking association can be performed directly based on the target number, which can shorten the calculation time and improve the efficiency of target association during the target tracking process. Newly added tracking targets cannot be tracked and associated using the target number if they have not been associated before.

[0144] The following explanation uses the observation targets output by the forward-looking camera and forward radar as examples.

[0145] like Figure 2 As shown, firstly, based on the CANFD signal transmission protocol of the front-view camera and the front radar, the CANFD data acquisition card is used to acquire the output signals of the front-view camera and the front radar.

[0146] The position coordinates of the observed target in the output signals of the forward-looking camera and the forward radar are represented by a set:

[0147]

[0148] In the formula, P i q represents the position coordinates of the detected target in the i-th frame of the test scene captured by the forward-looking camera; i This represents the position coordinates of the target in the i-th frame of the test scenario acquired by the radar.

[0149] Then, based on the field of view size of the forward-looking camera, the observed targets that are not within the field of view output by the forward-looking camera and the observed targets with a target number of 0 are filtered.

[0150] The field-of-view filtering based on the front radar identifies observed targets outside the field of view output by the front radar, as well as stationary targets output by the front radar with a ground height greater than 4 meters and an obstacle probability of less than 60.

[0151] Then, based on the pinhole camera model, the position coordinates of the radar observation target located in the radar coordinate system are transformed to the camera coordinate system. This transformation can be performed using the following formula:

[0152] P i =Rq i +T;

[0153] In the formula, P i R represents the coordinates of the radar-observed target output by the front radar in the camera coordinate system, R is the rotation matrix between the front radar coordinate system and the camera coordinate system, and T is the translation vector between the radar coordinate system and the camera coordinate system.

[0154] Then, based on the camera imaging principle, the position coordinates of the radar-observed target, transformed from the camera coordinate system, are converted to the image coordinate system. This conversion can be performed using the following formula:

[0155] p i =KP i ;

[0156] In the formula, K represents the intrinsic parameter matrix of the forward-looking camera, p i This represents the pixel coordinates of the radar-observed target in the image coordinate system.

[0157] Then, based on the pixel coordinates p of the radar-observed target in the image coordinate system... r (u r ,v r ) and the target detection box p output by the front-view camera lt (u lt ,v lt ) and p rb (u rb ,v rb The first radar observation target that can be projected into the forward-looking target detection frame is identified.

[0158] Then, the pixel coordinates corresponding to each target observed by the first radar and the pixel distance between them and the target detection box in the visual data are calculated and sorted.

[0159] Then, based on the elliptic association gate, the Mahalanobis distance relationship between the target's horizontal and vertical distances and velocities output by the forward-looking camera and the forward radar is used to achieve the initial association between the target output by the forward-looking camera and the target output by the forward radar and the tracked target.

[0160] After establishing the association between the target output by the forward-looking camera, the target output by the forward radar, and the tracked target, the associated forward-looking camera and forward radar target numbers are recorded. When the target numbers output by the forward-looking camera and forward radar remain unchanged, the association can be directly performed based on the target numbers.

[0161] Then, iterate through the tracking targets and, based on whether the tracking target is already associated with a radar observation target output by the previous radar, filter and output the first tracking target that is associated with a visual observation target but not associated with a radar observation target output by the previous radar.

[0162] For radar observation targets projected onto the forward-looking target detection box and sorted, calculate their relative distances to the vehicle, and select the radar observation target with the smallest distance from the target detection box and the shortest relative distance, and use it as the radar observation target associated with the tracking target.

[0163] In one application scenario of this application, when there is a traffic jam and the vehicle in front suddenly brakes to a stop, there will be a significant difference between the speed of the vehicle in the radar data and the speed in the visual image. The elliptic gate-based association method cannot achieve this. However, by calculating the pixel coordinates of the vehicle projected from the radar data into the visual data, the pixel distance between the target detection box in the visual data and the relative distance between the vehicle in the radar data and the current vehicle, the association between the vehicle in the radar data and the vehicle in the visual data can be achieved. The specific effect is as follows: Figure 3a and 3b As shown, in a scenario where a forward-looking camera and a forward-looking radar observe a stationary target ahead of the lane, the forward-looking radar outputs the target's speed as 0 m / s, while the forward-looking camera outputs an initial speed of 10 m / s, converging towards 0 m / s over time. Since the target is determined based on visual data, the forward-looking output of the stationary target can be correlated with the tracked target. However, the speed of the stationary target output by the forward-looking radar differs significantly from the tracked target's speed, making correlation impossible using target attribute-based methods; therefore, it is discarded. As the forward-looking output of the stationary target's speed converges, only after a certain period, when the speed converges to a certain value, can correlation between the forward-looking output of the stationary target and the tracked target, as well as between the forward-looking output of the stationary target and the tracked target, be achieved using target attribute-based correlation methods. Figure 3a The image illustrates an existing association method based on target attributes, which can achieve association within a 24-meter range. Figure 3b The method proposed in this application achieves association at a distance of 44 meters. In other words, using this method, even if the speed of the stationary target output by the front radar differs significantly from the speed of the tracked target, association can be achieved earlier than association methods based on target attributes. From the perspective of the vehicle's forward movement, existing association methods based on target attributes achieve association later and over shorter distances, while this application achieves target association earlier and over longer distances, which is more beneficial for the vehicle to take countermeasures.

[0164] In another application scenario of this application, if a vehicle is parked on the side of the road and the distance between the parked vehicle and the vehicle is large, the observation attribute error of the vehicle in the radar data and visual data will be large. In this case, it is also impossible to associate the vehicle using the target attribute-based association method. By calculating the pixel coordinates of the stationary vehicle projected from the radar data into the visual data, the pixel distance between the target detection box in the visual data and the relative distance between the stationary vehicle and the vehicle, it is possible to associate the stationary vehicle in the radar data with the stationary vehicle in the visual data. The specific effect is as follows: Figure 4b As shown, Figure 4a The image shows an existing association method based on target attributes, which can achieve association within 42 meters. Figure 4b This indicates that the method proposed in this application achieves association within 71 meters. From the perspective of the vehicle's forward movement, existing association methods based on target attributes achieve association later and over shorter distances, while this application achieves target association earlier and over longer distances, which is more conducive to the vehicle taking countermeasures.

[0165] In another scenario of this application, while the vehicle is in motion, the vehicle in front cuts out, and the vehicle in front of that is stationary. In this case, the specific effect is as follows: Figure 5a and 5b As shown, Figure 5a The image illustrates an existing association method based on target attributes, which is able to achieve association at a vertical distance of 38 meters. Figure 5b This indicates that the method proposed in this application achieves association at a longitudinal distance of 44 meters. The method in this application achieves target association earlier.

[0166] A second aspect of this application provides a cross-modal, cross-level target association device, such as... Figure 6 As shown, the device includes:

[0167] The data acquisition unit is used to collect visual and radar data within the target scene.

[0168] The data preprocessing unit is used to preprocess visual data and radar data;

[0169] The coordinate transformation unit is used to convert the position coordinates of radar observed targets in the preprocessed radar data into the corresponding pixel coordinates in the image coordinate system.

[0170] The target projection filtering unit is used to obtain the first radar observation target that can be projected into the target detection box in the visual data based on the pixel coordinates of the radar observation target.

[0171] The pixel distance calculation unit is used to calculate the pixel coordinates of each first radar-observed target and the pixel distance between the target detection box in the visual data;

[0172] The target association unit is used to associate radar observation targets in preprocessed radar data and visual observation targets in preprocessed visual data with the tracking target;

[0173] The unassociated observation target filtering unit is used to filter out the first tracking target that is associated with visual observation targets and unassociated with radar observation targets.

[0174] The relative calculation unit is used to calculate the relative distance from each first radar-observed target to the vehicle.

[0175] The associated target determination unit is used to determine the radar observation target with the smallest relative distance and pixel distance as the associated target of the corresponding first tracking target.

[0176] Through the aforementioned technical means, the device preprocesses visual and radar data using a data preprocessing unit to remove interference data, thereby improving the accuracy and efficiency of subsequent target association data. The observation target projection screening unit projects radar-observed targets into target detection boxes in the visual data, enabling the rapid identification of corresponding associated targets when processing the first tracking target that is not associated with radar-observed targets, further improving association efficiency.

[0177] In this embodiment of the application, the data preprocessing unit includes:

[0178] The visual data preprocessing module is used to filter out targets located outside the field of view of the visual data acquisition device and targets with a target code of 0 from the visual data.

[0179] The radar data preprocessing module is used to filter targets located outside the field of view of the radar data acquisition equipment, as well as stationary targets with a ground clearance greater than a preset height and an obstacle probability less than a preset threshold.

[0180] By employing the aforementioned technical means, some data in the actual collected data that does not conform to the standard field of view of the data acquisition equipment can be removed. At the same time, the visual data preprocessing module can remove targets that do not need to be tracked, and the radar data preprocessing module can remove targets that will not affect intelligent driving. This effectively prevents targets such as guardrails on ramps or radar observation targets in tunnels from being projected into the target detection box during subsequent processing, thus avoiding false association of targets and improving the efficiency of subsequent association.

[0181] In this embodiment of the application, the coordinate transformation unit includes:

[0182] The radar coordinate system transformation module is used to transform the position coordinates of radar observation targets located in the radar coordinate system to the camera coordinate system based on the pinhole camera model.

[0183] The camera coordinate system transformation module is used to transform the position coordinates of the radar observation target in the camera coordinate system to the image coordinate system according to the camera imaging principle, so as to obtain the pixel coordinates of the radar observation target in the image coordinate system.

[0184] By using the aforementioned technical means, data collected in different coordinate systems can be projected into the same coordinate system through radar coordinate system transformation module and camera coordinate system transformation module, which is beneficial to improving the processing efficiency of subsequent target association.

[0185] A third aspect of this application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the cross-modal, cross-level target association method.

[0186] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the cross-modal, cross-level target association method described above.

[0187] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a microcontroller, chip, or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0188] The optional embodiments of this application have been described in detail above with reference to the accompanying drawings. However, the embodiments of this application are not limited to the specific details described above. Within the scope of the technical concept of the embodiments of this application, various simple modifications can be made to the technical solutions of the embodiments of this application, and these simple modifications all fall within the protection scope of the embodiments of this application. It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, the embodiments of this application will not describe the various possible combinations separately.

[0189] Furthermore, various different implementations of this application can be combined arbitrarily, as long as they do not violate the spirit of the implementation of this application, they should also be regarded as the content disclosed in the implementation of this application.

Claims

1. A cross-modal, cross-level target association method, characterized in that, The method includes: Collect visual and radar data within the target scene; Preprocess visual and radar data; The position coordinates of radar-observed targets in the preprocessed radar data are converted into corresponding pixel coordinates in the image coordinate system; The first radar-observed target within the target detection box that can be projected into the visual data is obtained based on the pixel coordinates of the radar-observed target. Calculate the pixel coordinates of each target observed by the first radar and the pixel distance between the target detection box in the visual data; The radar observation targets in the preprocessed radar data and the visual observation targets in the preprocessed visual data are associated with the tracking targets; The first tracking target is selected from those that are associated with visual observation targets but not with radar observation targets; Calculate the relative distance from each target observed by the first radar to the vehicle; The radar observation target with the smallest relative distance and pixel distance is identified as the associated target of the corresponding first tracking target.

2. The cross-modal, cross-level target association method according to claim 1, characterized in that, Preprocessing of visual and radar data includes: Filter out targets located outside the field of view of the visual data acquisition device and targets with a target code of 0 from the visual data; Filter targets located outside the field of view of the radar data acquisition equipment in the radar data, and stationary targets with a ground height greater than a preset height and an obstacle probability less than a preset threshold.

3. The cross-modal, cross-level target association method according to claim 1, characterized in that, The position coordinates of radar-observed targets in the preprocessed radar data are converted into corresponding pixel coordinates in the image coordinate system, including: Based on the pinhole camera model, the position coordinates of the radar observation target located in the radar coordinate system are transformed to the camera coordinate system; Based on the camera imaging principle, the position coordinates of the radar-observed target, which are transformed from the camera coordinate system, are transformed back to the image coordinate system to obtain the pixel coordinates corresponding to the position coordinates of the radar-observed target in the image coordinate system.

4. The cross-modal, cross-level target association method according to claim 1, characterized in that, The first radar-observed target within the target detection box that can be projected into the visual data is obtained based on the pixel coordinates corresponding to the radar-observed target, including: The projection range corresponding to the radar-observed target is determined based on the target detection box in the visual data and the preset range parameters. If the pixel coordinates corresponding to the radar observation target are within the projection range, then the radar observation target belongs to the first radar observation target.

5. The cross-modal, cross-level target association method according to claim 4, characterized in that, The preset range parameters include the horizontal axis preset range parameter X and the vertical axis preset range parameter Y; The following formula is used to determine whether the pixel coordinates corresponding to the radar-observed target are within the projection range: Among them, u r It represents the horizontal pixel coordinates of the radar-observed target; v r It represents the vertical pixel coordinates of the radar-observed target; u lt It is the horizontal pixel coordinate of the top-left vertex of the object detection box; v lt It is the pixel coordinate of the top-left vertex of the target detection box along the vertical axis; u rb It is the horizontal pixel coordinate of the bottom right vertex of the object detection box; v rb It is the vertical pixel coordinate of the bottom right vertex of the target detection box.

6. The cross-modal, cross-level target association method according to claim 1, characterized in that, Associating radar-observed targets in preprocessed radar data and visually observed targets in preprocessed visual data with the tracking target includes: Calculate the first Mahalanobis distance between the longitudinal and lateral distances of each visually observed target in the visual data and the longitudinal and lateral distances of the tracked target; Calculate the second Mahalanobis distance between the longitudinal and lateral velocities of each visually observed target and the longitudinal and lateral velocities of the tracked target in the visual data; Calculate the third Mahalanobis distance between the longitudinal and lateral distances of the radar-observed target and the longitudinal and lateral distances of the tracked target in the radar data; Calculate the fourth Mahalanobis distance between the longitudinal and lateral velocities of the radar-observed target and the longitudinal and lateral velocities of the tracked target in the radar data; Based on the elliptic correlation gate, the correlation between the first and second Mahalanobis distances, as well as the correlation between the third and fourth Mahalanobis distances, are used to correlate visually observed targets with tracked targets and radar-observed targets with tracked targets, respectively.

7. The cross-modal, cross-level target association method according to claim 1, characterized in that, After associating the radar observation targets in the preprocessed radar data and the visual observation targets in the preprocessed visual data with the tracking targets, the associated target numbers are recorded. When the target number remains unchanged in both visual and radar data, tracking and association are performed based on the target number.

8. The cross-modal, cross-level target association method according to claim 1, characterized in that, The radar-observed target with the smallest relative distance and pixel distance is identified as the associated target of the corresponding first tracking target, including: Sort the pixel distances of the targets observed by each first radar in ascending order; Sort the relative distances from each target observed by the first radar to the vehicle in ascending order; Based on the sorting, the radar observation target with the smallest relative distance and pixel distance sorting value is identified as the associated target of the corresponding first tracking target.

9. A cross-modal, cross-level target association device, characterized in that, The device includes: The data acquisition unit is used to collect visual and radar data within the target scene. The data preprocessing unit is used to preprocess visual data and radar data; The coordinate transformation unit is used to convert the position coordinates of radar observed targets in the preprocessed radar data into the corresponding pixel coordinates in the image coordinate system. The target projection filtering unit is used to obtain the first radar observation target that can be projected into the target detection box in the visual data based on the pixel coordinates of the radar observation target. The pixel distance calculation unit is used to calculate the pixel coordinates of each first radar-observed target and the pixel distance between the target detection box in the visual data; The target association unit is used to associate radar observation targets in preprocessed radar data and visual observation targets in preprocessed visual data with the tracking target; The unassociated observation target filtering unit is used to filter out the first tracking target that is associated with visual observation targets and unassociated with radar observation targets. The relative distance calculation unit is used to calculate the relative distance from each target observed by the first radar to the vehicle. The associated target determination unit is used to determine the radar observation target with the smallest relative distance and pixel distance as the associated target of the corresponding first tracking target.

10. The cross-modal, cross-level target association device according to claim 9, characterized in that, The data preprocessing unit includes: The visual data preprocessing module is used to filter out targets located outside the field of view of the visual data acquisition device and targets with a target code of 0 from the visual data. The radar data preprocessing module is used to filter targets located outside the field of view of the radar data acquisition equipment, as well as stationary targets with a ground clearance greater than a preset height and an obstacle probability less than a preset threshold.

11. The cross-modal, cross-level target association device according to claim 9, characterized in that, The coordinate transformation unit includes: The radar coordinate system transformation module is used to transform the position coordinates of radar observation targets located in the radar coordinate system to the camera coordinate system based on the pinhole camera model. The camera coordinate system transformation module is used to transform the position coordinates of the radar observation target in the camera coordinate system to the image coordinate system according to the camera imaging principle, so as to obtain the pixel coordinates of the radar observation target in the image coordinate system.

12. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the cross-modal, cross-level target association method as described in any one of claims 1-8.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the cross-modal, cross-level target association method as described in any one of claims 1-8.

Images