Radar data and video data fusion method and device and related apparatus

Abstract

The application discloses a radar data and video data fusion method and device and related apparatus, and the radar data and video data fusion method comprises the following steps: obtaining a radar target in a radar image and a video target in a video image at a current time, and mapping the radar target and the video target to a same coordinate system; obtaining a target distance between the radar target and the video target in the same coordinate system, and historical matching data of the radar target and the video target at a historical time, the historical matching data being used for representing a matching condition of the radar target and the video target at the historical time; determining the radar target and the video target belonging to a same object according to the target distance and the historical matching data; and correlatively fusing radar data of the radar target and video data of the video target belonging to the same object. Through the above method, the application can improve the accuracy of radar target and video target correlation.

Description

Technical Field

[0001] This invention relates to the field of radar video technology, and in particular to a method, device, and related apparatus for fusing radar data and video data. Background Technology

[0002] In modern society, security is receiving increasing attention from the public, leading to a proliferation of security products, an expanding range of applications, and rapid advancements in security technology. Area surveillance technology based on radar and video is a recent research hotspot. Matching and combining radar and video targets that are geographically close can leverage their respective strengths. However, due to calibration errors or the influence of radar reflectors, the positions of the radar point and the center point of the video target in the same coordinate system always contain a certain degree of random error. Relying solely on the distance between the radar and video targets may result in incorrect matching and association, reducing detection accuracy. Summary of the Invention

[0003] The main technical problem solved by this invention is to provide a method, device and related apparatus for fusion of radar data and video data, which can improve the accuracy of correlation between radar targets and video targets.

[0004] To address the aforementioned technical problems, one technical solution adopted by this invention is to provide a method for fusing radar data and video data. This method includes: acquiring radar targets in a radar image and video targets in a video image at the current moment, and mapping the radar targets and video targets to the same coordinate system; acquiring the target distance between the radar targets and video targets in the same coordinate system; acquiring historical matching data of radar targets and video targets at historical moments, the historical matching data being used to characterize the matching status of radar targets and video targets at historical moments; determining radar targets and video targets belonging to the same object based on the target distance and historical matching data; and fusing the radar data of the radar targets belonging to the same object and the video data of the video targets.

[0005] The historical matching data includes historical fusion degree and historical trajectory overlap degree. Historical fusion degree is used to characterize the fusion of radar targets and video targets at historical moments. Based on the target distance and historical matching data, radar targets and video targets belonging to the same object are identified, including: weighted fusion of target distance, historical fusion degree and historical trajectory overlap degree, to obtain the matching degree between radar targets and video targets; radar targets and video targets belonging to the same object are identified based on the matching degree.

[0006] Among them, the matching degree is positively correlated with the target distance and the overlap of historical trajectories, and negatively correlated with the historical fusion degree. Determining radar targets and video targets belonging to the same object based on the matching degree includes: obtaining the matching degree between each radar target and video target at the current moment; when the matching degree is the minimum, the radar target and video target belong to the same object.

[0007] The historical fusion degree of radar targets and video targets in historical video frame images is determined by the following methods: obtaining the radar video detection confidence level at a historical moment and the number of radar video interference targets at a historical moment; fusing the radar video detection confidence level and the number of radar video interference targets to obtain the historical fusion degree of radar targets and video targets in historical video frame images.

[0008] Among them, the historical fusion degree is directly proportional to the credibility of the achieved video detection, and inversely proportional to the number of radar video interference targets.

[0009] In this context, the radar target and the video target belong to the same object at the historical moment. The number of radar video interference targets at the historical moment includes: the number of interference targets within the neighborhood of the fused coordinates. The interference targets include radar targets and video targets. The fused coordinates are the fused targets of radar targets and video targets at the historical moment. The neighborhood range is the distance between radar targets and video targets in the historical video frame image ± a predetermined distance.

[0010] The process of obtaining the radar video detection credibility at historical moments includes: obtaining the detection credibility of radar targets at historical moments, and the detection credibility of video targets at historical moments; and multiplying the detection credibility of radar targets and video targets by weight to obtain the radar video detection credibility.

[0011] Among them, the farther the video target is from the video detector, the lower the confidence level of the video target detection; as the distance between the radar target and the radar detector increases, the confidence level of the radar target detection first increases and then decreases.

[0012] The historical trajectory overlap is determined by the following methods: obtaining the positional distance between the radar target and the video target at historical moments; calculating the average positional distance at each historical moment to obtain the historical trajectory overlap between the radar target and the video target.

[0013] To solve the above-mentioned technical problems, another technical solution adopted by the present invention is to provide a radar data and video data fusion device, which includes a processor for executing the radar data and video data fusion method described above.

[0014] To solve the above-mentioned technical problems, another technical solution adopted by the present invention is to provide a computer-readable storage medium for storing instruction / program data, which can be executed to realize the above-mentioned radar data and video data fusion method.

[0015] The beneficial effects of this invention are as follows: Unlike the prior art, this invention, in addition to considering the actual distance between the radar and video targets at the current moment, also comprehensively considers the historical matching data of the two targets at historical moments, fuses two targets belonging to the same object, and combines the data at the current moment and the historical moment to make up for the calibration error that may exist in the data at the current moment, effectively improving the accuracy of the association between radar targets and video targets. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating one implementation method of the radar data and video data fusion method of this application;

[0017] Figure 2 This is a flowchart illustrating another embodiment of the radar data and video data fusion method of this application;

[0018] Figure 3 This is a schematic diagram of the collaborative monitoring of radar detectors and video detectors in a road scenario;

[0019] Figure 4 This is a flowchart illustrating a specific implementation of the radar data and video data fusion method of this application;

[0020] Figure 5 This is a schematic diagram of the radar detection image and video detection image of this application;

[0021] Figure 6 This is a schematic diagram of the radar data and video data fusion device in the embodiments of this application;

[0022] Figure 7 This is a schematic diagram of the structure of the radar data and video data fusion device in the embodiments of this application;

[0023] Figure 8 This is a schematic diagram of the structure of a computer-readable storage medium in an embodiment of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and effects of the present invention clearer and more explicit, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0025] Traditional security terminal equipment mainly consists of visible light cameras. While visible light cameras excel at obtaining target category information, they struggle to capture target movement information. For example, they easily identify target types (humans, non-machines), but struggle to estimate target speed and spatial location. Target identification accuracy is lower at longer distances, at night, and in rainy or foggy weather compared to close-range targets; that is, the temporal and spatial accuracy of the target within the monitored area is inconsistent, even significantly so. Millimeter-wave radar, on the other hand, can obtain target measurement information including distance, angle, radial speed, and radar cross-section (RCS). Millimeter-wave radar actively emits electromagnetic waves and receives signals of the same frequency, exhibiting a very high detection probability for moving objects or objects with a large RCS, but a lower (not zero) detection probability for stationary objects. Millimeter-wave radar can operate 24 hours a day and is less affected by weather. Therefore, combining the two technologies can leverage their respective strengths and compensate for their weaknesses, further improving the target detection and identification probability while also providing target distance, orientation, and speed information. This application provides a method for fusing radar data and video data. By considering the actual distance between the radar and video targets in the current frame, and also taking into account the fusion data of the two targets in historical video frames, the method fuses two targets belonging to the same object. By combining the data at the current moment and the data at the historical moment, the method compensates for the calibration error that may exist in the current moment image, and effectively improves the accuracy of the association between radar targets and video targets.

[0026] Please see Figure 1 , Figure 1 This is a flowchart illustrating one embodiment of the radar data and video data fusion method of this application. It should be noted that if substantially the same result is obtained, this embodiment does not necessarily reflect that outcome. Figure 1 The illustrated process sequence is limited. For example... Figure 1 As shown, this embodiment includes:

[0027] S110: Obtain radar targets in the radar image and video targets in the video image at the current moment.

[0028] The radar detector and video detector each acquire the current frame image of the current scene. At the current moment, the radar image includes one or more radar targets, and the video image includes one or more video targets. The radar targets and video targets are mapped to the same coordinate system. In the embodiments of this application, a one-to-one matching calculation is performed between the radar targets and the video targets to determine whether fusion is possible.

[0029] S130: Under the same coordinate system, obtain the target distance between the radar target and the video target.

[0030] First, the target distance between the radar target and the video target is obtained. This target data is used to reflect the probability that the radar target and the video target are the same object at the current moment.

[0031] S150: Acquire historical matching data of radar targets and video targets at historical moments.

[0032] Specifically, the radar target at a historical moment is the same as the radar target at the current moment, and the video target at a historical moment is the same as the video target at the current moment. Historical matching data reflects the probability that a radar target and a video target in a historical video frame are the same object, such as the target distance between the radar target and the video target in the historical video frame, whether the radar target and the video target are fused, and the accuracy of the fusion. Historical matching data can be data from a single historical frame or data from multiple historical frames, and it is used to characterize the matching status of radar targets and video targets at historical moments.

[0033] S170: Based on target distance and historical matching data, identify radar targets and video targets belonging to the same object.

[0034] By combining target data and historical matching data—that is, by combining data on the probability of radar targets and video targets being the same object at the current moment with data on the probability of radar targets and video targets being the same object at historical moments—it is determined whether radar targets and video targets are the same object.

[0035] S190: Correlate and fuse radar data of radar targets belonging to the same object and video data of video targets.

[0036] The radar targets and video targets belonging to the same object are associated, and the radar data of the radar targets and the video data of the video targets are fused.

[0037] In this implementation, by taking into account the actual distance between the radar and video targets at the current moment, and also taking into account the historical matching data of the two targets at historical moments, the two targets belonging to the same object are fused together. The combination of data from the current moment and historical moments makes up for the calibration error that may exist in the data at the current moment, effectively improving the accuracy of the association between radar targets and video targets.

[0038] Please see Figure 2 , Figure 2 This is a flowchart illustrating another embodiment of the radar data and video data fusion method of this application. It should be noted that if substantially the same result is achieved, this embodiment does not necessarily reflect that outcome. Figure 2 The illustrated process sequence is limited. For example... Figure 2 As shown, this embodiment includes:

[0039] S210: Obtain the target distance between the radar target and the video target at the current moment.

[0040] The target distance between a radar target and a video target can be used to reflect the probability that the radar target and the video target are the same object at the current moment. Although due to calibration errors and radar reflector errors, the radar target and the video target that are closest to each other are not necessarily the same object, radar targets and video targets that are farther apart are definitely not the same object. Therefore, the target distance between the radar target and the video target, i.e., the actual distance, is used as a criterion for judging whether they are the same object. Specifically, the coordinates (x1, y1) of the radar target and the coordinates (x2, y2) of the video target are obtained, where the two coordinates are in the same coordinate system. When the coordinates obtained directly are in different coordinate systems, coordinate transformation is performed beforehand. The target distance between the radar target and the video target is calculated as follows:

[0041]

[0042] S230: Acquire the historical fusion degree and historical trajectory overlap degree of radar targets and video targets.

[0043] Historical matching data between radar targets and video targets can be used to reflect the probability that a radar target and a video target in historical video frames are the same object. In one embodiment, the historical matching data includes historical fusion degree and historical trajectory overlap degree. The historical fusion degree can be the number of fusion attempts, fusion accuracy, etc. The historical fusion degree is used to characterize the fusion status of radar targets and video targets at historical moments.

[0044] In one implementation, the historical fusion frame count of radar targets and video targets is obtained as the historical fusion degree. The more fusion frames, the greater the probability that the radar target and video target belong to the same object. Specifically, historical video frame images are acquired, and it is detected whether the radar target and video target belong to the same object at each time moment. Finally, the number of times the radar target and video target belong to the same object at each historical time moment is accumulated to obtain the historical fusion frame count.

[0045] In another embodiment, the number of radar video interference targets at a historical moment is obtained as the historical fusion degree. The number of radar video interference targets at a historical moment represents the number of targets that interfered with the currently detected radar and video targets at that historical moment. The fewer the targets, the lower the probability of interference with the radar and video targets; conversely, the more targets, the higher the probability of interference. When the radar target and the video target belong to the same object at a historical moment, the fused coordinates are the coordinates of the radar target and the video target after successful fusion at that historical moment. The neighborhood range is the distance between the radar target and the video target at the historical moment ± a predetermined distance. The number of radar and video targets within the neighborhood range of the fused coordinates is obtained. Specifically, the number of interference targets is the number of radar and video targets within a circular area with a certain radius centered on the fused coordinates. In one embodiment, if the distance between the radar target and the video target at a historical moment is s, and the predetermined distance is u, then the neighborhood range is a circular area with a radius of s ± u centered on the fused coordinates.

[0046] In another implementation, the radar video detection confidence level at a historical moment is obtained as the historical fusion score. The higher the radar video detection confidence level, the more accurate the historical matching data between the radar target and the video target. Specifically, the detection confidence level of the radar target at a historical moment and the detection confidence level of the video target at a historical moment are obtained, and the detection confidence levels of the radar target and the video target are multiplied by a weighted sum to obtain the radar video detection confidence level. Specifically, if the detection confidence level of the radar target is f... i The confidence level for detecting the target in the video is f. j The reliability of radar video detection is w i f i w j f j , where w i w j , where represents the weighting coefficient. Targets within the detection range show better detection performance by radar and video sensors, and the reliability of the association / fusion results for targets in this range is relatively high. Specifically, the greater the distance between the video target and the video detector, the lower the detection reliability of the video target; conversely, as the distance between the radar target and the radar detector increases, the detection reliability of the radar target first increases and then decreases. The radar video detection reliability is obtained for each time step based on the distances of each radar target and video target from the detector at historical time points.

[0047] In another embodiment, the reliability of radar video detection and the number of interfering targets in the radar video are fused to obtain the historical fusion degree of radar targets and video targets at historical moments. The following example illustrates this using radar target i in the k-th frame of the historical radar image and video target j in the video image.

[0048] Get the number of radar video jamming targets card(A) kij Specifically, successfully fused historical video frame images are obtained. Assuming that radar target i in the k-th frame of the historical radar image and video target j in the video image match the same object and are successfully fused, the fused target of radar target i and video target j is obtained. The distance between the radar target and the video target at the historical moment is calculated, and all targets within the neighborhood of the distance fusion coordinates in the k-th frame image are calculated. The calculation is expressed as:

[0049] A kij ={Y|y<y kij ±u, Y∈R trg ∪V trg},

[0050] Where u is the predetermined distance, R trg V represents all radar targets. trg Represents all video targets, y kij Let A represent the distance between radar target i and video target j in the k-th frame of the successfully fused dataset, and let y represent the distance between the element target and the fused coordinates. kij Card(A) represents the set of all targets within the neighborhood of radar target i and video target j after fusion of the k-th historical video frame image. kij Let A be a set. kij The number of elements in A kij The smallest element in the middle is 1.

[0051] Obtain the historical detection confidence level of radar video. For video detectors, the detection confidence level of video targets has a linear relationship with the distance to the video targets. For radar detectors, the detection confidence level of radar targets has a quadratic relationship with the distance to the radar targets. In this implementation, the historical detection confidence level of the radar targets is multiplied by the historical detection confidence level of the video targets. The specific calculation formula is as follows:

[0052] ρ kij =(k v y kj +bv)(a r y ki 2 +b r y ki +c r ),

[0053] Where, k v and b v For the video fitting parameters, a r b r c r These are the radar fitting parameters. kiy represents the position of radar target i in the k-th frame of the historical video image. kj ρ represents the position of video target j in the k-th historical video frame image. kij This represents the reliability of radar video detection fused from radar target i and video target j in the k-th frame image.

[0054] By fusing the reliability of radar video detection and the number of radar video interference targets, the historical fusion degree of radar targets and video targets in historical images is obtained as follows:

[0055]

[0056] Where, ρ kij Let card(A) be the radar video detection confidence level of the k-th historical image. kij δ represents the number of radar video interference targets in the k-th frame of the historical image. δ is the number of radar target i and video target j belonging to the same object. kij δ is 1 when radar target i and video target j do not belong to the same object. kij w is 0 21 w 22 This is a correction factor. The higher the historical fusion rate, the greater the likelihood that radar targets and video targets belong to the same object.

[0057] In another embodiment, the historical fusion degree of radar targets and video targets in historical images is:

[0058]

[0059] Among them, w 23 σ represents the correction factor. ij This indicates whether radar target i and video target j have a fused record. When radar target i and video target j have a fused record, σ ij σ is 1 when radar target i and video target j have no fused records. ij It is 0.

[0060] When both radar and video detectors simultaneously detect the same object, the trajectory of the radar target and the video target is unique and similar. Therefore, the higher the overlap between the historical trajectories of the radar target and the video target, the greater the likelihood that they belong to the same object. Specifically, the distance between the radar target and the video target in historical images is obtained; the average distance in each historical image is calculated to obtain the overlap between the historical trajectories of the radar target and the video target. The smaller the average value, the higher the overlap. The specific calculation formula is:

[0061]

[0062] Among them, (xi ,y i (x) represents the position of radar target i, (x) j ,y j The value () represents the position of video target j. Each calculated radar target i and video target j should be targets in the same frame. If a frame contains only radar targets or only video targets, then trajectory overlap is not calculated for that frame.

[0063] S250: Weighted fusion of target distance, historical fusion degree, and historical trajectory overlap degree to obtain the matching degree between radar target and video target.

[0064] By weighted fusing target distance, historical fusion reliability, and historical trajectory overlap, the matching degree between radar targets and video targets is obtained as follows:

[0065]

[0066] Where F is the target matching degree, f1 is the target distance, f2 is the historical fusion degree of radar targets and video targets in multiple historical images, f3 is the historical trajectory overlap degree of radar targets and video targets in multiple historical images, and w1, w2, and w3 are the proportional weights.

[0067] S270: Obtain the matching degree between each radar target and the video target at the current moment. When the matching degree is the minimum, the radar target and the video target belong to the same object.

[0068] Obtain all radar targets and video targets at the current frame, calculate the matching degree for each radar target and video target, and determine that the radar target and video target with the smallest matching degree are the same object.

[0069] S290: Correlate and fuse radar data of radar targets belonging to the same object and video data of video targets.

[0070] In this implementation, in addition to considering the current actual distance between radar and video targets, the historical fusion degree and historical trajectory overlap of the two targets are also comprehensively considered. Introducing historical fusion information reduces the impact of detection errors when a large number of targets are clustered together, improving the fusion accuracy in congested scenarios. By comparing the trajectory characteristics of radar and video targets using historical trajectory overlap, the high calibration accuracy requirements during fusion are reduced. Combining current and historical data to select two targets belonging to the same object for fusion compensates for potential calibration errors in the current image, effectively improving the accuracy of radar and video target association.

[0071] In practical applications, such as surveillance in parks, roads, bridges, industrial parks, and squares, target types are obtained through video, and the spatial location and speed of the targets are obtained through radar. The radar targets and video targets are then correctly matched / fused. Based on this accurate fusion, data services can be provided for road traffic indicators such as traffic volume, space occupancy, time occupancy, and queue length. Furthermore, the accurate fusion result of radar and video targets forms the basis for subsequent event processing. Please refer to [link / reference]. Figure 3 , Figure 3 This is a schematic diagram illustrating the collaborative monitoring of radar and video detectors in a road scene. The radar detector targets a clustered area, but for decision-level target fusion, the radar target is merely a clustered point, while the video target is a rectangular region within the image. For example... Figure 3 The left image shows a monitoring scene of a vehicle in motion, while the right image shows a monitoring scene during traffic congestion. In each scene, target detection boxes ABCD represent four video targets, and target points 1, 2, 3, and 4 represent radar targets mapped to the image after calibration. Due to calibration errors or radar reflector surface errors, the positions of the radar points and the center points of the video targets in the same coordinate system always have a certain degree of random error. Figure 3 In the left image, due to the large distance between vehicles during travel, the calibrated video and radar targets can be correctly correlated / fused, ignoring random errors. However, Figure 3 In the scenario shown in the right figure, the correct matching result should be that radar target 1 is associated with video target A, radar target 2 with video target B, radar target 3 with video target C, and radar target 4 with video target D. Due to traffic congestion and small distances between vehicles, random errors can cause radar target 2 to be closer to the center point of video target A, and radar target 3 to be closer to video target B, leading to erroneous association results such as radar target 2 being associated with video target A and radar target 3 with video target B. Therefore, this application proposes a weighted distance-based method for fusing radar data and video data.

[0072] Please see Figure 4 , Figure 4 This is a flowchart illustrating a specific implementation of the radar data and video data fusion method of this application. It should be noted that if substantially the same result is achieved, this embodiment does not necessarily reflect that outcome. Figure 4 The illustrated process sequence is limited. For example... Figure 4 As shown, this embodiment includes:

[0073] The system initializes and runs the radar detector and video detector, setting the parameter thresholds used in the radar and video data fusion method, and the target type to be fused. It further configures the intrinsic and extrinsic parameters of the radar detector and video detector, such as the intrinsic and extrinsic parameter calibration method and the four-point calibration method.

[0074] By utilizing configured radar and video detectors to acquire current video frames, and excluding targets outside the sensor's monitoring range or those obstructed, the system can accurately detect the number and location of vehicles and assign a unique ID to each target. Please refer to [link / reference]. Figure 5 , Figure 5 This is a schematic diagram of the radar detection image and video detection image of this application. For example... Figure 5 As shown, the video detector detects video target AD, and the radar detector detects radar targets 1-4.

[0075] Specifically, for video detectors, video tracking algorithms are used to provide the location of the bounding boxes of video targets in real time, and a target sequence and its corresponding bounding box sequence are established:

[0076]

[0077]

[0078]

[0079] in, Indicates the video target ID, BoxPt i Indicates the position of the video target bounding box. (Target bounding box position) in, These represent the pixel coordinates of the top-left, top-right, bottom-left, and bottom-right points, respectively. Further, the pixel coordinates of the target bounding box are converted to world coordinates, and the input video target bounding box is BoxPt. i Then, the center point (U, V) of the target bounding box is obtained, and finally, (U, V) is converted into the corresponding output (X, Y) using a calibration algorithm. The conversion method is as follows:

[0080] Vloc i =G(BoxPt) i ),

[0081] Among them, Vloc i This represents the position of the video target in the world coordinate system. G(*) represents the calibration function that converts the pixels of the target bounding box to their positions in the world coordinate system.

[0082] For radar detectors, radar can detect and track the number of radar targets and point cloud data. Radar sensors can acquire measurements in the environment, including point cloud data from real targets and point cloud data from other false targets. The following radar target sequence and corresponding point cloud set sequence can be established:

[0083] Rtrg1,CartPtSet1;

[0084] Rtrg2,CartPtSet2;

[0085] Rtrg3, CartPtSet3; ...

[0086] in, Indicates the radar target ID, CartPtSet i This represents the set of radar target point cloud locations.

[0087] All measurement data are further processed using clustering and calibration algorithms. Specifically, density clustering is used to obtain the number of radar targets and the point cloud data corresponding to each radar target. Then, the regions of the radar targets are extracted. The rectangular outer contour of the point cloud data of each radar target is determined, and the actual position of the radar target in the world coordinate system is calculated. The following mapping relationship can be constructed:

[0088] Rloc i =H(CartPtSet) i ),

[0089] Among them, Rloc i H(*) represents the position of the radar target in the world coordinate system, and H(*) represents the clustering algorithm used to process the radar point cloud data.

[0090] After obtaining the coordinates of multiple radar targets and multiple video targets using the above method, the radar targets and multiple video targets are combined into pairs of radar target data and video target data based on timestamps. The weighted distance between any radar target and a video target is calculated, and this weighted distance is used as a measure of whether they are the same object. After processing all measurement data using clustering and calibration algorithms, based on the processed radar target and target information, the following mapping relationship can be constructed:

[0091]

[0092] Where d weight Indicates radar target With video target The weighted distance between the radar target and the video target. F(*) represents the function for calculating the weighted distance between the radar target and the video target.

[0093] Multiple influencing indicators for calculating the weighted distance between radar targets and video targets are obtained, such as target distance, historical fusion degree, and historical trajectory overlap degree.

[0094] First, obtain the target distance between the radar target and the video target. In single-frame target association, the actual distance between the radar target and the video target is a crucial basis for target association. Although due to calibration errors and radar reflector errors, the closest radar target and the video target may not necessarily be the same target, radar targets that are farther apart and video targets are definitely not the same target. In the previous steps, the radar and video targets have been converted to points (X,Y) in the world coordinate system. Therefore, the target distance between radar target i and video target j is:

[0095]

[0096] Where (x) i ,y i (x) represents the location of the radar target. j ,y j () indicates the location of the video target.

[0097] Second, obtain the historical fusion degree of radar targets and video targets. Target distance alone can only roughly determine whether two targets in a single frame might be the same target. In situations with dense targets, relying solely on radar and video target distances is prone to error. Therefore, historical fusion degree is needed to assist in the judgment. The historical fusion degree is affected by several factors, including the number of historical fusion frames, the number of interfering targets in the radar video, and the reliability of radar video detection.

[0098] First, radar and video sensors perform well in detecting targets within their detection range, and the reliability of the association / fusion results for targets in this area is relatively high. For video sensors, in traffic road scenes, their detection performance is linearly related to distance, meaning the detection performance deteriorates with increasing distance. For radar sensors, in traffic road scenes, their detection performance should be a quadratic function of distance, meaning the accuracy of detecting the nearest and farthest targets is relatively poor. Multiplying the radar target detection reliability by the video target detection reliability yields the radar video detection reliability at a historical moment, which can be expressed mathematically as:

[0099] ρ kij =(k v y kj +b v (a) r y ki 2 +b r y ki +c r ),

[0100] Where, k v and b v For the video fitting parameters, a r b r cr These are the radar fitting parameters. ki y represents the position of radar target i in the k-th frame. kj ρ represents the position of target j in the k-th video frame. kij This represents the reliability of radar video detection fused from radar target i and video target j in the k-th frame.

[0101] Secondly, sparse targets in traffic areas have higher reliability in target association / fusion results due to the absence of interference; the more surrounding targets there are, the greater the possibility of interference with the target association / fusion results. The successful fusion of radar target i and video target j in the k-th frame yields the fused target. The number of interfering targets within the neighborhood of the fused target can be represented as: card(A kij ), where set A kij This represents the set of all targets within the neighborhood of radar target i and video target j after the fusion of the k-th frame.

[0102] A kij ={Y|y<y kij ±u, Y∈R trg ∪V trg},

[0103] Where u is a predetermined range, R trgg V represents all radar targets. trg Represents all video targets, y kij Let A represent the distance between radar target i and video target j in the k-th frame of successful fusion, and let y represent the distance between the element target and the historical fused targets. kij Card(A) represents the set of all targets within the neighborhood of radar target i and video target j after fusion of the k-th historical video frame image. kij Let A be a set. kij The number of elements in A kij The smallest element in the middle is 1.

[0104] Targets that were successfully fused in the historical context are highly likely to be the same target in the current frame; that is, the more fusion attempts, the higher the fusion reliability. However, the fusion results are not reliable in some cases. Therefore, a confidence level should be determined for each fusion result to calculate the truly reliable number of fusion attempts, i.e., the historical fusion degree. Whether targets i and j in the k-th frame are fused is determined by δ. kij In other words, the historical fusion degree of radar targets and video targets at a historical moment can be expressed as:

[0105]

[0106] Furthermore, the function model can be represented as:

[0107]

[0108] Where, ρ kij Let card(A) be the radar video detection confidence level at the k-th historical frame. kij δ represents the number of radar video interference targets at the k-th historical frame. When radar target i and video target j belong to the same object, δ... kij δ is 1 when radar target i and video target j do not belong to the same object. kij σ is 0 ij This indicates whether radar target i and video target j have a fused record. When radar target i and video target j have a fused record, σ ij σ is 1 when radar target i and video target j have no fused records. ij w is 0 21 w 22 w 23 This is a correction factor.

[0109] Third, obtain the historical trajectory overlap between radar and video targets. For both radar and video sensors, if they simultaneously detect the same target, then the target's trajectory is unique, and the trajectories detected by both sensors should be similar. Calculate the average distance between the positions at various historical moments to obtain the historical trajectory overlap between the radar and video targets. The smaller the average value, the higher the historical trajectory overlap. This can be expressed mathematically as:

[0110]

[0111] Where, x i ,y i Indicates the radar target position, x j ,y j This indicates the location of the video target. Each calculated radar target i and video target j should be targets from the same frame. If a frame contains only a radar target or only a video target, then trajectory overlap is not calculated for that frame.

[0112] Considering the varying degrees of influence of the above factors under different circumstances, the matching degree calculation formula can be expressed as:

[0113]

[0114] Where F is the weighted distance (matching degree), f1 is the target distance, f2 is the historical fusion degree of radar target and video target at multiple historical moments, f3 is the historical trajectory overlap degree of radar target and video target at multiple historical moments, and w1, w2, and w3 are the proportional weights.

[0115] Calculate the weighted distance between each radar target and each video target separately. The radar target and video target with the smallest weighted distance, i.e. the smallest matching degree value, are identified as the same object, and then the radar targets and video targets of the same object are fused.

[0116] In this implementation, in addition to considering the current actual distance between radar and video targets, the historical fusion degree and historical trajectory overlap of the two targets are also comprehensively considered. Historical fusion information is introduced, and accurate historical fusion results are calculated using the number of targets in the neighborhood and the sensor detection effectiveness curve. This reduces the impact of detection errors when a large number of vehicle targets are clustered together, improving the fusion accuracy in congested scenarios. By calculating the mean square error of trajectory points in the effective frames of the two vehicle targets, the trajectory characteristics of the radar and video targets are compared, reducing the high calibration accuracy requirements during fusion. By combining current and historical data, two targets belonging to the same object are selected for fusion, compensating for potential calibration errors in the current frame image and effectively improving the correlation accuracy between radar and video targets.

[0117] Please see Figure 6 , Figure 6 This is a schematic diagram of the radar data and video data fusion device according to an embodiment of this application. In this embodiment, the radar data and video data fusion device includes an acquisition module 61, a combination module 62, and a fusion module 63.

[0118] The device comprises several modules: an acquisition module 61, which acquires the target distance between radar targets and video targets in the current frame image, and historical matching data of radar targets and video targets in historical video frames; a combination module 62, which combines the target distance and historical matching data to determine radar targets and video targets belonging to the same object; and a fusion module 63, which fuses the radar data of radar targets belonging to the same object and the video data of video targets. This radar data and video data fusion device considers not only the actual distance between radar and video targets at the current moment, but also the historical matching data of the two targets at previous moments, fusing two targets belonging to the same object. By combining data from the current moment and historical moments, it compensates for potential calibration errors in the current moment data, effectively improving the accuracy of radar target and video target association.

[0119] Please see Figure 7 , Figure 7 This is a schematic diagram of the radar data and video data fusion device according to an embodiment of this application. In this embodiment, the radar data and video data fusion device 71 includes a processor 72.

[0120] Processor 72 can also be referred to as a CPU (Central Processing Unit). Processor 72 may be an integrated circuit chip with signal processing capabilities. Processor 72 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor, or processor 72 can be any conventional processor.

[0121] The radar data and video data fusion device 71 may further include a memory (not shown) for storing instructions and data required for the processor 72 to run.

[0122] The processor 72 is used to execute instructions to implement the method provided by any embodiment and any non-conflicting combination of the radar data and video data fusion method of this application.

[0123] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of a computer-readable storage medium in an embodiment of this application. The computer-readable storage medium 81 in this embodiment stores instruction / program data 82. When executed, this instruction / program data 82 implements the method provided by any embodiment of the radar data and video data fusion method of this application, as well as any non-conflicting combination thereof. The instruction / program data 82 can be formed into a program file and stored in the aforementioned storage medium 81 in the form of a software product, so that a computer device (which may be a personal computer, server, or network device, etc.) or processor can execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium 81 includes various media capable of storing program code, such as a USB flash drive, mobile hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, or terminal devices such as computers, servers, mobile phones, and tablets.

[0124] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0125] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0126] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for fusing radar data and video data, characterized in that, The method includes: Acquire radar targets in radar images and video targets in video images at the current moment, and map the radar targets and video targets to the same coordinate system; In the same coordinate system, the target distance between the radar target and the video target is obtained; and historical matching data of the radar target and the video target at historical times is obtained, the historical matching data being used to characterize the matching status of the radar target and the video target at historical times; Based on the target distance and the historical matching data, radar targets and video targets belonging to the same object are identified; The radar data of radar targets belonging to the same object and the video data of video targets are correlated and fused. The historical matching data includes historical fusion degree and historical trajectory overlap degree. The historical fusion degree is used to characterize the fusion status of the radar target and the video target at a historical moment. The step of determining radar targets and video targets belonging to the same object based on the target distance and the historical matching data includes: The target distance, the historical fusion degree, and the historical trajectory overlap degree are weighted and fused to obtain the matching degree between the radar target and the video target; Based on the matching degree, radar targets and video targets belonging to the same object are identified; The historical fusion degree of the radar target and the video target at the historical moment is determined by the following method: Obtain the reliability of radar video detection at historical moments, as well as the number of radar video interference targets at historical moments; By combining the reliability of the radar video detection and the number of radar video interference targets, the historical fusion degree of the radar target and the video target at the historical moment is obtained.

2. The radar data and video data fusion method according to claim 1, characterized in that, The matching degree is positively correlated with the target distance and the historical trajectory overlap, and negatively correlated with the historical fusion degree. Determining radar targets and video targets belonging to the same object based on the matching degree includes: Obtain the matching degree between each radar target and the video target at the current moment; When the matching degree is at its minimum, the radar target and the video target belong to the same object.

3. The radar data and video data fusion method according to claim 1, characterized in that, The historical fusion degree is directly proportional to the reliability of the radar video detection and inversely proportional to the number of radar video interference targets.

4. The radar data and video data fusion method according to claim 1, characterized in that, The radar target and the video target at the historical moment belong to the same object. The number of radar video interference targets at the historical moment includes: The number of interfering targets within the neighborhood of the fused coordinates is obtained, and the interfering targets include radar targets and video targets; The fused coordinates are the coordinates of the radar target and the video target after successful fusion at the historical moment, and the neighborhood range is the distance between the radar target and the video target at the historical moment ± a predetermined distance.

5. The radar data and video data fusion method according to claim 1, characterized in that, The reliability of radar video detection at historical moments includes: Obtain the detection confidence level of the radar target at the historical time and the detection confidence level of the video target at the historical time; The radar video detection confidence level is obtained by weighted multiplying the detection confidence level of the radar target and the detection confidence level of the video target.

6. The radar data and video data fusion method according to claim 5, characterized in that, The farther the video target is from the video detector, the lower the detection reliability of the video target. As the distance between the radar target and the radar detector increases, the reliability of the radar target detection first increases and then decreases.

7. The radar data and video data fusion method according to claim 1, characterized in that, The degree of overlap in historical trajectories is determined by the following methods: Obtain the positional distance between the radar target and the video target at historical moments; The average distance between the locations at each historical moment is calculated to obtain the historical trajectory overlap between the radar target and the video target.

8. A radar data and video data fusion device, characterized in that, Includes a processor for executing instructions to implement the radar data and video data fusion method as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store instruction / program data that can be executed to implement the radar data and video data fusion method as described in any one of claims 1-7.

Images