Method and device for fusing multi-radio data of urban roads

By transforming and matching the radar data and video data of multiple intersections global coordinate system, the problem of large amount of calculation and low real-time fusion of target fusion of multiple intersections in the prior art is solved, and more efficient data fusion and accuracy are achieved , while reducing hardware processing costs.

CN117541910BActive Publication Date: 2024-11-22BEIJING URBAN PLANNING & DESIGN INST +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311412405.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-10-27
Publication Date
2024-11-22
Estimated Expiration
2043-10-27

AI Technical Summary

Technical Problem

The prior art has large computing volume and low real-time performance in the target data fusion between multiple intersections, and cannot effectively solve the target fusion problem of multiple intersections.

Method used

By transforming the radar data and video data of multiple intersections from their respective sub-coordinate systems to the global coordinate system, and performing data fusion, obtaining the camera target ID and radar target ID, and then matching, the optimal matching algorithm determines the fusion target ID , realize data fusion at multiple intersections.

Benefits of technology

It improves the accuracy and real-timeness of data fusion, reduces the computing volume, and reduces the hardware processing requirements in individual Leishi devices, and reduces the hardware cost.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117541910B_ABST
    Figure CN117541910B_ABST
Patent Text Reader

Abstract

The present invention provides a method and device for fusing multiple radar vision data of urban roads, which relate to the field of intelligent transportation technology. The method comprises: obtaining radar data and video data respectively collected by multiple radar vision devices at multiple intersections; respectively transforming the radar data and video data respectively collected by the multiple radar vision devices from corresponding sub-coordinate systems to global coordinate systems to obtain corresponding transformed video data and transformed radar data; performing data fusion on the transformed video data corresponding to the multiple radar vision devices to obtain at least one camera target ID; performing data fusion on the transformed radar data corresponding to the multiple radar vision devices to obtain at least one radar target ID; matching the at least one camera target ID and the at least one radar target ID to obtain a fused target ID.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation technology, and in particular to a method and device for fusing multi-radar visual data of urban roads. Background Art

[0002] Urban intersections are important nodes for vehicle-road collaboration. At crossroads or multi-road intersections, radar vision devices are often used for holographic applications. In order to cover the entire intersection, a radar vision device needs to be placed at each intersection.

[0003] In the prior art, in order to fuse the target data of multiple intersections, each radar vision device fuses the data of its own target (such as vehicles, pedestrians, and non-motor vehicles) respectively, and the single intersection target fusion adopts the deep learning method, and then the edge computing device performs target fusion between multiple devices again. However, the prior art has the following problems: large amount of calculation, low real-time performance, high hardware computing power and real-time performance requirements of the equipment, and cannot solve the problem of target fusion at multiple intersections. Summary of the invention

[0004] In view of the problems existing in the prior art, the present invention provides a method and device for fusing multi-radar visual data of urban roads.

[0005] The present invention provides a method for fusing multi-radio visual data of urban roads, which is applied to edge devices. The method comprises:

[0006] Obtain radar data and video data collected by multiple radar vision devices at multiple intersections;

[0007] Respectively transforming the radar data and video data collected by the plurality of radar vision devices from the corresponding sub-coordinate system to the global coordinate system to obtain corresponding transformed video data and transformed radar data;

[0008] Performing data fusion on the transformed video data corresponding to each of the plurality of radar vision devices to obtain at least one camera target ID;

[0009] Performing data fusion on the transformed radar data corresponding to each of the plurality of radar vision devices to obtain at least one radar target ID;

[0010] The camera target IDs and the radar target IDs are matched to obtain a fused target ID.

[0011] According to a method for fusing multiple radar vision data of urban roads provided by the present invention, the radar data and video data collected by the multiple radar vision devices are respectively transformed from the corresponding sub-coordinate system to the global coordinate system to obtain the corresponding transformed video data and transformed radar data, including:

[0012] For each of the plurality of radar vision devices, according to a transformation matrix corresponding to the camera of the radar vision device, transform the video data collected by the camera from the sub-coordinate system corresponding to the radar vision device to the global coordinate system associated with the plurality of intersections, so as to obtain the transformed video data corresponding to the radar vision device;

[0013] According to the conversion matrix corresponding to the radar of the radar vision device, the radar data collected by the radar is transformed from the sub-coordinate system corresponding to the radar vision device to the global coordinate system associated with the multiple intersections to obtain the transformed radar data corresponding to the radar vision device.

[0014] According to a method for fusing multi-radio visual data of urban roads provided by the present invention, the method further includes:

[0015] For the camera of each of the multiple radar vision devices, a transformation matrix corresponding to the camera is determined based on a preset corner point; the transformation matrix corresponding to the camera includes homography transformation matrix extrinsics of the camera in different directions.

[0016] According to a method for fusing multi-radio visual data of urban roads provided by the present invention, the method further includes:

[0017] For each radar of the multiple radar vision devices, a conversion matrix corresponding to the radar is determined based on a preset angle inversion; the conversion matrix corresponding to the radar includes homography transformation matrix extrinsics of the radar in different directions.

[0018] According to a method for fusing multiple radar vision data of urban roads provided by the present invention, the method of fusing the transformed video data corresponding to each of the multiple radar vision devices to obtain at least one camera target ID includes:

[0019] According to the transformed video data corresponding to each of the cameras of the plurality of radar vision devices, matching and associating are performed on the same target detected by the cameras of the plurality of radar vision devices to obtain at least one camera target ID.

[0020] According to a method for fusing multiple radar vision data of urban roads provided by the present invention, the transformed radar data corresponding to each of the multiple radar vision devices are fused to obtain at least one radar target ID, including:

[0021] According to the transformed radar data corresponding to the radars of the multiple radar vision devices, matching and association are performed on the same target detected by the radars of the multiple radar vision devices to obtain at least one radar target ID.

[0022] According to a method for fusing multi-radar visual data of urban roads provided by the present invention, the matching of each camera target ID and each radar target ID to obtain a fused target ID includes:

[0023] Based on the optimal matching algorithm, the optimal matching detection result is determined according to each of the camera target IDs and each of the radar target IDs; the optimal matching detection result includes the fused target ID, and the camera target ID and radar target ID that belong to the same fused target ID and have a matching relationship.

[0024] The present invention also provides a device for fusing multi-radar visual data of urban roads, comprising:

[0025] An acquisition module is used to acquire radar data and video data collected by multiple radar vision devices at multiple intersections;

[0026] A transformation module, used to transform the radar data and video data collected by the plurality of radar vision devices respectively from the corresponding sub-coordinate system to the global coordinate system, so as to obtain corresponding transformed video data and transformed radar data;

[0027] A first data fusion module is used to perform data fusion on the transformed video data corresponding to each of the plurality of radar vision devices to obtain at least one camera target ID;

[0028] A second data fusion module is used to perform data fusion on the transformed radar data corresponding to each of the plurality of radar vision devices to obtain at least one radar target ID;

[0029] The matching module is used to match each of the camera target IDs and each of the radar target IDs to obtain a fused target ID.

[0030] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for fusing multi-radar visual data of urban roads as described in any one of the above is implemented.

[0031] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for fusing multi-radar visual data of urban roads as described in any one of the above is implemented.

[0032] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for fusing multi-radar visual data of urban roads as described in any one of the above is implemented.

[0033] The method for fusing multiple radar vision data of urban roads provided by the present invention transmits radar data and video data respectively collected by multiple radar vision devices at multiple intersections to the same edge device, and the edge device first performs coordinate conversion and data fusion on the video data respectively collected by the multiple radar vision devices to obtain at least one camera target ID, and performs coordinate conversion and data fusion on the radar data respectively collected by the multiple radar vision devices to obtain at least one radar target ID; and then the edge device matches each camera target ID and each radar target ID to obtain a fused target ID, so that the radar data and video data respectively collected by the multiple radar vision devices at the multiple intersections are fused by one edge device, the fusion process in a single radar vision device is reduced, homogeneous sensors are fused first, and then heterogeneous sensors are fused, so that the accuracy and real-time performance of the fusion target can be improved, and the amount of calculation can be reduced. Since the fusion processing of radar data and video data is not performed on a single radar vision device, there is no need to configure a separate hardware processing device in a single radar vision device, and the hardware processing cost of the radar vision can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 It is a flow chart of the method for fusing multi-radio visual data of urban roads provided by the present invention;

[0036] Figure 2 is an example schematic diagram of a multi-intersection coordinate system provided by the present invention;

[0037] Figure 3 It is a structural schematic diagram of a device for fusing multi-radar visual data of urban roads provided by the present invention;

[0038] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] The following describes the method and device for fusing multi-radar visual data of urban roads provided by the present invention in conjunction with the accompanying drawings of the specification.

[0041] Figure 1 FIG. 1 is a flow chart of a method for fusing multi-radio visual data of urban roads provided by the present invention. Figure 1 As shown, the method includes steps 101 to 105, wherein:

[0042] Step 101, obtaining radar data and video data collected by multiple radar vision devices at multiple intersections.

[0043] It should be noted that the method for fusing multi-radar vision data of urban roads provided by the present invention can be applied in the fields of intelligent transportation, industrial detection and autonomous driving. The method for fusing multi-radar vision data of urban roads provided by the present invention can be applicable to scenarios where it is necessary to perform target fusion on the data of multiple radar vision devices at multiple intersections. The execution subject of the method can be an edge device or a fusion device for multi-radar vision data of urban roads, such as an electronic device, a server, or a control module in the device for executing the method for fusing multi-radar vision data of urban roads provided by the present invention, wherein the electronic device can include but is not limited to a mobile phone, a tablet computer or a desktop computer.

[0044] Optionally, radar targets may be detected based on radar data, and radar targets may include vehicles, pedestrians, and / or non-motor vehicles. The radar data includes radar coordinates corresponding to the radar targets. Camera targets may be detected based on video data, and camera targets may include vehicles, pedestrians, and / or non-motor vehicles. The video data includes video coordinates or detection frame coordinates corresponding to the camera targets.

[0045] Optionally, at least one radar vision device is deployed at each of the multiple intersections, and different radar vision devices at different intersections transmit their collected radar data and video data to the same edge device, which performs target fusion.

[0046] Step 102 , respectively transforming the radar data and video data collected by the plurality of radar vision devices from the corresponding sub-coordinate system to the global coordinate system to obtain corresponding transformed video data and transformed radar data.

[0047] Optionally, after acquiring the radar data and video data collected by multiple radar vision devices, the edge device first performs a coordinate transformation, transforming the radar data and video data collected by the multiple radar vision devices from the corresponding sub-coordinate system to the global coordinate system to obtain the corresponding transformed video data and transformed radar data.

[0048] That is, the radar data collected by each of the multiple radar vision devices is transformed from the sub-coordinate system corresponding to the corresponding radar vision device to the global coordinate system to obtain the corresponding transformed radar data; and the video data collected by each of the multiple radar vision devices is transformed from the sub-coordinate system corresponding to the corresponding radar vision device to the global coordinate system to obtain the corresponding transformed video data.

[0049] Optionally, a specific implementation method of step 102 may include: for each of the multiple radar vision devices, according to the transformation matrix corresponding to the camera of the radar vision device, transforming the video data collected by the camera from the sub-coordinate system corresponding to the radar vision device to the global coordinate system associated with the multiple intersections, so as to obtain the transformed video data corresponding to the radar vision device; and, according to the transformation matrix corresponding to the radar of the radar device, transforming the radar data collected by the radar from the sub-coordinate system corresponding to the radar vision device to the global coordinate system associated with the multiple intersections, so as to obtain the transformed radar data corresponding to the radar vision device.

[0050] Optionally, the present invention provides a method for obtaining a transformation matrix corresponding to a camera of a radar vision device, specifically: for each camera of a plurality of radar vision devices, based on preset corner points, determining a transformation matrix corresponding to the camera; the transformation matrix corresponding to the camera includes homography transformation matrix extrinsics of different directions of the camera.

[0051] Optionally, the present invention provides a method for obtaining a transformation matrix corresponding to a radar of a radar vision device, specifically: for each radar of a plurality of radar vision devices, based on a preset angle inversion, determining a transformation matrix corresponding to the radar; the transformation matrix corresponding to the radar includes homography transformation matrix extrinsics of different directions of the radar.

[0052] Step 103 , data fusion is performed on the transformed video data corresponding to each of the plurality of radar vision devices to obtain at least one camera target ID.

[0053] Optionally, a specific implementation of step 103 may include: matching and associating the same target detected by the cameras of the multiple radar vision devices according to the transformed video data corresponding to each of the cameras of the multiple radar vision devices, to obtain at least one camera target ID.

[0054] Step 104 , performing data fusion on the transformed radar data corresponding to each of the plurality of radar vision devices to obtain at least one radar target ID.

[0055] Optionally, a specific implementation of step 104 may include: matching and associating the same target detected by the radars of the multiple radar vision devices according to the transformed radar data corresponding to the radars of the multiple radar vision devices, to obtain at least one radar target ID.

[0056] Step 105, matching each camera target ID and each radar target ID to obtain a fused target ID.

[0057] Optionally, a specific implementation of step 105 may include: determining an optimal matching detection result based on each camera target ID and each radar target ID based on an optimal matching algorithm; the optimal matching detection result includes a fused target ID, and camera target IDs and radar target IDs that have a matching relationship and belong to the same fused target ID. For example, the optimal matching algorithm may include an improved Hungarian algorithm.

[0058] The method for fusing multiple radar vision data of urban roads provided by the present invention transmits radar data and video data respectively collected by multiple radar vision devices at multiple intersections to the same edge device, and the edge device first performs coordinate conversion and data fusion on the video data respectively collected by the multiple radar vision devices to obtain at least one camera target ID, and performs coordinate conversion and data fusion on the radar data respectively collected by the multiple radar vision devices to obtain at least one radar target ID; and then the edge device matches each camera target ID and each radar target ID to obtain a fused target ID, so that the radar data and video data respectively collected by the multiple radar vision devices at the multiple intersections are fused by one edge device, the fusion process in a single radar vision device is reduced, homogeneous sensors are fused first, and then heterogeneous sensors are fused, so that the accuracy and real-time performance of the fusion target can be improved, and the amount of calculation can be reduced. Since the fusion processing of radar data and video data is not performed on a single radar vision device, there is no need to configure a separate hardware processing device in a single radar vision device, and the hardware processing cost of the radar vision can be reduced.

[0059] Here, a crossroad is taken as an example to describe the method for fusing multi-radar visual data of urban roads provided by the present invention. Figure 2 It is an example schematic diagram of the multi-intersection coordinate system provided by the present invention. Figure 2 The large coordinate system xoy of the intersection is shown, that is, the global coordinate system xoy associated with multiple intersections. Figure 2 It also shows that the radar coordinate system of intersection 1 is x1o1y1 (i.e. the sub-coordinate system x1o1y1 corresponding to the radar device at intersection 1), the radar coordinate system of intersection 2 is x2o2y2 (i.e. the sub-coordinate system x2o2y2 corresponding to the radar device at intersection 2), the radar coordinate system of intersection 3 is x3o3y3 (i.e. the sub-coordinate system x3o3y3 corresponding to the radar device at intersection 3), and the radar coordinate system of intersection 4 is x4o4y4 (i.e. the sub-coordinate system x4o4y4 corresponding to the radar device at intersection 4).

[0060] (1) In the present invention, the fusion processing of radar data (such as millimeter-wave radar data) and video data (such as camera data) is not performed on a single radar device (i.e., a radar machine) (i.e., there is no separate hardware processing device in a single radar machine), and the raw data of the millimeter-wave radar and camera on the radar machine are uniformly transmitted to the same edge device, and then the homogeneous sensors are fused in the edge device (each fusion of multi-intersection camera data and each fusion of multi-intersection radar data), and then the heterogeneous sensors are fused with each other.

[0061] In the present invention, homogeneous sensors at different intersections are fused separately, and heterogeneous sensors are fused again. There are separate cameras and radars at each intersection. For example, at a certain intersection, there are 4 radar cameras, that is, 4 cameras and 4 radars. Then the 4 cameras are homogeneous sensors, the 4 radars are homogeneous sensors, and the cameras and radars are heterogeneous sensors.

[0062] (2) After the edge device obtains the raw data from the camera and radar of each radar camera, it then obtains the transformation matrix corresponding to the radar camera at each intersection and transforms the raw data from the camera and radar of each radar camera from the sub-coordinate system to the global coordinate system using the transformation matrix.

[0063] Multiple intersection radar cameras are each used as a sub-coordinate system. Through the conversion matrix, that is, the coordinate mapping relationship between the sub-coordinate system and the global coordinate system, the radar camera at intersection 1 (facing north), the radar camera at intersection 2 (facing east), the radar camera at intersection 3 (facing south), and the radar camera at intersection 4 (facing west) are obtained. The homography transformation matrix extrinsic parameters of each radar camera selected by the preset corner point (appropriate corner point) are used, and the extrinsic parameters of the homography transformation matrix of each radar radar are obtained by using the angle back-calibration reflector.

[0064] Specifically, firstly, the pixel coordinates of obvious corner points on the urban road are obtained; then, the coordinates of the target calibration points of the lane line are determined using the coordinates of the corner points on the road; then, the perspective transformation is performed on the selected pixel coordinates to obtain the homography transformation matrix.

[0065] Taking a certain intersection as an example, some corner points are selected as calibration corner points to obtain the h of the camera in different directions. 11 、h 12 、h 13 and h 14 Matrix, use angle inversion to get the R11, R12, R13 and R14 matrices in different directions of the radar.

[0066] For example, the radar coordinates of the radar imaging machine (facing south) at intersection 3 are frameRadar=[Radar1…Radarm], Radar1 is the radar coordinates of the first target, m is the number of targets, and the homography transformation matrix obtained above is used to perform homography transformation on each target to obtain the transformed radar detection coordinates Framesn=[Radar1…Radarm].

[0067] Specifically, the radar coordinates of Radar1 are substituted into x and y in Formula 1 to obtain (X, Y) and assigned to the transformed radar coordinates Radar1 to obtain data data_east in the global coordinate system.

[0068]

[0069] Optionally, Equation 1 uses h at different camera orientations 21 、h 22 、h 23 、h 31 and h 32 matrix.

[0070] Optionally, obtain the current frame data (radar-vision fusion data after coordinate conversion); obtain the target coordinates of intersection 1, intersection 2, intersection 3 and intersection 4; use the KM algorithm to determine the matching relationship between the targets at the four intersections; determine whether the multi-sensor data is successfully matched; for the successfully matched multi-sensor target pairs, select the target data with the front intersection number as the fusion data; or, for the targets that are not successfully matched, retain the target data; then, determine the fusion data of the four targets based on the fusion data and the retained target data, and then update the multi-sensor feature library, and update the tracking filter and track.

[0071] (3) Define a large area for global target integration. Figure 2 In the intersection area shown, the same target vehicle will be detected by multiple radars at the same time, so there will be multiple detection frame values ​​and coordinate values. Therefore, it is necessary to perform multi-channel camera processing and multi-channel radar target fusion processing separately within the area, and then perform radar and camera target fusion processing.

[0072] (4) The central intersection fusion area is generally covered by radar cameras in the four directions of east, west, south and north. This time, multiple cameras match and associate the same target to obtain the camera target ID, and multiple radars match and associate the same target to obtain the radar target ID. The results of the fusion of cameras and radars for the same target are processed. At this time, the optimal association matrix is ​​solved, and the optimal matching algorithm (such as the improved Hungarian algorithm) is used for multiple detection results of the same target. Multiple detected shapes, center point ranges, and IOU matching are used to score and obtain the optimal matching detection results, thereby obtaining the detected and tracked matching relationship. By fusing the data after matrix conversion and performing asynchronous fusion association matching, the unique fusion result can be solved according to the association relationship, and the fusion target ID can be obtained according to the matching relationship.

[0073] Optionally, the association matrix A is obtained according to a specific association algorithm (such as a nearest neighbor data association algorithm) and the radar data and the camera data:

[0074]

[0075] Among them, i is the target number detected by the radar, j is the target number detected by the camera, nr is the maximum target number detected by the radar, nc is the maximum target number detected by the camera, and a i,j is defined as follows:

[0076] a i,j =1 / F i,j , F i,j The distance f between the radar and the camera i,j The specific process is as follows:

[0077] Calculate first

[0078] Then f i,j Compare with the threshold distance;

[0079] if

[0080] Input the association matrix A into the improved Hungarian KM algorithm to maximize the sum of multiple elements. The formula is as follows:

[0081]

[0082] Among them, H(i,j) satisfies the following relationship: H(i,j) = 0 or 1: When H(i,j) takes the value of 0, it means that the two targets are not optimally matched; when H(i,j) takes the value of 1, it means that the two targets are optimally matched;

[0083]

[0084] Thus, the pairing relationship between the remaining radar targets and the remaining camera targets is obtained.

[0085] (5) At this point, the global target fusion is completed, and the total fusion target data of the intersection can be obtained.

[0086] The present invention utilizes a method for fusing radar edge data, rather than fusing individual radar vision data first and then fusing them, and the fusion accuracy is higher: the method for fusing radar edge data is utilized to reduce the fusion within individual radar vision hardware processing equipment, so that homogeneous sensors can have the same target characteristics, and can be fused separately first, and then heterogeneous sensors can be fused, thereby improving the fusion target accuracy.

[0087] Lower cost: The method of edge data fusion of the radar vision can reduce the data processing flow of each individual radar vision, thus reducing the hardware processing cost of the radar vision.

[0088] It can be seen that the method for fusing multiple radar vision data of urban roads provided by the present invention can improve the accuracy and real-time performance of radar vision target fusion at crossroads or intersections, and completes the target fusion at intersections by utilizing the target correspondence and algorithm of radar vision fusion at intersections in multiple directions.

[0089] The following is a description of a fusion device for urban road multi-radar visual data provided by the present invention. The fusion device for urban road multi-radar visual data described below and the fusion method for urban road multi-radar visual data described above can be referred to each other.

[0090] Figure 3 300 is a schematic diagram of the structure of the fusion device of multi-radar visual data of urban roads provided by the present invention; the fusion device 300 of multi-radar visual data of urban roads comprises: an acquisition module 301, a transformation module 302, a first data fusion module 303, a second data fusion module 304 and a matching module 305; wherein,

[0091] An acquisition module 301 is used to acquire radar data and video data collected by multiple radar vision devices at multiple intersections;

[0092] The transformation module 302 is used to transform the radar data and the video data collected by the plurality of radar vision devices from the corresponding sub-coordinate system to the global coordinate system to obtain the corresponding transformed video data and transformed radar data;

[0093] A first data fusion module 303 is used to perform data fusion on the transformed video data corresponding to each of the plurality of radar vision devices to obtain at least one camera target ID;

[0094] A second data fusion module 304 is used to perform data fusion on the transformed radar data corresponding to each of the plurality of radar vision devices to obtain at least one radar target ID;

[0095] The matching module 305 is used to match each of the camera target IDs and each of the radar target IDs to obtain a fused target ID.

[0096] The device for fusing multiple radar vision data of urban roads provided by the present invention transmits radar data and video data respectively collected by multiple radar vision devices at multiple intersections to the same edge device, and the edge device first performs coordinate conversion and data fusion on the video data respectively collected by the multiple radar vision devices to obtain at least one camera target ID, and performs coordinate conversion and data fusion on the radar data respectively collected by the multiple radar vision devices to obtain at least one radar target ID; then the edge device matches each camera target ID and each radar target ID to obtain a fused target ID, so that the radar data and video data respectively collected by the multiple radar vision devices at the multiple intersections are fused by one edge device, the fusion process in a single radar vision device is reduced, homogeneous sensors are fused first, and then heterogeneous sensors are fused, so that the accuracy and real-time performance of the fusion target can be improved, and the amount of calculation can be reduced. Since the fusion processing of radar data and video data is not performed on a single radar vision device, there is no need to configure a separate hardware processing device in a single radar vision device, which can reduce the hardware processing cost of the radar vision.

[0097] Optionally, the transformation module 302 is specifically configured to:

[0098] For each of the plurality of radar vision devices, according to a transformation matrix corresponding to the camera of the radar vision device, transform the video data collected by the camera from the sub-coordinate system corresponding to the radar vision device to the global coordinate system associated with the plurality of intersections, so as to obtain the transformed video data corresponding to the radar vision device;

[0099] According to the conversion matrix corresponding to the radar of the radar vision device, the radar data collected by the radar is transformed from the sub-coordinate system corresponding to the radar vision device to the global coordinate system associated with the multiple intersections to obtain the transformed radar data corresponding to the radar vision device.

[0100] Optionally, the urban road multi-radio data fusion device further includes:

[0101] The first determination module is used to determine the transformation matrix corresponding to the camera of each of the multiple radar vision devices based on the preset corner points; the transformation matrix corresponding to the camera includes the homography transformation matrix extrinsic parameters of the camera in different directions.

[0102] Optionally, the urban road multi-radio data fusion device further includes:

[0103] The second determination module is used to determine the transformation matrix corresponding to each radar of the multiple radar vision devices based on a preset angle inversion; the transformation matrix corresponding to the radar includes the homography transformation matrix extrinsic parameters of the radar in different directions.

[0104] Optionally, the first data fusion module 303 is specifically used to match and associate the same target detected by the cameras of the multiple radar vision devices according to the transformed video data corresponding to each of the cameras of the multiple radar vision devices, so as to obtain at least one camera target ID.

[0105] Optionally, the second data fusion module 304 is specifically used to match and associate the same target detected by the radars of the multiple radar vision devices according to the transformed radar data corresponding to each of the radars of the multiple radar vision devices, so as to obtain at least one radar target ID.

[0106] Optionally, the matching module 305 is specifically used to: determine the optimal matching detection result based on the optimal matching algorithm according to each of the camera target IDs and each of the radar target IDs; the optimal matching detection result includes the fused target ID, and the camera target ID and radar target ID that belong to the same fused target ID and have a matching relationship.

[0107] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute a method for fusing multiple radar vision data of urban roads, the method comprising: obtaining radar data and video data collected by multiple radar vision devices at multiple intersections; transforming the radar data and video data collected by the multiple radar vision devices from the corresponding sub-coordinate system to the global coordinate system to obtain corresponding transformed video data and transformed radar data; fusing the transformed video data corresponding to the multiple radar vision devices to obtain at least one camera target ID; fusing the transformed radar data corresponding to the multiple radar vision devices to obtain at least one radar target ID; matching the camera target IDs and the radar target IDs to obtain a fused target ID.

[0108] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0109] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the method for fusing multiple radar vision data of urban roads provided by the above methods, and the method includes: obtaining radar data and video data collected by multiple radar vision devices at multiple intersections; respectively transforming the radar data and video data collected by the multiple radar vision devices from the corresponding sub-coordinate system to the global coordinate system to obtain corresponding transformed video data and transformed radar data; performing data fusion on the transformed video data corresponding to each of the multiple radar vision devices to obtain at least one camera target ID; performing data fusion on the transformed radar data corresponding to each of the multiple radar vision devices to obtain at least one radar target ID; matching each of the camera target IDs and each of the radar target IDs to obtain a fused target ID.

[0110] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the method for fusing multiple radar vision data of urban roads provided by the above-mentioned methods. The method includes: obtaining radar data and video data collected by multiple radar vision devices at multiple intersections; transforming the radar data and video data collected by the multiple radar vision devices from the corresponding sub-coordinate system to the global coordinate system to obtain corresponding transformed video data and transformed radar data; performing data fusion on the transformed video data corresponding to each of the multiple radar vision devices to obtain at least one camera target ID; performing data fusion on the transformed radar data corresponding to each of the multiple radar vision devices to obtain at least one radar target ID; matching each of the camera target IDs and each of the radar target IDs to obtain a fused target ID.

[0111] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0112] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for fusing multi-radio data of urban roads. It is characterized in that Applied to an edge device, the method comprises: Obtain radar data and video data collected by multiple radar vision devices at multiple intersections; Respectively transforming the radar data and video data collected by the plurality of radar vision devices from the corresponding sub-coordinate system to the global coordinate system to obtain corresponding transformed video data and transformed radar data; According to the transformed video data corresponding to the cameras of the multiple radar vision devices, matching and associating the same target detected by the cameras of the multiple radar vision devices to obtain at least one camera target ID; According to the transformed radar data corresponding to the radars of the multiple radar vision devices, matching and associating the same target detected by the radars of the multiple radar vision devices to obtain at least one radar target ID; Based on the optimal matching algorithm, according to each of the camera target IDs and each of the radar target IDs, an optimal matching detection result is determined; the optimal matching detection result includes a fused target ID, and a camera target ID and a radar target ID that belong to the same fused target ID and have a matching relationship; specifically, the method includes: obtaining an association matrix A based on each of the camera target IDs and each of the radar target IDs, ; in, is the target number detected by the radar, is the target number detected by the camera, nr is the maximum target number detected by the radar, nc is the maximum target number detected by the camera, and is defined as follows: , The distance between the radar and the camera and threshold distance jointly determined; in, ; ; The incidence matrix Input into the improved Hungarian algorithm to maximize the sum of multiple elements. The formula is as follows: ; in, Satisfies the following relationship: =0 or 1: When the value is 0, it means that the two targets are not optimally matched; When the value is 1, it means that the two targets achieve the best match; ; The fused target ID, and the camera target ID and radar target ID that have a matching relationship and belong to the same fused target ID are obtained.

2. The method for fusing multi-radio data of urban roads according to claim 1, It is characterized in that The step of transforming the radar data and the video data collected by the plurality of radar vision devices from the corresponding sub-coordinate system to the global coordinate system to obtain the corresponding transformed video data and transformed radar data comprises: For each of the plurality of radar vision devices, according to a transformation matrix corresponding to the camera of the radar vision device, transform the video data collected by the camera from the sub-coordinate system corresponding to the radar vision device to the global coordinate system associated with the plurality of intersections, so as to obtain the transformed video data corresponding to the radar vision device; According to the conversion matrix corresponding to the radar of the radar vision device, the radar data collected by the radar is transformed from the sub-coordinate system corresponding to the radar vision device to the global coordinate system associated with the multiple intersections to obtain the transformed radar data corresponding to the radar vision device.

3. The method for fusing multi-radio data of urban roads according to claim 2, It is characterized in that The method further comprises: For the camera of each of the multiple radar vision devices, a transformation matrix corresponding to the camera is determined based on a preset corner point; the transformation matrix corresponding to the camera includes homography transformation matrix extrinsics of the camera in different directions.

4. The method for fusing multi-radio data of urban roads according to claim 2, It is characterized in that The method further comprises: For each radar of the multiple radar vision devices, a conversion matrix corresponding to the radar is determined based on a preset angle inversion; the conversion matrix corresponding to the radar includes homography transformation matrix extrinsics of the radar in different directions.

5. A device for fusing multi-radar data of urban roads, It is characterized in that include: An acquisition module is used to acquire radar data and video data collected by multiple radar vision devices at multiple intersections; A transformation module, used to transform the radar data and video data collected by the plurality of radar vision devices respectively from the corresponding sub-coordinate system to the global coordinate system, so as to obtain corresponding transformed video data and transformed radar data; A first data fusion module is used to match and associate the same target detected by the cameras of the multiple radar vision devices according to the transformed video data corresponding to each of the cameras of the multiple radar vision devices, so as to obtain at least one camera target ID; A second data fusion module is used to match and associate the same target detected by the radars of the multiple radar vision devices according to the transformed radar data corresponding to each of the radars of the multiple radar vision devices, so as to obtain at least one radar target ID; The matching module is used to determine the best matching detection result based on the best matching algorithm according to each of the camera target IDs and each of the radar target IDs; the best matching detection result includes the fused target ID, and the camera target ID and the radar target ID that have a matching relationship and belong to the same fused target ID; specifically, it is used to obtain the association matrix A based on each of the camera target IDs and each of the radar target IDs, ; in, is the target number detected by the radar, is the target number detected by the camera, nr is the maximum target number detected by the radar, nc is the maximum target number detected by the camera, and is defined as follows: , The distance between the radar and the camera and threshold distance jointly determined; in, ; ; The incidence matrix Input into the improved Hungarian algorithm to maximize the sum of multiple elements. The formula is as follows: ; in, Satisfies the following relationship: =0 or 1: When the value is 0, it means that the two targets are not optimally matched; When the value is 1, it means that the two targets achieve the best match; ; The fused target ID, and the camera target ID and radar target ID that have a matching relationship and belong to the same fused target ID are obtained.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the method for fusing multi-radar visual data of urban roads as described in any one of claims 1 to 4 is implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the method for fusing multi-radar visual data of urban roads as claimed in any one of claims 1 to 4 is implemented.