Target detection device based on radar and vision fusion

Abstract

The target detection device based on radar and vision fusion provided by the embodiment of the application comprises a reconstruction unit and a detection unit, wherein firstly, the spatial position of a potential target is characterized according to a millimeter wave radar, and the characterization result is used for division of a long-distance target region in an original image collected by a camera, further, the divided region image is reconstructed and detected, so as to improve the visual detection precision of the long-distance target, then modeling is performed from the fusion of radar and visual detection features, and according to the characteristics that the millimeter wave radar detection is less affected by light and the image detection has more texture information, the detection precision of the system in a low-light environment is improved.

Description

Technical Field

[0001] The embodiments of this application relate to the field of machine vision, and more specifically, the embodiments of this application relate to a target detection device based on radar-visual fusion. Background Technology

[0002] Driven by the demand for intelligent traffic management, higher requirements are being placed on target detection technologies for vehicles, ships, and other transportation tools.

[0003] In related technologies, cameras and millimeter-wave radar are commonly used to detect vehicles. While cameras can capture images that provide detailed texture information of objects and the environment, their detection performance is poor in low-light conditions. Although millimeter-wave radar is not affected by light, has strong anti-interference capabilities, and a long detection range, it cannot obtain texture information of objects, which greatly reduces the accuracy of detection.

[0004] Therefore, target detection at long distances and in low light conditions is a huge challenge for intelligent transportation systems. Summary of the Invention

[0005] This application provides a target detection device based on radar-visual fusion to solve the technical problem of low accuracy in target detection in current intelligent transportation systems.

[0006] In a first aspect of the embodiments of this application, a target detection device based on radar-visual fusion is provided, comprising:

[0007] The reconstruction unit is used to reconstruct the original image based on the detection result of the millimeter-wave radar of the target to be detected, and obtain the image to be detected, wherein the image to be detected contains the original image and at least one target detection region, and the target detection region contains at least one target to be detected.

[0008] The detection unit is used to input the image to be detected into the radar vision detection network, and to detect the target to be detected in the target detection area through the radar vision detection network to obtain the feature image output by the radar vision detection network. The feature image includes the target detection result of the target to be detected.

[0009] In some alternative implementations, the following are included:

[0010] The restoration unit is used to restore the feature image based on the target detection region corresponding to the feature image and the original image to obtain a target image, wherein the region where the target to be detected is located in the target image is the same as the region where the target to be detected is located in the original image.

[0011] The detection results include the width and height of the target detection region in the original image, as well as the coordinate information of the target detection region in the original image.

[0012] The determining unit is configured to determine the width and height of the target detection region in the original image according to the following formula:

[0013] Where w is the width of the target detection region in the image to be detected, h is the height of the target detection region in the image to be detected, and λ is the actual width of the target to be detected. denoted as the actual height of the target to be detected, f is the focal length of the camera used to acquire the original image, and d is the distance of the target to be detected detected by the millimeter-wave radar.

[0014] In some optional implementations, the reconstruction unit is specifically used to: determine the second size information of the target detection region in the image to be detected based on the first size information of the original image, the number of target detection regions in the original image, and the preset distance between the target detection regions;

[0015] Based on the third size information of the original image and the second size information of the target detection region, the fourth size information of the original image in the image to be detected is determined;

[0016] Based on the second size information of the target detection region and the fourth size information of the original image in the image to be detected, the first coordinate information of the target detection region in the image to be detected and the second coordinate information of the original image in the image to be detected are determined.

[0017] The image to be detected is obtained based on the first coordinate information and the second coordinate information.

[0018] In some optional implementations, the reconstruction unit is specifically used to: determine the second size information of the target detection region in the image to be detected according to the following formula:

[0019]

[0020] Where cut.size2 is the second size information of the target detection region, img.size1 is the first size information of the original image, n is the number of target detection regions in the original image, and v is the preset distance between target detection regions.

[0021] In some optional implementations, the reconstruction unit is specifically used to: determine the fourth size information of the original image in the image to be detected according to the following formula:

[0022] orisin.size4=img.size3-cut.size2

[0023] Wherein, orisin.size4 is the fourth size information of the original image in the image to be detected, and img.size3 is the third size information of the original image.

[0024] In some optional implementations, the reconstruction unit is specifically used to: obtain the first coordinate information of the target detection region in the image to be detected according to the following formula:

[0025]

[0026] The second coordinate information of the original image in the image to be detected is obtained according to the following formula:

[0027]

[0028] Where i is the index of the target detection region. The first coordinate information of the target detection region in the image to be detected; The second coordinate information of the original image in the image to be detected.

[0029] In some optional implementations, the radar-visual detection network includes a millimeter-wave radar feature extraction unit, a first visual extraction unit, a second visual extraction unit, and an adaptive feature fusion unit.

[0030] The detection unit is specifically used to: extract features from the image to be detected through the millimeter-wave radar feature extraction unit to obtain a millimeter-wave radar feature map and a first weight coefficient corresponding to the millimeter-wave radar feature map;

[0031] The first visual extraction unit extracts low-level visual features from the image to be detected, and obtains a low-level visual feature map and a second weighting coefficient corresponding to the low-level visual feature map.

[0032] The second visual extraction unit extracts high-level visual features from the image to be detected, and obtains a high-level visual feature map and a third weighting coefficient corresponding to the high-level visual feature map.

[0033] The adaptive feature fusion unit performs adaptive feature fusion processing on the millimeter-wave radar feature map, the first weight coefficient, the low-level visual feature map, the second weight coefficient, the high-level visual feature map, and the third weight coefficient to obtain the feature image.

[0034] In some optional implementations, the restoration unit is specifically used to: obtain the target image according to the following formula:

[0035]

[0036] in, The coordinates of the detection box to be reconstructed from the image to be detected in the original image. Let be the coordinates of the detection box within the target detection region, where the detection box includes the target detection result. Here, v represents the coordinate information of the target detection region in the original image, v represents the preset distance between target detection regions, and cut.size2 represents the second size information of the target detection region.

[0037] The target detection device based on radar-visual fusion provided in this application includes a reconstruction unit and a detection unit. First, it starts from the data level, characterizing the spatial position of potential targets based on millimeter-wave radar, and using the characterization results to divide the distant target region in the original image captured by the camera. Further, the image of the divided region is reconstructed and detected, thereby improving the visual detection accuracy of distant targets. Then, it performs modeling by fusing radar and visual detection feature layers. Taking advantage of the characteristics of millimeter-wave radar detection being less affected by illumination and the abundance of image detection texture information, it improves the detection accuracy of the system in low-light environments. Attached Figure Description

[0038] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of this application are illustrated in the drawings by way of example and not limitation, in which:

[0039] Figure 1 A schematic diagram of the application scenario provided for the implementation of this application;

[0040] Figure 2 A flowchart illustrating a moving target recognition method provided in an embodiment of this application;

[0041] Figure 3 A schematic diagram of the moving target recognition method provided in the embodiments of this application. Figure 1 ;

[0042] Figure 4 A schematic diagram of the principle of the moving target recognition method provided in the embodiments of this application. Figure 2 ;

[0043] Figure 5 Schematic diagram of the structure and principle of the radar detection network provided in the embodiments of this application Figure 1 ;

[0044] Figure 6 Schematic diagram of the structure and principle of the radar detection network provided in the embodiments of this application Figure 2 ;

[0045] Figure 7 This is a schematic diagram of the structure of the storage medium provided in the embodiments of this application;

[0046] Figure 8 This is a schematic diagram of the target detection device provided in the embodiments of this application;

[0047] Figure 9 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0048] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0049] The principles and spirit of this application will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are provided merely to enable those skilled in the art to better understand and implement this application, and are not intended to limit the scope of this application in any way. Rather, these embodiments are provided to make this application more thorough and complete, and to fully convey the scope of this application to those skilled in the art.

[0050] Those skilled in the art will understand that the embodiments of this application can be implemented as a system, apparatus, device, method, or computer program product. Therefore, this application can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0051] Furthermore, the number of any elements in the accompanying drawings is for illustrative purposes only and not for limitation, and any naming is for distinction only and has no limiting meaning. The principles and spirit of this application are explained in detail below with reference to several representative embodiments.

[0052] Driven by the demand for intelligent traffic management, target detection technology for vehicles, ships, and other transportation tools has developed rapidly. However, in harsh environments such as low light, poor visibility limits the performance of target detection based solely on visual images. Meanwhile, practical engineering deployments require long-range detection coverage to reduce the number of devices needed, achieving cost reduction and efficiency improvement. However, when targets are too far away, their limited pixel count in the image significantly reduces detection accuracy. In summary, target detection at long distances and in low light conditions poses a significant challenge to target detection in intelligent transportation systems.

[0053] In existing intelligent transportation systems, cameras and millimeter-wave radar are commonly used in target detection. While cameras can capture detailed texture information of objects and their environment, their detection performance is poor in low-light conditions. Millimeter-wave radar, although unaffected by light, has strong anti-interference capabilities, and a long detection range, cannot acquire texture information. Therefore, a complementary radar-visual fusion approach, combining cameras and millimeter-wave radar, is beneficial for improving the accuracy and robustness of the detection system. In recent years, research on radar-visual fusion methods has gradually increased. From the perspective of fusion methods, they are mainly divided into decision-level fusion, data-level fusion, and feature-level fusion. Among them, the radar-visual decision-level fusion approach involves sensors first detecting and obtaining preliminary data, then sending the data from each sensor to a decision terminal. By analyzing multiple sets of data, the final output is the target information.

[0054] For decision-level radar-visual fusion methods, the radar-measured target distance is associated with the same target in the image, providing additional distance information for image detection. However, it does not consider the issue of low accuracy in low-light and long-distance camera detection based on radar information. The radar-visual data-level fusion approach uses millimeter-wave radar to generate a list of target boxes, where the boxes represent hypotheses about the existence of targets, and then uses a visual detection system to verify these hypotheses.

[0055] The feature-level fusion approach of Rayvision feeds data from both millimeter-wave radar and visual sensors into the network, combining features from both to output detection results. Decision-level and data-level fusion often only detect targets that can be identified by both millimeter-wave radar and cameras, while feature-level fusion extracts features from both data, allowing the detection model to learn the correlation between the two. Even if one sensor performs poorly, data from the other sensor can be used to supplement it, making it the most effective method for simultaneously utilizing information from both millimeter-wave radar and cameras.

[0056] In view of this, embodiments of this application provide a mobile target recognition method, device, and storage medium based on radar-visual fusion. First, starting from the data level, the spatial position of potential targets is characterized based on millimeter-wave radar, and the characterization results are used to divide the distant target region in the image acquired by the camera. Further, the image of the divided region is reconstructed and detected, thereby improving the visual detection accuracy of distant targets. Then, modeling is performed by fusion of radar-visual detection features, taking advantage of the characteristics of millimeter-wave radar detection being less affected by illumination and the abundance of texture information in image detection, thereby improving the detection accuracy of the system in low-light environments.

[0057] First refer to Figure 1 , Figure 1 This is a schematic diagram of an application scenario provided for an embodiment of this application. The device involved in this application scenario includes a target to be detected, a millimeter-wave radar, a camera unit, and a computing module.

[0058] In practical applications, the camera module can connect to the computing module via a network interface, thereby sending the images it has acquired to the computing module through the network interface. The millimeter-wave radar can connect to the GPIO input / output interface of the computing module via a CAN transceiver, thereby sending the millimeter-wave radar signals it has acquired to the computing module.

[0059] Accordingly, the calculation module is used to perform target detection according to the moving target recognition method provided in the embodiments of this application, and output an image containing the detection results.

[0060] In some specific implementations, an Nvidia AGX Xavier embedded device can be used as the computing module. The computing module is connected to a CAN transceiver through its GPIO expansion port. The CAN transceiver is then connected to a millimeter-wave radar for communication. The camera unit and the computing module's built-in RJ45 gigabit Ethernet port are connected via a network cable to complete communication between them.

[0061] It should be understood that the types of computing modules are not particularly limited in the embodiments of this application. Computing modules may include, for example, personal digital assistant (PDA) devices, handheld devices with wireless communication capabilities (e.g., smartphones, tablets), computing devices (e.g., personal computers, PCs), in-vehicle devices, wearable devices (e.g., smartwatches, smart bracelets), smart in-vehicle devices (e.g., smart display devices), etc., and are not particularly limited in the embodiments of this application.

[0062] The embodiments of this application can be applied to various target detection scenarios. For example, in the intelligent transportation scenario, vehicles, ships and other means of transportation are detected. The "target" mentioned in the embodiments of this application refers to vehicles, ships and other means of transportation.

[0063] In some alternative implementations, the application scenario may also include a power module for supplying power to one or more of the computing module, camera module, and millimeter-wave radar.

[0064] The following is combined with Figure 1 Application scenarios, refer to Figure 2-6 This application describes a moving target recognition method according to exemplary embodiments thereof. It should be noted that the above application scenarios are shown only to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way. Rather, the embodiments of this application can be applied to any applicable scenario.

[0065] refer to Figure 2 , Figure 2A flowchart illustrating the moving target recognition method provided in the embodiments of this application. Figure 1 .like Figure 2 As shown, the moving target recognition method includes the following steps:

[0066] S201. Based on the detection results of the millimeter-wave radar on the target to be detected, the original image is reconstructed to obtain the image to be detected.

[0067] The image to be detected includes the original image and at least one target detection region, and the target detection region contains at least one target to be detected.

[0068] For example, Figure 3 A schematic diagram of the principle of the moving target recognition method provided in the embodiments of this application. Figure 1 .like Figure 3 As shown, P1 is the original image captured by the camera unit, and C1, C2, ..., Cn are the target to be detected captured by the millimeter-wave radar, mapped onto the original image and divided into target candidate regions.

[0069] In this step, the enlarged target detection area is stitched together with the reduced original image P1 to form the image to be detected, P2.

[0070] In some embodiments, the detection result of the millimeter-wave radar on the target to be detected includes the width and height of the target detection region in the original image, as well as the coordinate information of the target detection region in the original image. In this embodiment, the width and height of the target detection region in the original image can be determined according to the following formula:

[0071]

[0072] Where w is the width of the target detection region in the image to be detected, h is the height of the target detection region in the image to be detected, and λ is the actual width of the target to be detected. Let f be the actual height of the target to be detected, f be the focal length of the camera used to acquire the original image, and d be the distance of the target to be detected detected by the millimeter-wave radar. Alternatively, the coordinate information of the target detection area in the original image can be determined according to the following formula:

[0073]

[0074] in, x represents the coordinate information of the target detection region in the original image. r y r The location coordinates of the millimeter-wave radar used to acquire the location of the target to be detected are mapped onto the original image.

[0075] Specifically, The coordinates of the top-left vertex of the target region. The coordinates of the lower left vertex of the target region. The coordinates of the top-right corner vertex of the target region. The coordinates of the lower right corner vertex of the target area.

[0076] Furthermore, considering that in real road conditions, the traffic flow of vehicles (i.e., the targets to be detected) is relatively large, there may be overlap between the acquired target detection areas. In view of this, in this embodiment, the Non-Maximum Suppression (NMS) method can be used to remove redundant target detection areas.

[0077] In some embodiments of this application, the image to be detected can be obtained through the following steps:

[0078] (1) Determine the second size information of the target detection region in the image to be detected based on the first size information of the original image, the number of target detection regions in the original image, and the preset distance between the target detection regions.

[0079] Specifically, the second size information of the target detection region in the image to be detected can be determined according to the following formula:

[0080]

[0081] Where cut.size2 is the second size information of the target detection region, img.size1 is the first size information of the original image, n is the number of target detection regions in the original image, and v is the preset distance between target detection regions.

[0082] In some optional implementations, the original image P1 has the same size as the image to be detected P2, or the original image P1 may have a different size than the image to be detected P2. When the original image P1 has the same size as the image to be detected P2, the detection efficiency of the subsequent radar detection network can be improved.

[0083] It should be noted that there are multiple ways to stitch the target detection region and the original image together, for example, Figure 3 In the stitching method shown, the target detection region is arranged horizontally on the upper side of the target detection image, and the original image is located on the lower side of the target detection image; or, in some optional embodiments, the target detection region may also be arranged horizontally on the lower side of the target detection image, and the original image is located on the upper side of the target detection image; or, the target detection region may also be arranged vertically on the right (or left) side of the target detection image, and the original image is located on the left (or right) side of the target detection image.

[0084] by Figure 3 Taking the stitching method shown as an example, the first size information img.size1 is the width of the original image P1 (or the image to be detected P2), and the second size information cut.size2 is the width of the target detection area.

[0085] It should be noted that, since the radar detection network tends to identify different objects in two adjacent candidate regions as the same one, separating adjacent candidate regions by a certain distance, in this embodiment, the distance between each target detection region is set to v. This can prevent the target detection box in the final output image from crossing two candidate regions, thereby improving the accuracy of the detection results. It should be understood that the specific value of v is not limited in this embodiment.

[0086] As an alternative, the distances between the target detection regions can also be different. For example, the distance between target detection regions C1 and C2 is V1, and the distance between target detection regions C2 and C3 is V2...

[0087] (2) Based on the third size information of the original image and the second size information of the target detection area, determine the fourth size information of the original image in the image to be detected.

[0088] Specifically, the fourth dimension information of the original image in the image to be detected can be determined according to the following formula:

[0089] orisin.size4=img.size3-cut.size2

[0090] Here, orisin.size4 represents the fourth size information of the original image in the image to be detected, and img.size3 represents the third size information of the original image.

[0091] It should be noted that, in this embodiment of the application, the target detection area has the same width and height as an example. Here, the second dimension information of the target detection area is the height of the target detection area.

[0092] Taking the above example, the fourth size information orisin.size4 is the height of the original image P1 in the image to be detected P2, and the third size information img.size3 is the original height of the original image P1.

[0093] (3) Based on the second size information of the target detection area and the fourth size information of the original image in the image to be detected, determine the first coordinate information of the target detection area in the image to be detected and the second coordinate information of the original image in the image to be detected;

[0094] Specifically, on the one hand, the first coordinate information of the target detection region in the image to be detected can be obtained according to the following formula:

[0095]

[0096] Where i is the index of the target detection region. This represents the first coordinate information of the target detection region in the image to be detected.

[0097] Specifically, The coordinates of the top-left corner of the target detection region in the image to be detected, The coordinates of the target detection region in the lower left corner of the image to be detected, The coordinates of the upper right corner of the target detection region in the image to be detected. The coordinates of the lower right corner of the target detection region in the image to be detected.

[0098] On the other hand, the second coordinate information of the original image in the image to be detected can be obtained according to the following formula:

[0099]

[0100] in, This refers to the second coordinate information of the original image within the image to be detected. Specifically, The coordinates of the top-left corner of the original image in the image to be detected. The coordinates of the lower left corner of the original image in the image to be detected. The coordinates of the top right corner of the original image in the image to be detected. The coordinates are the bottom right corner of the original image within the image to be detected.

[0101] (4) Obtain the image to be detected based on the first coordinate information and the second coordinate information.

[0102] Specifically, after obtaining the above coordinate information, the images are stitched together sequentially from the top left corner of the image according to the sequence number of each target detection area, the first coordinate information of each target detection area, and the second coordinate information of the original image, thereby obtaining the image to be detected.

[0103] S202. Input the image to be detected into the RaidVision Detection Network, and use the RaidVision Detection Network to detect the target in the target detection area to obtain the feature image output by the RaidVision Detection Network.

[0104] The feature image includes the target detection results for identifying the target to be detected.

[0105] The structure and implementation principle of the radar vision detection network will be shown in subsequent embodiments.

[0106] S203. Based on the target detection region corresponding to the feature image and the original image, perform restoration processing on the feature image to obtain the target image.

[0107] In this case, the region where the target to be detected is located in the target image is the same as the region where the target to be detected is located in the original image.

[0108] It should be noted that since the above target detection region is generated based on the target image after coordinate transformation and mapping of the original image, the feature image needs to be restored to obtain the target image.

[0109] Figure 4 A schematic diagram of the principle of the moving target recognition method provided in the embodiments of this application. Figure 2 .like Figure 4 As shown, P3 is the feature image, and P4 is the target image after reconstruction based on the feature image.

[0110] In this embodiment of the application, during the image restoration process, the target detection area where the detection box is located is first determined, and then the detection box is restored according to the coordinate information in the original image corresponding to the target candidate area. The detection box includes the target detection result, which is the vehicle, ship or other means of transportation.

[0111] Specifically, the target image can be obtained using the following formula:

[0112]

[0113] in, To reconstruct the coordinates of the detection box from the image to be detected in the original image. The coordinates of the detection box within the target detection area are given. The detection box includes the target detection result.

[0114] In this formula, the aspect ratio of the detection box to the target detection region is first determined, and then multiplied by the aspect ratio of the target detection region in the original image, plus... This gives you the coordinates of the detection box on the original image.

[0115]

[0116] In this formula, These are the coordinates of the detection box in the restored target image.

[0117] Next, the structure and principle of the radar-based detection network will be explained in detail with reference to specific embodiments:

[0118] Figure 5 A schematic diagram of the structure of the radar-visual detection network provided in the embodiments of this application. Figure 1 .like Figure 5As shown, the radar-visual detection network provided in this application embodiment includes: a millimeter-wave radar feature extraction unit, a first visual extraction unit, a second visual extraction unit, and an adaptive feature fusion unit.

[0119] The first visual extraction unit is used to extract low-level visual feature maps, and the second visual extraction unit is used to extract high-level visual feature maps. The RaidVision Detection Network can obtain more semantic information by fusing the two.

[0120] Implementing step S202 specifically includes the following steps:

[0121] (1) The feature extraction unit of the millimeter-wave radar is used to extract features from the image to be detected, and the millimeter-wave radar feature map and the first weight coefficient corresponding to the millimeter-wave radar feature map are obtained.

[0122] (2) Through the first visual extraction unit, extract the low-level visual features in the image to be detected, and obtain the low-level visual feature map and the second weight coefficient corresponding to the low-level visual feature map.

[0123] (3) Extract high-level visual features from the image to be detected through the second visual extraction unit to obtain the high-level visual feature map and the third weight coefficient corresponding to the high-level visual feature map;

[0124] In this embodiment, a 1×1 convolutional layer is used to adjust the millimeter-wave radar feature extraction unit, the first visual extraction unit, and the second visual extraction unit to have the same number of channels. Next, the size is adjusted through upsampling or downsampling operations, resulting in feature maps that are respectively a millimeter-wave radar feature map, a low-level visual feature map, and a high-level visual feature map.

[0125] Furthermore, by performing a 1×1 convolution on the millimeter-wave radar feature map, the low-level visual feature map, and the high-level visual feature map, we can obtain their respective first weight parameter α, second weight coefficient β, and third weight coefficient γ.

[0126] (4) Through the adaptive feature fusion unit, the millimeter-wave radar feature map, the first weight coefficient, the low-level visual feature map, the second weight coefficient, the high-level visual feature map and the third weight coefficient are subjected to adaptive feature fusion processing to obtain the feature image.

[0127] Furthermore, the weighted fused feature image is obtained by multiplying the millimeter-wave radar feature map, low-level visual feature map, and high-level visual feature map by the first weight parameter α, the second weight coefficient β, and the third weight coefficient γ, respectively, through the adaptive feature fusion unit. Specifically, the feature image can be obtained using the following formula:

[0128] y = α·x 1 +β·x 2 +γ·x3

[0129] Where y is the feature image, x 1 For millimeter-wave radar feature maps, x 2 For low-level visual feature maps, x 3 For high-level visual feature maps, α+β+γ=1 (α, β, γ∈[0,1]).

[0130] In this embodiment, since millimeter-wave radar detection is less affected by illumination, while image detection has more texture information, weighted fusion of the feature advantages of both is beneficial to improving the detection accuracy of the system in low-light environments. However, the differences in the structural mechanism and viewing angle of millimeter-wave radar and camera will result in different sizes and spatial positions. At the same time, for target detection, the feature contribution of different layers is different. If the millimeter-wave radar feature map and the visual feature map are simply spatially transformed and superimposed after being adjusted to the same size, the characteristics of different feature maps cannot be fully utilized. Therefore, it is necessary to make full use of the features of radar and vision.

[0131] In addition, increasing the network width and depth can further improve detection accuracy. The weighted fused output is passed through different branch networks, and the feature maps are processed by convolutions of different sizes on the branches to extract information from different receptive fields in the image. Finally, the outputs of the three branches are merged to obtain stronger image representation capabilities and improve the accuracy of the detection results.

[0132] Based on the above ideas of visual enhancement algorithms and feature weighted fusion methods, a radar-visual detection network is designed, which includes two parts: millimeter-wave radar and camera fusion calibration and network structure design.

[0133] In the radar-visual detection network, both visual enhancement and feature weighting calculation require obtaining the spatial coordinate relationship between the data acquired by the millimeter-wave radar and camera. Therefore, the first step is to fuse and calibrate the millimeter-wave radar and camera data to map the target points in the millimeter-wave radar coordinate system to the image coordinate system. Next, the millimeter-wave radar data is processed to generate millimeter-wave images. Specifically, the distance D, velocity V, and RCS detected by the millimeter-wave radar are converted into pixel values ​​in different channels (R, G, B), as shown in the following conversion relationship:

[0134]

[0135] Among them, V max V represents the maximum speed, i.e., the current speed limit on the road; min Indicates the minimum speed; RCS max This represents the average of the maximum RCS values ​​measured multiple times from the millimeter-wave radar output; RCS min D represents the minimum RCS value; maxThis indicates the maximum detectable distance by millimeter-wave radar. (D) min This indicates the minimum distance detected.

[0136] Next, this application designs a radar detection network based on the YOLOv4-tiny basic framework. Figure 6 A schematic diagram of the structure of the radar-visual detection network provided in the embodiments of this application. Figure 2 .like Figure 6 As shown, the radar-visual detection network mainly consists of three parts: a millimeter-wave radar feature extraction branch (Radar_Net), a visual image feature extraction branch (CSPDarknet53-tiny), and a radar-visual weighted fusion framework (A-Net).

[0137] In this embodiment, the millimeter-wave feature extraction branch Radar-Net consists of basic convolutional units CBLM and CBL, while the visual extraction branch CSPDarknet53-tiny mainly consists of CBL units and Resblock units. CBLM consists of convolutional layers, batch normalization, activation functions, and max pooling layers; CBL consists of convolutional layers, batch normalization, and activation functions; and Resblock combines four CBLs through residual nesting, followed by max pooling.

[0138] To improve detection accuracy by fusing millimeter-wave radar with visual feature maps, the 26×26 feature map Layer1 from Radar_Net is first upsampled and then fused with the 52×52 feature map Layer2 from CSPDarknet53-Tiny using a Concat operation. This fusion is then input into the subsequent visual extraction network. Next, the 26×26 feature map Layer3, the 13×13 feature map Layer4, and Layer1 from CSPDarknet53-Tiny are input into the A-Net framework, and the output is sent to the YOLO Head to complete the detection.

[0139] Experimental Results and Analysis: To test the application effect of the proposed algorithm, it was compared with the pure vision algorithm YOLOv4, YOLOv4-tiny, and the typical radar-visual fusion algorithm RVNet. Tests were conducted on the Nuscenes public dataset and a dataset of urban roads under low-light conditions collected by the project team at night.

[0140] Tables 1 and 2 show the detection performance of the test model on the Nuscenes test set in two scenarios: low brightness at night and normal brightness during the day.

[0141] Table 1. Performance comparison of different networks on Nuscenes nighttime data.

[0142] Network AP (%) FPS YOLOv4-tiny 47.01 33 YOLOv4 57.86 10 RVNet 60.70 14 This application's radar detection network 67.26 12

[0143] Table 2 Comparison of performance of different networks on Nuscenes daytime data.

[0144] Network AP (%) FPS YOLOv4-tiny 74 33 YOLOv4 78.1 10 RVNet 82.28 14 This application's radar detection network 83.06 12

[0145] As can be seen from Table 2, under normal brightness scene testing, the AP value of the radar-visual fusion network in this application embodiment is only 5% higher than that of the pure vision network. However, under low brightness scene, Table 1 shows that the AP value of the method in this paper is 10% higher than that of YOLOv4 and 7% higher than that of RVNet. The results show that the radar-visual detection network provided in this application embodiment has improved the target detection performance in both low brightness at night and daytime.

[0146] To further illustrate the detection performance of the method proposed in this application in real-world scenarios, an evaluation and verification were conducted on a self-made dataset. The experimental results are shown in Table 3 below:

[0147] Table 3. Performance comparison of different networks on the self-made dataset.

[0148] Network AP (%) FPS YOLOv4-tiny 41.29 33 YOLOv4 50.52 10 RVNet 65.29 14 This application's radar detection network 70.60 12

[0149] As can be seen from Table 3, the performance of the radar-visual converged network AP value provided in this application embodiment is improved compared with other networks.

[0150] Table 4 shows the performance of each algorithm in detecting targets at different distances. Through field measurements, positions 40 meters, 80 meters, and 120 meters from the detection device were marked. Experimental vehicles were driven near these designated locations, and test sets of vehicle targets at different distances were collected. The YOLOv4 and RVNet algorithms, which have high detection accuracy, were compared with the radar-based detection network provided in this application embodiment. Since the YOLOv4-tiny algorithm does not have an advantage in detecting long-distance targets, it was not compared.

[0151] Table 4. Average accuracy of vehicle detection at different distances

[0152]

[0153] As shown in Table 4, all algorithms can achieve good detection results at a distance of 40 meters from the target. As the distance increases, the detection results of YOLOv4 and RVNet algorithms gradually decrease. When the distance increases to more than 120 meters, only the Raivision detection network provided in this application embodiment can detect the target. It uses millimeter-wave distance information for image reconstruction, which has an advantage over the other two algorithms in detecting distant vehicle targets.

[0154] Table 5 shows the performance test results of the Rayvision weighted fusion framework using different combinations of convolutional layers:

[0155] Table 5. Detection performance test under different convolutional layer configurations.

[0156]

[0157] As shown in Table 5, the 1x1, 5x5, and 7x7 convolutional layer combinations are more effective than the four-convolutional layer combination. Therefore, in this embodiment, the Ravis Plus detection network is composed of 1x1, 5x5, and 7x7 convolutional layers.

[0158] Table 6 shows the detection results of weighted fusion of millimeter-wave radar feature maps and visual feature maps of different sizes. Feature maps of sizes 104, 52, and 26 were selected from the YOLOv4-Tiny backbone feature extraction network and fused with millimeter-wave radar feature maps, respectively.

[0159] Table 6 shows the detection results of weighted fusion of visual feature maps of different sizes.

[0160]

[0161] As shown in Table 6, fusion is better at the location where the visual image feature map size is 52.

[0162] To test the impact of Ravage backbone network fusion, Ravage weighted fusion framework, and image reconstruction on detection performance, ablation experiments were conducted. The model was trained and tested on the same training and test sets. The test results are shown in Table 7 below:

[0163] Leishi Backbone Network Convergence Rayvision Weighted Fusion Framework Image reconstruction AP (%) √ 66.60 √ 66.54 √ √ 67.20 √ √ √ 70.60

[0164] As shown in Table 7, it can be seen that all three improved methods have optimized the detection effect, and the detection effect is best when all three methods are used at the same time.

[0165] After introducing the methods of exemplary embodiments of this application, the following references are made. Figure 7 The storage medium of an exemplary embodiment of this application will be described.

[0166] refer to Figure 7 As shown, the storage medium 700 stores a program product for implementing the above-described method according to an embodiment of this application. This program product may be a portable compact disc read-only memory (CD-ROM) and includes program code, and can run on a terminal device, such as a personal computer. However, the program product of this application is not limited thereto.

[0167] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0168] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium.

[0169] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN).

[0170] refer to Figure 8 , Figure 8 Schematic diagram of the target detection device provided in the embodiments of this application Figure 1 .like Figure 8 As shown, the target detection device 800 includes:

[0171] The reconstruction unit 801 is used to reconstruct the original image based on the detection result of the millimeter-wave radar of the target to be detected, and obtain the image to be detected. The image to be detected contains the original image and at least one target detection region, and the target detection region contains at least one target to be detected.

[0172] The detection unit 802 is used to input the image to be detected into the radar vision detection network, detect the target to be detected in the target detection area through the radar vision detection network, and obtain the feature image output by the radar vision detection network. The feature image includes the target detection result of the target to be detected.

[0173] The restoration unit 803 is used to restore the feature image based on the target detection region corresponding to the feature image and the original image to obtain the target image. The region where the target to be detected is located in the target image is the same as the region where the target to be detected is located in the original image.

[0174] In one possible implementation, the reconstruction unit 801 is specifically configured to: determine the second size information of the target detection region in the image to be detected based on the first size information of the original image, the number of target detection regions in the original image, and the preset distance between the target detection regions; determine the fourth size information of the original image in the image to be detected based on the third size information of the original image and the second size information of the target detection region; determine the first coordinate information of the target detection region in the image to be detected and the second coordinate information of the original image in the image to be detected based on the second size information of the target detection region and the fourth size information of the original image in the image to be detected; and obtain the image to be detected based on the first coordinate information and the second coordinate information.

[0175] In one possible implementation, the reconstruction unit 801 is specifically used to: determine the second size information of the target detection region in the image to be detected according to the following formula:

[0176]

[0177] Where cut.size2 is the second size information of the target detection region, img.size1 is the first size information of the original image, n is the number of target detection regions in the original image, and v is the preset distance between target detection regions.

[0178] In one possible implementation, the reconstruction unit 801 is specifically used to: determine the fourth size information of the original image in the image to be detected according to the following formula:

[0179] orisin.size4=img.size3-cut.size2

[0180] Here, orisin.size4 represents the fourth size information of the original image in the image to be detected, and img.size3 represents the third size information of the original image.

[0181] In one possible implementation, the reconstruction unit 801 is specifically used to: obtain the first coordinate information of the target detection region in the image to be detected according to the following formula:

[0182]

[0183] The second coordinate information of the original image in the image to be detected is obtained using the following formula:

[0184]

[0185] Where i is the index of the target detection region. This refers to the first coordinate information of the target detection region in the image to be detected; This represents the second coordinate information of the original image within the image to be detected.

[0186] In one possible implementation, the radar-visual detection network includes a millimeter-wave radar feature extraction unit, a first visual extraction unit, a second visual extraction unit, and an adaptive feature fusion unit.

[0187] The detection unit 802 is specifically used to: extract features from the image to be detected through the millimeter-wave radar feature extraction unit to obtain a millimeter-wave radar feature map and a first weight coefficient corresponding to the millimeter-wave radar feature map;

[0188] The first visual extraction unit extracts low-level visual features from the image to be detected, obtaining a low-level visual feature map and a second weighting coefficient corresponding to the low-level visual feature map. The second visual extraction unit extracts high-level visual features from the image to be detected, obtaining a high-level visual feature map and a third weighting coefficient corresponding to the high-level visual feature map. The adaptive feature fusion unit performs adaptive feature fusion processing on the millimeter-wave radar feature map, the first weighting coefficient, the low-level visual feature map, the second weighting coefficient, the high-level visual feature map, and the third weighting coefficient to obtain a feature image.

[0189] In one possible implementation, the detection result includes the width and height of the target detection region in the original image, as well as the coordinate information of the target detection region in the original image; the target detection device 800 further includes: a determination unit 804, configured to determine the width and height of the target detection region in the original image according to the following formula:

[0190] The coordinates of the target detection region in the original image are determined using the following formula:

[0191]

[0192] Where w is the width of the target detection region in the image to be detected, h is the height of the target detection region in the image to be detected, and λ is the actual width of the target to be detected. Where is the actual height of the target to be detected, f is the focal length of the camera used to acquire the original image, and d is the distance of the target to be detected detected by the millimeter-wave radar. This refers to the coordinate information of the target detection region in the original image.

[0193] In one possible implementation, the restoration unit 803 is specifically used to: obtain the target image according to the following formula:

[0194]

[0195] in, To reconstruct the coordinates of the detection box from the image to be detected in the original image. The coordinates of the detection box within the target detection area are given. The detection box includes the target detection result.

[0196] It should be understood that the target detection device 1000 provided in this application embodiment is used to implement the moving target recognition method in any of the above-described method embodiments on the receiving device side. Its implementation principle and technical effect are similar, and will not be described again here.

[0197] After introducing the methods, media, and apparatus of exemplary embodiments of this application, the following references are made. Figure 9 The computing device according to exemplary embodiments of this application will be described. It should be understood that... Figure 9 The computing device 900 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0198] Figure 9 A schematic diagram of the structure of a computing device provided in an embodiment of this application. Figure 9 As shown, the computing device 900 is presented in the form of a general-purpose computing device. The components of the computing device 900 may include, but are not limited to: at least one processing unit 901, at least one storage unit 902, and a bus 903 connecting different system components (including the processing unit 901 and the storage unit 902).

[0199] Bus 903 includes a data bus, a control bus, and an address bus. Storage unit 902 may include readable media in the form of volatile memory, such as random access memory (RAM) 913 and / or cache memory 922, and may further include readable media in the form of non-volatile memory, such as read-only memory (ROM) 932.

[0200] Storage unit 902 may also include a program / utility 952 having a set (at least one) of program units 942, such program units 942 including but not limited to: an operating system, one or more application programs, other program units, and program data, each or some combination of these examples may include an implementation of a network environment.

[0201] The computing device 900 can also communicate with one or more external devices 904 (e.g., keyboard, pointing device, etc.). This communication can be performed via input / output (I / O) interface 905. Furthermore, the computing device 900 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 906. Figure 9 As shown, network adapter 906 communicates with other units of computing device 900 via bus 903. It should be understood that, although not shown in the figure, other hardware and / or software units may be used in conjunction with computing device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0202] It should be noted that although several units / units or sub-units / units of the timing update device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more units / units described above can be embodied in one unit / unit. Conversely, the features and functions of one unit / unit described above can be further divided and embodied by multiple units / units.

[0203] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0204] While the spirit and principles of this application have been described with reference to several specific embodiments, it should be understood that this application is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for convenience of expression. This application is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A target detection device based on radar-visual fusion, characterized in that, include: The reconstruction unit is used to reconstruct the original image based on the detection result of the millimeter-wave radar of the target to be detected, and obtain the image to be detected, wherein the image to be detected contains the original image and at least one target detection region, and the target detection region contains at least one target to be detected. The detection unit is used to input the image to be detected into the radar vision detection network, detect the target to be detected in the target detection area through the radar vision detection network, and obtain the feature image output by the radar vision detection network. The feature image includes the target detection result of the target to be detected. The restoration unit is used to restore the feature image based on the target detection region corresponding to the feature image and the original image to obtain a target image, wherein the region where the target to be detected is located in the target image is the same as the region where the target to be detected is located in the original image. The detection results include the width and height of the target detection region in the original image, as well as the coordinate information of the target detection region in the original image. The determining unit is configured to determine the width and height of the target detection region in the original image according to the following formula: , ; in, The width of the target detection region in the image to be detected. λ is the height of the target detection region in the image to be detected, and λ is the actual width of the target to be detected. d is the actual height of the target to be detected, f is the focal length of the camera used to acquire the original image, and d is the distance of the target to be detected detected by the millimeter-wave radar. The reconstruction unit is specifically used to: determine the second size information of the target detection region in the image to be detected based on the first size information of the original image, the number of target detection regions in the original image, and the preset distance between the target detection regions; Based on the third size information of the original image and the second size information of the target detection region, the fourth size information of the original image in the image to be detected is determined; Based on the second size information of the target detection region and the fourth size information of the original image in the image to be detected, the first coordinate information of the target detection region in the image to be detected and the second coordinate information of the original image in the image to be detected are determined. The image to be detected is obtained based on the first coordinate information and the second coordinate information; The restoration unit is specifically used to: obtain the target image according to the following formula: in, , , , The coordinates of the detection box to be reconstructed from the image to be detected in the original image. , , , Let be the coordinates of the detection box within the target detection region, where the detection box includes the target detection result. , , , Here, v represents the coordinates of the target detection region in the original image, and v is the preset distance between the target detection regions. 2 represents the second size information of the target detection area.

2. The target detection device based on radar-visual fusion according to claim 1, characterized in that, The reconstruction unit is specifically used to: determine the second size information of the target detection region in the image to be detected according to the following formula: - v in, 2 represents the second size information of the target detection area. 1 represents the first size information of the original image, n represents the number of target detection regions in the original image, and v represents the preset distance between target detection regions.

3. The target detection device based on radar-visual fusion according to claim 2, characterized in that, The reconstruction unit is specifically used to: determine the fourth size information of the original image in the image to be detected according to the following formula: 3 - 2 in, The fourth size information of the original image in the image to be detected. 3 represents the third dimension information of the original image.

4. The target detection device based on radar-visual fusion according to claim 3, characterized in that, The reconstruction unit is specifically used to: obtain the first coordinate information of the target detection region in the image to be detected according to the following formula: The second coordinate information of the original image in the image to be detected is obtained according to the following formula: Where i is the index of the target detection region. , , , The first coordinate information of the target detection region in the image to be detected; , , , The second coordinate information of the original image in the image to be detected.

5. The target detection device based on radar-visual fusion according to claim 1, characterized in that, The radar vision detection network includes a millimeter-wave radar feature extraction unit, a first vision extraction unit, a second vision extraction unit, and an adaptive feature fusion unit. The detection unit is specifically used to: extract features from the image to be detected through the millimeter-wave radar feature extraction unit to obtain a millimeter-wave radar feature map and a first weight coefficient corresponding to the millimeter-wave radar feature map; The first visual extraction unit extracts low-level visual features from the image to be detected, and obtains a low-level visual feature map and a second weighting coefficient corresponding to the low-level visual feature map. The second visual extraction unit extracts high-level visual features from the image to be detected, and obtains a high-level visual feature map and a third weighting coefficient corresponding to the high-level visual feature map. The adaptive feature fusion unit performs adaptive feature fusion processing on the millimeter-wave radar feature map, the first weight coefficient, the low-level visual feature map, the second weight coefficient, the high-level visual feature map, and the third weight coefficient to obtain the feature image.

Images