Target detection method and device based on radar-optical cross-modal feature point fusion

Abstract

The present application relates to a target detection method and device based on radar-optical cross-modal feature point fusion. A backbone network is used to extract features from the acquired optical image and radar RD spectrum data, respectively, to obtain semantic dense point features and RD dense point features. The radar point cloud is projected onto the optical image. Based on the position of the two-dimensional detection box on the semantic dense point features, a radar point cloud corresponding to the region of interest is obtained on the optical image. This portion of the radar point cloud is then projected onto the semantic dense point features and RD dense point features to obtain the corresponding features on the region of interest. These two features are then fused to obtain fused point features. Local coordinates are then constructed based on the target. The region of interest is divided into multiple regions with the local coordinates as the origin. The target detection problem is converted into a classification problem. A classification neural network is used to perform regional classification on the fused point features to achieve target detection. This method can improve the accuracy of target detection and effectively detect targets in harsh environments.

Description

Technical Field

[0001] The present application relates to the field of target detection technology, and in particular to a target detection method and device based on radar-optical cross-modal feature point fusion. Background Art

[0002] Autonomous driving and advanced driver assistance systems (ADAS) typically rely on different types of sensors to obtain reliable perception. Due to their unique working mechanisms and characteristics, the mainstream sensors onboard cars include cameras, lidar, and radar. The current mainstream multi-sensor fusion strategy uses cameras and lidar sensors for 3D object detection. o Because the lidar's dense point cloud has high angular resolution and distance detection accuracy, it complements the rich scene context and semantic information in the optical image acquired by the camera, allowing for better target detection.

[0003] However, in adverse weather conditions, the performance of both cameras and lidars drops significantly, resulting in unstable target detection, a key requirement for long-term stable autonomous driving. Summary of the Invention

[0004] Based on this, it is necessary to provide a target detection method and device based on radar-optical cross-modal feature point fusion that can perform stable target detection in bad weather to address the above technical problems.

[0005] A target detection method based on radar-optical cross-modal feature point fusion, the method comprising:

[0006] Acquire radar echo data and an optical image, wherein the radar echo data and the optical image are respectively obtained by detecting the target area by a millimeter wave radar system and a camera, wherein the radar echo data includes radar range and velocity spectrum data and radar point cloud data;

[0007] Input the optical image and radar range and velocity spectrum data into the corresponding backbone network to obtain semantic dense point features, two-dimensional detection boxes, and range and azimuth dense point features respectively;

[0008] Processing the two-dimensional detection frame, the radar point cloud data, and the optical image to obtain partial radar point cloud data corresponding to the target area of ​​interest, and obtaining corresponding partial semantic dense point features and partial range and azimuth dense point features from the semantic dense point features and the range and azimuth dense point features based on the partial radar point cloud data;

[0009] Inputting some of the semantic dense point features and some of the distance and azimuth dense point features into a feature fusion network to obtain cross-modal fusion point features;

[0010] Constructing a local coordinate system centered on a target, and dividing the target region of interest into a plurality of regions with the local coordinate system as an origin;

[0011] The classification result is obtained by performing regional classification on the cross-modal fusion point features relative to the local coordinates using a classification neural network, as well as the quantization error generated during the discretization process. The target position is obtained based on the classification result and the quantization error to achieve target detection.

[0012] In one embodiment, the backbone network corresponding to the optical image adopts a ResNet-18 network, and the optical image is processed by the ResNet-18 network to obtain the semantic dense point features and the two-dimensional detection box.

[0013] In one embodiment, the backbone network corresponding to the radar range and velocity spectrum data adopts an FFT-RadNet network, and the radar range and velocity spectrum data is processed by the FFT-RadNet network to obtain the range and azimuth dense point features.

[0014] In one embodiment, the processing of the two-dimensional detection frame, the radar point cloud data, and the optical image to obtain partial radar point cloud data corresponding to the target area of ​​interest includes:

[0015] The radar point cloud data is projected onto the optical image, and the radar point cloud projected onto the optical image is filtered according to the position of the two-dimensional detection box on the semantic dense point feature to obtain partial radar point cloud data corresponding to the target area of ​​interest.

[0016] In one embodiment, obtaining corresponding partial semantic dense point features and partial range and azimuth dense point features from the semantic dense point features and the range and azimuth dense point features according to the partial radar point cloud data includes:

[0017] The part of the radar point cloud data is projected onto the semantic dense point features and the range and azimuth dense point features respectively to obtain the part of the semantic dense point features and the part of the range and azimuth dense point features respectively.

[0018] In one embodiment, the feature fusion network includes a two-sided multilayer perceptron and a maximum pooling layer, wherein the two-sided multilayer perceptron includes a first layer of perceptrons and a second layer of perceptrons.

[0019] In one embodiment, inputting the partial semantic dense point features and the partial distance and azimuth dense point features into a feature fusion network to obtain cross-modal fusion point features includes:

[0020] Inputting the partial semantic dense point features and the partial distance and azimuth dense point features into the multilayer perceptrons on both sides of the corresponding side respectively;

[0021] Inputting the partial semantic dense point features and the partial distance and azimuth dense point features into the corresponding first layer perceptrons for feature extraction, respectively, to obtain first semantic features and first distance and azimuth features, and fusing these two features to obtain low-level features of the target;

[0022] Inputting the first semantic feature and the first distance and azimuth feature into the corresponding second-layer perceptron for feature extraction, respectively, to obtain a second semantic feature and a second distance and azimuth feature, and fusing the two obtained features to obtain a high-level feature of the target;

[0023] The low-level features and high-level features of the target are passed through the maximum pooling layer to obtain the global features of the target;

[0024] The low-level features, high-level features and global features of the target are fused to obtain the cross-modal fusion point features.

[0025] In one embodiment, the processing according to the classification result and the quantization error to obtain the target position includes: decoding the classification result and the quantization error to obtain the target position.

[0026] A target detection device based on radar-optical cross-modal feature point fusion, the device comprising:

[0027] A detection data acquisition module is used to obtain radar echo data and optical images, wherein the radar echo data and optical images are respectively obtained by detecting the target area by a millimeter wave radar system and a camera, wherein the radar echo data includes radar range and velocity spectrum data and radar point cloud data;

[0028] A dense point feature extraction module is used to input the optical image and radar range and velocity spectrum data into the corresponding backbone network to obtain semantic dense point features, two-dimensional detection boxes, and range and azimuth dense point features respectively;

[0029] a dense feature point extraction module corresponding to the region of interest, configured to obtain partial radar point cloud data corresponding to the target region of interest based on the two-dimensional detection frame, the radar point cloud data, and the optical image, and obtain corresponding partial semantic dense point features and partial range and azimuth dense point features from the semantic dense point features and the range and azimuth dense point features based on the partial radar point cloud data;

[0030] A cross-modal fusion point feature acquisition module is used to input the partial semantic dense point features and the partial distance and azimuth dense point features into a feature fusion network to obtain cross-modal fusion point features;

[0031] A local coordinate construction module is used to construct local coordinates centered on the target, and divide the target region of interest into multiple regions with the local coordinates as the origin;

[0032] The target detection module is used to obtain a classification result by regionally classifying the cross-modal fusion point features relative to the local coordinates using a classification neural network, as well as a quantization error generated during the discretization process, and to obtain a target position based on the classification result and the quantization error to achieve target detection.

[0033] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0034] Acquire radar echo data and an optical image, wherein the radar echo data and the optical image are respectively obtained by detecting the target area by a millimeter wave radar system and a camera, wherein the radar echo data includes radar range and velocity spectrum data and radar point cloud data;

[0035] Input the optical image and radar range and velocity spectrum data into the corresponding backbone network to obtain semantic dense point features, two-dimensional detection boxes, and range and azimuth dense point features respectively;

[0036] Processing the two-dimensional detection frame, the radar point cloud data, and the optical image to obtain partial radar point cloud data corresponding to the target area of ​​interest, and obtaining corresponding partial semantic dense point features and partial range and azimuth dense point features from the semantic dense point features and the range and azimuth dense point features based on the partial radar point cloud data;

[0037] Inputting some of the semantic dense point features and some of the distance and azimuth dense point features into a feature fusion network to obtain cross-modal fusion point features;

[0038] Constructing a local coordinate system centered on a target, and dividing the target region of interest into a plurality of regions with the local coordinate system as an origin;

[0039] The classification result is obtained by performing regional classification on the cross-modal fusion point features relative to the local coordinates using a classification neural network, as well as the quantization error generated during the discretization process. The target position is obtained based on the classification result and the quantization error to achieve target detection.

[0040] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0041] Acquire radar echo data and an optical image, wherein the radar echo data and the optical image are respectively obtained by detecting the target area by a millimeter wave radar system and a camera, wherein the radar echo data includes radar range and velocity spectrum data and radar point cloud data;

[0042] Input the optical image and radar range and velocity spectrum data into the corresponding backbone network to obtain semantic dense point features, two-dimensional detection boxes, and range and azimuth dense point features respectively;

[0043] Processing the two-dimensional detection frame, the radar point cloud data, and the optical image to obtain partial radar point cloud data corresponding to the target area of ​​interest, and obtaining corresponding partial semantic dense point features and partial range and azimuth dense point features from the semantic dense point features and the range and azimuth dense point features based on the partial radar point cloud data;

[0044] Inputting some of the semantic dense point features and some of the distance and azimuth dense point features into a feature fusion network to obtain cross-modal fusion point features;

[0045] Constructing a local coordinate system centered on a target, and dividing the target region of interest into a plurality of regions with the local coordinate system as an origin;

[0046] The classification result is obtained by performing regional classification on the cross-modal fusion point features relative to the local coordinates using a classification neural network, as well as the quantization error generated during the discretization process. The target position is obtained based on the classification result and the quantization error to achieve target detection.

[0047] The above-mentioned target detection method and device based on radar-optical cross-modal feature point fusion adopts a backbone network to perform feature extraction on the acquired optical image and radar RA spectrum data respectively to obtain semantic dense point features and RD dense point features, and projects the radar point cloud to the optical image. According to the position of the two-dimensional detection box on the semantic dense point feature, the radar point cloud corresponding to the area of ​​interest is obtained on the optical image, and then this part of the radar point cloud is projected onto the semantic dense point feature and the RA dense point feature respectively to obtain the corresponding features on the area of ​​interest, and the two features are fused using a fusion network to obtain fusion point features, and then local coordinates are constructed based on the target, and the area of ​​interest is divided into multiple regions with the local coordinates as the origin, and the problem of target detection is converted into a classification problem. The fusion point features are regionally classified using a classification neural network to realize target detection. The use of this method can improve the accuracy of target detection and enable it to effectively detect targets in harsh environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 1 is a flow chart of a target detection method based on radar-optical cross-modal feature point fusion in one embodiment;

[0049] Figure 2 A schematic diagram of obtaining partial semantic dense point features and partial range and azimuth dense point features from radar point cloud data in one embodiment;

[0050] Figure 3 Schematic diagram of the structure of a feature fusion network in one embodiment;

[0051] Figure 4 A schematic diagram of local coordinate construction in one embodiment;

[0052] Figure 5 A schematic diagram of the structure of a target detection network in one embodiment

[0053] Figure 6 1 is a structural block diagram of a target detection device based on radar-optical cross-modal feature point fusion in one embodiment;

[0054] Figure 7 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0056] In existing technologies, driver assistance systems rely on different types of sensors to obtain reliable perception of the vehicle's surrounding environment. In the current mainstream multi-sensor fusion strategy, cameras, lidar, or radar are often used to detect 3D objects. However, in bad weather, the performance of cameras and radars will be greatly reduced, resulting in unstable target detection accuracy. Figure 1 As shown, a target detection method based on radar-optical cross-modal feature point fusion is provided, comprising the following steps:

[0057] Step S100: Acquire radar echo data and an optical image. The radar echo data and the optical image are obtained by detecting the target area using a millimeter-wave radar system and a camera, respectively. The radar echo data includes radar range and velocity spectrum data and radar point cloud data.

[0058] Step S110: Input the optical image and radar range and velocity spectrum data into the corresponding backbone network to obtain semantic dense point features, two-dimensional detection boxes, and range and azimuth dense point features respectively;

[0059] Step S120: Processing the two-dimensional detection frame, the radar point cloud data, and the optical image to obtain partial radar point cloud data corresponding to the target area of ​​interest, and obtaining corresponding partial semantic dense point features and partial range and azimuth dense point features from the semantic dense point features and the range and azimuth dense point features based on the partial radar point cloud data;

[0060] Step S130, inputting part of the semantic dense point features and part of the distance and azimuth dense point features into a feature fusion network to obtain cross-modal fusion point features;

[0061] Step S140, constructing a local coordinate system centered on the target, and dividing the target region of interest into multiple regions with the local coordinate system as the origin;

[0062] In step S150, a classification result is obtained by regionally classifying the cross-modal fusion point features relative to the local coordinates using a classification neural network, and a quantization error generated during the discretization process is processed to obtain the target position based on the classification result and the quantization error to achieve target detection.

[0063] In this method, target detection is achieved by fusing millimeter-wave radar data and dense contextual features of optical images. First, radar features and optical features are extracted through a feature extraction network respectively. Then, optical-radar correlation and a hybrid point-based fusion strategy are used to merge them at multiple levels to obtain cross-modal fusion features. Finally, in this method, a target-centered local coordinate is constructed to classify the target detection task into classification and regression tasks, and then the target is detected.

[0064] In this embodiment, a millimeter wave radar is used to detect the target area. The millimeter wave radar has the characteristics of being able to detect targets at all times and in all weather conditions without being affected by weather.

[0065] In step S100, the method described herein is applied to a driver assistance system to detect a detection area in front of the vehicle. The radar echo data and optical images acquired can be obtained by real-time detection of the detection area by a lidar device and camera installed on the vehicle. The detection area can contain one or more targets. Hereinafter, the detection area will be referred to as the target area.

[0066] In this embodiment, radar range and velocity spectrum data, namely, RD spectrum image and radar point cloud data can be obtained by processing radar echo data.

[0067] In step S110 , the RD spectrum image and the optical image are respectively subjected to the corresponding backbone network to extract dense point features.

[0068] In this embodiment, the backbone network for processing optical images uses a ResNet-18 network. This network processes the optical images to obtain semantically dense point features and a two-dimensional detection bounding box. The two-dimensional detection bounding box provides the approximate area where the target is located. The ResNet-18 network is a pre-trained network.

[0069] In this embodiment, the backbone network for radar RD spectrum data uses an FFT-RadNet network. This network processes the radar RD spectrum data to obtain range-azimuth (RA) dense point features. Similarly, the FFT-RadNet network used here is a pre-trained network.

[0070] Specifically, the backbone network for feature extraction can also use other neural networks of the same type with higher computing performance. For example, the visual neural network used to process optical images can also use ResNet-152 and Transformers.

[0071] Since semantic dense point features and RA dense point features have different imaging mechanisms, it is difficult to align dense points. Therefore, this method relies on radar point clouds to associate the two at the point cloud level. Each radar point cloud represents a 3D point ,in, and Represent the position coordinates of the point cloud on the RD spectrum image and the optical image respectively. Using the internal and external parameters of the camera, the radar point cloud coordinate system can be converted into the optical image coordinate system using the following formula:

[0072] (1)

[0073] (2)

[0074] In formula (1) and formula (2), are the internal and external parameters of the camera, is the position coordinate of the radar point cloud in the world coordinate system.

[0075] The two-dimensional detection box obtained by the ResNet-18 network can be used as a filter for the target region of interest (ROI) to separate the target region of interest from the background and noise, such as Figure 2 As shown in Figure 2, the two-dimensional detection box provides the prior angle information of the target and eliminates the sidelobe interference caused by the radar. Here, all points within these two-dimensional detection boxes are considered as candidate points for the next fusion stage.

[0076] like Figure 2 As shown in (a), since these two-dimensional detection boxes in the optical image cannot be used for orientation estimation, a frustum region is generated using the azimuth angle of interest provided by the two-dimensional detection box to select candidate point clouds related to the target, as shown in Figure 2 (b) As shown. The filtering of noise and background points in the candidate point cloud depends on whether their distance to the target center exceeds a certain threshold, such as Figure 2 (c) shown.

[0077] Specifically, in step S120, processing is performed based on the two-dimensional detection frame, the radar point cloud data, and the optical image to obtain partial radar point cloud data corresponding to the target area of ​​interest, including: first, projecting the radar point cloud data onto the optical image, and then filtering the radar point cloud projected on the optical image according to the position of the two-dimensional detection frame on the semantic dense point feature to obtain partial radar point cloud data corresponding to the target area of ​​interest, that is, a candidate radar point cloud related to the target.

[0078] Furthermore, obtaining corresponding partial semantic dense point features and partial RA dense point features from the semantic dense point features and RA dense features based on partial radar point cloud data includes: projecting partial radar point cloud data onto the semantic dense point features and RA dense point features, respectively, to obtain partial semantic dense point features and partial RA dense point features at corresponding positions of the partial radar point cloud data.

[0079] Assume that some radar point cloud data have 3D radar points, according to this 3D radar points Some semantic dense point features, namely semantic pixel features, can be extracted from the corresponding positions of semantic dense point features and RD dense point features. , and some RD dense point features, namely RD pixel features .in, and for point Features in RD spectrum images and optical images.

[0080] Then, in step S130, a feature fusion network is used to fuse some semantic dense point features related to the target and some RA dense point features. The structure of the feature fusion network is as follows: Figure 3 As shown in Figure 2, in the feature fusion network, a series of MLPs (multi-layer perceptrons) are used to process and fuse at multiple scales, and the global features of each target are fused through the maximum pooling layer, ultimately obtaining a multimodal fusion feature with spatial and semantic information by fusing features of various scales.

[0081] In this embodiment, the feature fusion network includes two-sided multilayer perceptrons and a maximum pooling layer, wherein the two-sided multilayer perceptrons include a first layer perceptron and a second layer perceptron.

[0082] Specifically, part of the semantic dense point features and part of the RD dense point features are input into the feature fusion network to obtain cross-modal fusion point features, including: respectively inputting part of the semantic dense point features and part of the distance azimuth dense point features into the multi-layer perceptrons on the corresponding sides, respectively inputting part of the semantic dense point features and part of the distance azimuth dense point features into the corresponding first layer of perceptrons for feature extraction, and correspondingly obtaining the first semantic feature and the first distance azimuth feature, and fusing these two features to obtain the low-level features of the target, respectively inputting the first semantic feature and the first distance azimuth feature into the corresponding second layer of perceptrons for feature extraction, and correspondingly obtaining the second semantic feature and the second distance azimuth feature, and fusing these two features to obtain the high-level features of the target, and obtaining the global features of the target through the maximum pooling layer of the low-level features and the high-level features of the target, and finally fusing the low-level features, high-level features and global features of the target to obtain cross-modal fusion point features.

[0083] In this embodiment, considering the differences in geometric distribution and semantics between the two feature spaces being fused, point-by-point features are combined in a hierarchical manner. Because the fusion of both low-level and high-level features can effectively discriminate point cloud-level features, fusion is performed at different scales through two sets of shared MLP processing. Simultaneously, global context features are obtained for each target object, revealing the properties of the same target shared across domains. Global point features are obtained by performing a maximum pooling operation on all candidate points corresponding to the same target.

[0084] Directly calculating the target position from a sparse point cloud is very difficult and involves the sensing environment, imaging resolution, and the absolute scale of the target position. To address this issue, in this method, a local coordinate system is re-established with the target as the center, transforming the target detection task into a combination of classification and regression subtasks.

[0085] Specifically, in step S140, if Figure 4 As shown, a local coordinate is established with the target center as the origin. In this local coordinate, The axes correspond to the range and azimuth axes in the range-azimuth spectrum diagram, respectively. Based on the local coordinates, the area around the target is divided with the target as the center area to obtain multiple other areas. In this way, the cross-modal fusion point features related to the target are classified using a classification network to achieve coarse classification. For the radar points within the two-dimensional detection box, based on a set of discrete phase models with a specific resolution, the relative distance to the target center position on the two axes is encoded. Then, regression is performed based on the classification results, and further prediction errors are used to finally determine the target position.

[0086] In this embodiment, each target center point is provided by millimeter-wave radar detection. The target center point provided by the millimeter-wave radar is compared with the real-world coordinate system of the radar point cloud to obtain a local coordinate system. In the specific implementation process, the target center point is used during the encoding phase to generate the ground truth of the local coordinate system of the radar point cloud for supervised learning of the network. After training is completed, the target center point is no longer needed to generate the ground truth of the local coordinate system.

[0087] Therefore, the local coordinate system with the target center as its origin plays a role in encoding and filtering throughout the entire process, serving as prior information for supervised training. The relative distance between the target center and the radar point is obtained by discretizing the target dimensions into azimuth and range, and modeling them using 5 and 11 bins, respectively.

[0088] Furthermore, during the encoding phase of the radar point cloud, the real-world coordinate system of the radar is not directly used for encoding. Instead, the relative position of the point cloud to the target center point is used for encoding, i.e., the concept of a local coordinate system. After obtaining the relative position of the radar point cloud using the target center point detected by the radar, this relative position is encoded as a 5-bin azimuth and an 11-bin distance. This encoding process also performs a filtering operation. That is, radar point clouds with relative positions outside the 5-bin and 11-bin ranges are not input into the network for prediction and are therefore not considered candidate point clouds.

[0089] It's important to note that while radar points surrounding the target are considered candidate points, they also fall within the bounding box (here, the bounding box refers to the area within 5 bins of azimuth or 11 bins of distance from the central region). However, points reflected by the target's context also have value. These points are considered "non-trivial" if their relative distances along any axis satisfy the discretization mentioned above. For example, points reflected from non-object areas within the bounding box may exhibit significant variation in range but exhibit strong correlation with the object along the angular dimension. In this case, these points can still be used as training data, and the prediction network can be trained solely on the deviations from the angular predictions. Furthermore, this method can filter out radar foreground and background points, eliminating range uncertainty.

[0090] In this embodiment, obtaining the target position by processing according to the classification result and the quantization error includes: decoding the classification result and the quantization error to obtain the target position.

[0091] In this embodiment, classification and regression based on cross-modal fusion point features and target local coordinates can be implemented using neural networks respectively, and cross entropy loss and smoothing are used when training the classification and regression neural networks. The combined loss function constructed by the loss is trained, and the combined loss function is expressed as:

[0092] (3)

[0093] In formula (3), , They represent the prediction results of the neural network for classification and the neural network for regression, respectively.

[0094] In this embodiment, radar point cloud data is also used to provide the real-world coordinates of the radar point cloud during training of the classification and regression neural networks. During neural network encoding, the relative coordinates, i.e., the ground truth values ​​in the local coordinate system, are obtained by subtracting each radar point cloud from the target center point. This ground truth value is then used to supervise neural network training.

[0095] In this embodiment, the data processing flow in this method can also be as follows Figure 5 As shown in the figure, after deleting the radar point cloud and local coordinate system in the input detection head, it can also be represented that the above processing networks and steps are integrated to obtain a complete target detection network structural diagram.

[0096] In the above-mentioned target detection method based on radar-optical cross-modal feature point fusion, the backbone network is used to extract features from the acquired optical image and radar RD spectrum data respectively to obtain semantic dense point features and RD dense point features, and the radar point cloud is projected onto the optical image. According to the position of the two-dimensional detection box on the semantic dense point feature, the radar point cloud corresponding to the region of interest is obtained on the optical image, and then this part of the radar point cloud is projected onto the semantic dense point feature and the RD dense point feature respectively to obtain the corresponding features on the region of interest, and the two features are fused using the fusion network to obtain fused point features, and then local coordinates are constructed based on the target, and the region of interest is divided into multiple regions with the local coordinates as the origin, and the problem of target detection is converted into a classification problem. The classification neural network is used to perform regional classification on the fused point features to realize target detection. The method can improve the accuracy of target detection and effectively detect targets in harsh environments.

[0097] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0098] In one embodiment, Figure 6 As shown, a target detection device based on radar-optical cross-modal feature point fusion is provided, comprising: a detection data acquisition module 200, a dense point feature extraction module 210, a dense feature point extraction module 220 corresponding to a region of interest, a cross-modal fusion point feature acquisition module 230, a local coordinate construction module 240, and a target detection module 250, wherein:

[0099] The detection data acquisition module 200 is used to obtain radar echo data and optical images, wherein the radar echo data and optical images are respectively obtained by detecting the target area by a millimeter wave radar system and a camera, wherein the radar echo data includes radar range and velocity spectrum data and radar point cloud data;

[0100] The dense point feature extraction module 210 is used to input the optical image and radar range and velocity spectrum data into the corresponding backbone network to obtain semantic dense point features, two-dimensional detection boxes, and range and azimuth dense point features respectively;

[0101] The dense feature point extraction module 220 corresponding to the target region of interest is configured to process the two-dimensional detection frame, the radar point cloud data, and the optical image to obtain partial radar point cloud data corresponding to the target region of interest, and obtain corresponding partial semantic dense point features and partial range and azimuth dense point features from the semantic dense point features and the range and azimuth dense point features based on the partial radar point cloud data;

[0102] A cross-modal fusion point feature obtaining module 230 is used to input the partial semantic dense point features and the partial distance and azimuth dense point features into a feature fusion network to obtain cross-modal fusion point features;

[0103] A local coordinate construction module 240 is used to construct local coordinates centered on the target, and divide the target region of interest into multiple regions with the local coordinates as the origin;

[0104] The target detection module 250 is used to obtain a classification result by regionally classifying the cross-modal fusion point features relative to the local coordinates using a classification neural network, as well as a quantization error generated during the discretization process, and to obtain a target position based on the classification result and the quantization error to achieve target detection.

[0105] For the specific definition of the target detection device based on radar-optical cross-modal feature point fusion, please refer to the definition of the target detection method based on radar-optical cross-modal feature point fusion above, which will not be repeated here. The various modules in the above-mentioned target detection device based on radar-optical cross-modal feature point fusion can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0106] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a target detection method based on radar-optical cross-modal feature point fusion is implemented. The display screen of the computer device can be a liquid crystal display or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0107] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0108] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0109] Acquire radar echo data and an optical image, wherein the radar echo data and the optical image are respectively obtained by detecting the target area by a millimeter wave radar system and a camera, wherein the radar echo data includes radar range and velocity spectrum data and radar point cloud data;

[0110] Input the optical image and radar range and velocity spectrum data into the corresponding backbone network to obtain semantic dense point features, two-dimensional detection boxes, and range and azimuth dense point features respectively;

[0111] Processing the two-dimensional detection frame, the radar point cloud data, and the optical image to obtain partial radar point cloud data corresponding to the target area of ​​interest, and obtaining corresponding partial semantic dense point features and partial range and azimuth dense point features from the semantic dense point features and the range and azimuth dense point features based on the partial radar point cloud data;

[0112] Inputting some of the semantic dense point features and some of the distance and azimuth dense point features into a feature fusion network to obtain cross-modal fusion point features;

[0113] Constructing a local coordinate system centered on a target, and dividing the target region of interest into a plurality of regions with the local coordinate system as an origin;

[0114] The classification result is obtained by performing regional classification on the cross-modal fusion point features relative to the local coordinates using a classification neural network, as well as the quantization error generated during the discretization process. The target position is obtained based on the classification result and the quantization error to achieve target detection.

[0115] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0116] Acquire radar echo data and an optical image, wherein the radar echo data and the optical image are respectively obtained by detecting the target area by a millimeter wave radar system and a camera, wherein the radar echo data includes radar range and velocity spectrum data and radar point cloud data;

[0117] Input the optical image and radar range and velocity spectrum data into the corresponding backbone network to obtain semantic dense point features, two-dimensional detection boxes, and range and azimuth dense point features respectively;

[0118] Processing the two-dimensional detection frame, the radar point cloud data, and the optical image to obtain partial radar point cloud data corresponding to the target area of ​​interest, and obtaining corresponding partial semantic dense point features and partial range and azimuth dense point features from the semantic dense point features and the range and azimuth dense point features based on the partial radar point cloud data;

[0119] Inputting some of the semantic dense point features and some of the distance and azimuth dense point features into a feature fusion network to obtain cross-modal fusion point features;

[0120] Constructing a local coordinate system centered on a target, and dividing the target region of interest into a plurality of regions with the local coordinate system as an origin;

[0121] The classification result is obtained by performing regional classification on the cross-modal fusion point features relative to the local coordinates using a classification neural network, as well as the quantization error generated during the discretization process. The target position is obtained based on the classification result and the quantization error to achieve target detection.

[0122] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0123] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0124] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A target detection method based on radar-optical cross-modal feature point fusion, characterized in that: The method comprises: Acquire radar echo data and an optical image, wherein the radar echo data and the optical image are respectively obtained by detecting the target area by a millimeter wave radar system and a camera, wherein the radar echo data includes radar range and velocity spectrum data and radar point cloud data; Input the optical image and radar range and velocity spectrum data into the corresponding backbone network to obtain semantic dense point features, two-dimensional detection boxes, and range and azimuth dense point features respectively; Processing the two-dimensional detection frame, the radar point cloud data, and the optical image to obtain partial radar point cloud data corresponding to the target area of ​​interest, and obtaining corresponding partial semantic dense point features and partial range and azimuth dense point features from the semantic dense point features and the range and azimuth dense point features based on the partial radar point cloud data; Inputting the part of semantic dense point features and the part of distance azimuth dense point features into the feature fusion network to obtain cross-modal fusion point features, wherein the feature fusion network includes a multi-layer perceptron on both sides and a maximum pooling layer, wherein the multi-layer perceptron on both sides includes a first layer of perceptron and a second layer of perceptron, inputting the part of semantic dense point features and the part of distance azimuth dense point features into the feature fusion network to obtain cross-modal fusion point features includes: respectively inputting the part of semantic dense point features and the part of distance azimuth dense point features into the multi-layer perceptron on both sides of the corresponding side, respectively inputting the part of semantic dense point features and the part of distance azimuth dense point features into the multi-layer perceptron on both sides of the corresponding side, respectively Input the corresponding first-layer perceptron for feature extraction, and obtain a first semantic feature and a first distance azimuth feature, and fuse the two features to obtain a low-level feature of the target. Input the first semantic feature and the first distance azimuth feature into the corresponding second-layer perceptron for feature extraction, and obtain a second semantic feature and a second distance azimuth feature, and fuse the two obtained features to obtain a high-level feature of the target. The low-level feature and the high-level feature of the target are passed through the maximum pooling layer to obtain a global feature of the target. The low-level feature, the high-level feature and the global feature of the target are fused to obtain the cross-modal fusion point feature. Constructing a local coordinate system centered on a target, and dividing the target region of interest into a plurality of regions with the local coordinate system as an origin; The classification result is obtained by performing regional classification on the cross-modal fusion point features relative to the local coordinates using a classification neural network, as well as the quantization error generated during the discretization process. The target position is obtained based on the classification result and the quantization error to achieve target detection.

2. The target detection method according to claim 1, wherein: The backbone network corresponding to the optical image adopts a ResNet-18 network, and the optical image is processed by the ResNet-18 network to obtain the semantic dense point features and the two-dimensional detection frame.

3. The target detection method according to claim 1, wherein: The backbone network corresponding to the radar RD spectrum data adopts the FFT-RadNet network, and the radar RD spectrum data is processed by the FFT-RadNet network to obtain the range and azimuth dense point features.

4. The target detection method according to claim 1, wherein: The processing of the two-dimensional detection frame, the radar point cloud data, and the optical image to obtain partial radar point cloud data corresponding to the target area of ​​interest includes: The radar point cloud data is projected onto the optical image, and the radar point cloud projected onto the optical image is filtered according to the position of the two-dimensional detection box on the semantic dense point feature to obtain partial radar point cloud data corresponding to the target area of ​​interest.

5. The target detection method according to claim 4, characterized in that: The obtaining corresponding partial semantic dense point features and partial range and azimuth dense point features from the semantic dense point features and the range and azimuth dense point features according to the partial radar point cloud data includes: The part of the radar point cloud data is projected onto the semantic dense point features and the range and azimuth dense point features respectively to obtain the part of the semantic dense point features and the part of the range and azimuth dense point features respectively.

6. The target detection method according to claim 1, wherein: The obtaining of the target position by processing the classification result and the quantization error includes: decoding the classification result and the quantization error to obtain the target position.

7. A target detection device based on radar-optical cross-modal feature point fusion, characterized in that: The device comprises: A detection data acquisition module is used to obtain radar echo data and optical images, wherein the radar echo data and optical images are respectively obtained by detecting the target area by a millimeter wave radar system and a camera, wherein the radar echo data includes radar range and velocity spectrum data and radar point cloud data; A dense point feature extraction module is used to input the optical image and radar range and velocity spectrum data into the corresponding backbone network to obtain semantic dense point features, two-dimensional detection boxes, and range and azimuth dense point features respectively; a dense feature point extraction module corresponding to the region of interest, configured to obtain partial radar point cloud data corresponding to the target region of interest based on the two-dimensional detection frame, the radar point cloud data, and the optical image, and obtain corresponding partial semantic dense point features and partial range and azimuth dense point features from the semantic dense point features and the range and azimuth dense point features based on the partial radar point cloud data; A cross-modal fusion point feature acquisition module is used to input the partial semantic dense point features and partial distance azimuth dense point features into a feature fusion network to obtain cross-modal fusion point features, wherein the feature fusion network includes a multi-layer perceptron on both sides and a maximum pooling layer, wherein the multi-layer perceptron on both sides includes a first layer of perceptron and a second layer of perceptron, inputting the partial semantic dense point features and partial distance azimuth dense point features into the feature fusion network to obtain cross-modal fusion point features, including: respectively inputting the partial semantic dense point features and partial distance azimuth dense point features into the multi-layer perceptron on both sides of the corresponding side, respectively inputting the partial semantic dense point features and partial distance azimuth dense point features into the multi-layer perceptron on both sides of the corresponding side, respectively The angle dense point feature is input into the corresponding first layer perceptron for feature extraction, and the first semantic feature and the first distance azimuth feature are obtained accordingly, and the two features are fused to obtain the low-level feature of the target. The first semantic feature and the first distance azimuth feature are respectively input into the corresponding second layer perceptron for feature extraction, and the second semantic feature and the second distance azimuth feature are obtained accordingly, and the two obtained features are fused to obtain the high-level feature of the target. The low-level feature and the high-level feature of the target are passed through the maximum pooling layer to obtain the global feature of the target. The low-level feature, the high-level feature and the global feature of the target are fused to obtain the cross-modal fusion point feature; A local coordinate construction module is used to construct local coordinates centered on the target, and divide the target region of interest into multiple regions with the local coordinates as the origin; The target detection module is used to obtain a classification result by regionally classifying the cross-modal fusion point features relative to the local coordinates using a classification neural network, as well as a quantization error generated during the discretization process, and to obtain a target position based on the classification result and the quantization error to achieve target detection.

Images