Obstacle detection methods, devices, electronic equipment and storage media

Abstract

This application provides an obstacle detection method, apparatus, electronic device, and storage medium, comprising: acquiring raw millimeter-wave radar data and camera images corresponding to the area around a vehicle using millimeter-wave radar and a camera, respectively; preprocessing the raw millimeter-wave radar data to obtain millimeter-wave radar images; acquiring target millimeter-wave radar image features and target camera image features; fusing the target millimeter-wave radar image features and target camera image features to obtain target image fusion features; constructing a multi-adversarial learning model based on the raw millimeter-wave radar data, camera images, and target image fusion features; and determining whether obstacles exist around the vehicle based on the multi-adversarial learning model. Specifically, in this embodiment, by extracting dual-modal fusion features and then constructing a multi-adversarial learning model, the fused image can simultaneously retain the most important feature information of each modality, achieving accurate obstacle identification under complex conditions.

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology, and in particular to an obstacle detection method, device, electronic device, and storage medium. Background Technology

[0002] In the field of autonomous driving, commonly used obstacle detection sensors include cameras, millimeter-wave radar, and lidar. Sensors need to maintain their accuracy and robustness under diverse, often adverse, environmental conditions. Most sensors can only meet one of these requirements, not both simultaneously. For example, camera systems produce accurate environmental images but are highly sensitive in low light conditions. Radar sensors meet robustness requirements but their accuracy is far lower than cameras. LiDAR offers high resolution but is expensive and sensitive to weather and atmospheric conditions. In contrast, millimeter-wave radar is less expensive and more robust to environmental factors. Even in harsh environmental conditions, millimeter-wave radar demonstrates powerful capabilities, helping to accurately and quickly acquire obstacle target attributes, including speed and distance. Therefore, multi-sensor fusion technology based on millimeter-wave radar and cameras has become a key approach to obstacle detection in autonomous driving. By combining the advantages of different sensors, multi-sensor fusion can overcome the limitations of individual sensors and achieve better obstacle detection performance.

[0003] In related technologies, multimodal stitched data is used as network input, and single-modal images are used as real data to optimize the training of the generator network. This usually causes the generated image to be biased towards the features of a certain single-modal image, while ignoring the correlation between the two modalities. As a result, the obstacle detection results are easily affected by interference factors such as radar noise and visual blur, which leads to a decrease in detection accuracy. Summary of the Invention

[0004] This application provides an obstacle detection method, apparatus, system, electronic device, and computer-readable storage medium.

[0005] In a first aspect, embodiments of this application provide an obstacle detection method, the method comprising:

[0006] The vehicle's surroundings are captured using millimeter-wave radar and camera data, respectively.

[0007] The raw millimeter-wave radar data is preprocessed to obtain a millimeter-wave radar image.

[0008] Obtain the target millimeter-wave radar image features and the target camera image features corresponding to the millimeter-wave radar image and the camera image, respectively;

[0009] The target millimeter-wave radar image features and the target camera image features are fused to obtain the target image fusion features;

[0010] A multi-adversarial learning model is constructed based on the original millimeter-wave radar data, the camera images, and the fusion features of the target images.

[0011] The multi-adversarial learning model is used to determine whether there are obstacles around the vehicle.

[0012] Optionally, acquiring the target millimeter-wave radar image features and target camera image features corresponding to the millimeter-wave radar image and the camera image, respectively, includes:

[0013] Obtain the correlation of features between the millimeter-wave radar image and the camera image;

[0014] Based on the correlation, weights are assigned to obtain the target millimeter-wave radar image features and the target camera image features.

[0015] Optionally, obtaining the correlation of features between the millimeter-wave radar image and the camera image includes:

[0016] The millimeter-wave radar image and the camera image are respectively extracted using the same preset backbone network to obtain millimeter-wave radar feature map and camera image feature map;

[0017] The millimeter-wave radar feature map and the camera image feature map are input into the modal similarity generation module to obtain the correlation between the features of the millimeter-wave radar image and the camera image.

[0018] Optionally, the correlation includes a first correlation based on the millimeter-wave radar image and a second correlation based on the camera image, and the step of assigning weights according to the correlation to obtain the target millimeter-wave radar image features and the target camera image features includes:

[0019] Assign a first weight to the millimeter-wave radar feature map based on the first correlation, and assign a second weight to the camera image feature map based on the second correlation;

[0020] The target millimeter-wave radar image features are obtained based on the first weight and the millimeter-wave radar feature map, and the target camera image features are obtained based on the second weight and the camera image feature map.

[0021] Optionally, the construction of the multi-adversarial learning model based on the original millimeter-wave radar data, the camera image, and the target image fusion features includes:

[0022] The raw millimeter-wave radar data and the camera images are used as inputs to the generator;

[0023] The original millimeter-wave radar data and the target image fusion features are used as inputs to the first discriminator;

[0024] The fused features of the camera image and the target image are used as input to the second discriminator;

[0025] A multi-adversarial learning model is constructed based on the generator, the first discriminator, and the second discriminator.

[0026] Optionally, the step of preprocessing the raw millimeter-wave radar data to obtain a millimeter-wave radar image includes:

[0027] The raw millimeter-wave radar data is mapped onto the image plane to obtain a millimeter-wave radar image.

[0028] Optionally, the step of fusing the target millimeter-wave radar image features and the target camera image features to obtain target image fusion features includes:

[0029] The target millimeter-wave radar image features and the target camera image features are input into a two-layer hybrid pooling attention module, and feature fusion is performed in parallel at two layers to obtain target image fusion features.

[0030] Secondly, embodiments of this application provide an obstacle detection device, the device comprising:

[0031] The device includes:

[0032] The acquisition module is used to acquire raw millimeter-wave radar data and camera images of the vehicle's surroundings using millimeter-wave radar and camera, respectively.

[0033] The preprocessing module is used to preprocess the raw millimeter-wave radar data to obtain a millimeter-wave radar image.

[0034] The acquisition module is used to acquire the target millimeter-wave radar image features and target camera image features corresponding to the millimeter-wave radar image and the camera image, respectively.

[0035] The fusion module is used to fuse the target millimeter-wave radar image features and the target camera image features to obtain target image fusion features;

[0036] The construction module is used to build a multi-adversarial learning model based on the original millimeter-wave radar data, the camera images, and the fusion features of the target image;

[0037] An obstacle determination module is used to determine whether there are obstacles around the vehicle based on the multi-adversarial learning model.

[0038] Optionally, the acquisition module includes:

[0039] The first acquisition submodule is used to acquire the correlation of features between the millimeter-wave radar image and the camera image;

[0040] The second acquisition submodule is used to assign weights according to the correlation to obtain the target millimeter-wave radar image features and the target camera image features.

[0041] Optionally, the first acquisition submodule includes:

[0042] The first acquisition unit is used to extract features from the millimeter-wave radar image and the camera image respectively through the same preset backbone network to obtain millimeter-wave radar feature map and camera image feature map;

[0043] The second acquisition unit is used to input the millimeter-wave radar feature map and the camera image feature map into the modal similarity generation module to obtain the correlation between the features of the millimeter-wave radar image and the camera image.

[0044] Optionally, the correlation includes a first correlation based on millimeter-wave radar images and a second correlation based on the camera images, wherein the second acquisition submodule includes:

[0045] The third acquisition unit is configured to assign a first weight to the millimeter-wave radar feature map based on the first correlation, and to assign a second weight to the camera image feature map based on the second correlation.

[0046] The fourth acquisition unit is used to obtain target millimeter-wave radar image features based on the first weight and the millimeter-wave radar feature map, and to obtain target camera image features based on the second weight and the camera image feature map.

[0047] Optionally, the building module includes:

[0048] The first construction submodule is used to take the raw millimeter-wave radar data and the camera image as input to the generator;

[0049] The second construction submodule is used to take the original millimeter-wave radar data and the target image fusion features as input to the first discriminator;

[0050] The third construction submodule is used to use the fused features of the camera image and the target image as input to the second discriminator;

[0051] The fourth construction submodule is used to construct a multi-adversarial learning model based on the generator, the first discriminator, and the second discriminator.

[0052] Optionally, the preprocessing module includes:

[0053] The preprocessing submodule is used to map the raw millimeter-wave radar data onto the image plane to obtain a millimeter-wave radar image.

[0054] Optionally, the fusion module includes:

[0055] The fusion submodule is used to input the target millimeter-wave radar image features and the target camera image features into the two-layer hybrid pooling attention module, and perform feature fusion through two-layer parallel processing to obtain the target image fusion features.

[0056] Thirdly, embodiments of this application also provide an electronic device, including: a processor; and a memory for storing processor-executable instructions, wherein the processor is configured to execute the instructions to implement any of the obstacle detection methods described above.

[0057] Fourthly, embodiments of this application also provide a storage medium that, when the instructions in the storage medium are executed by the processor of an electronic device, enables the electronic device to perform any of the obstacle detection methods described above.

[0058] Fifthly, embodiments of this application also provide a vehicle, the vehicle including the obstacle detection device described above.

[0059] In this embodiment, millimeter-wave radar and camera images are acquired around the vehicle, respectively. The original millimeter-wave radar data is preprocessed to obtain millimeter-wave radar images. Target millimeter-wave radar image features and target camera image features corresponding to the millimeter-wave radar images and camera images are obtained, respectively. The target millimeter-wave radar image features and target camera image features are fused to obtain target image fusion features. A multi-adversarial learning model is constructed based on the original millimeter-wave radar data, the camera images, and the target image fusion features. The presence of obstacles around the vehicle is determined based on the multi-adversarial learning model. In other words, in this embodiment, feature extraction and fusion between the original millimeter-wave radar data and camera images are achieved through preprocessing, resulting in efficient extraction of dual-modal fusion features. Furthermore, a dual-discriminator multi-adversarial learning model is constructed based on the original millimeter-wave radar data, camera images, and target image fusion features to ensure that the fused image simultaneously retains the most important feature information of each modality, ultimately achieving accurate obstacle identification under complex conditions.

[0060] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0061] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0062] Figure 1 This is a flowchart illustrating the steps of an obstacle detection method provided in an embodiment of this application;

[0063] Figure 2 This is a flowchart of another obstacle detection method provided in an embodiment of this application;

[0064] Figure 3 This is a flowchart of another obstacle detection method provided in an embodiment of this application;

[0065] Figure 4 This is a block diagram of an obstacle detection device provided in an embodiment of this application;

[0066] Figure 5 This is a structural diagram of an electronic device provided in an embodiment of this application;

[0067] Figure 6 This is an exemplary flowchart of a dual-discriminator GAN obstacle detection based on multimodal fusion provided in an embodiment of this application;

[0068] Figure 7 This is a schematic diagram of an exemplary modal similarity generation module structure provided in an embodiment of this application;

[0069] Figure 8 This is a schematic diagram of an exemplary two-layer hybrid pooling attention module structure provided in an embodiment of this application;

[0070] Figure 9 This is a flowchart of an exemplary dual-discriminator multi-adversarial learning model provided in an embodiment of this application. Detailed Implementation

[0071] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0072] Figure 1 This is a flowchart illustrating the steps of an obstacle detection method provided in an embodiment of this application, as follows: Figure 1 As shown, the method may include:

[0073] Step 101: Collect raw millimeter-wave radar data and camera images of the area around the vehicle using millimeter-wave radar and camera respectively.

[0074] It should be noted that, in this embodiment of the application, in order to ensure the accuracy of obstacle detection, the correlation between the two modalities, namely millimeter-wave radar data and camera images, needs to be considered. Specifically, refer to... Figure 6 The system uses millimeter-wave radar and cameras to collect raw millimeter-wave radar data and camera images of the area around the vehicle, respectively.

[0075] Step 102: Preprocess the raw millimeter-wave radar data to obtain a millimeter-wave radar image;

[0076] Furthermore, the preprocessing of the raw millimeter-wave radar data to obtain a millimeter-wave radar image includes:

[0077] The raw millimeter-wave radar data is mapped onto the image plane to obtain a millimeter-wave radar image.

[0078] It should be noted that, in the embodiments of this application, the millimeter-wave radar data is processed and the radar data is projected onto the image plane as a single pixel.

[0079] Specifically, the millimeter-wave radar data is processed first. This requires establishing coordinate transformation relationships between the radar coordinate system, world coordinate system, and image coordinate system to project the radar data onto the image plane. To project the radar data to the corresponding positions in the image, multiple coordinate transformations are necessary. First, the radar points in the radar coordinate system are rotated and transformed into the world coordinate system. Then, through another rotation and translation, they are transformed into the camera coordinate system. Finally, using the camera's inherent parameters, the camera coordinates are transformed into image coordinates. The transformation of radar points from the radar coordinate system to the world coordinate system can be expressed as:

[0080]

[0081] In Formula 1 above, [Xr,Yr,1]T and [Xw,Yw,1]T represent the coordinates of the radar point in the radar coordinate system and the world coordinate system, respectively. R1 and T1 are the rotation and translation matrices during the transformation process. The radar detection scanning plane is a two-dimensional plane, which can obtain the target's X and Y coordinate information, but lacks the target's Z coordinate information. Therefore, the coordinates obtained in the world coordinate system at this time are also two-dimensional. The missing Z coordinate information can be given by prior knowledge. After obtaining the X, Y, and Z coordinates in the world coordinate system, they are transformed from the world coordinate system to the image coordinate system, expressed as:

[0082]

[0083] In Formula 2 above, [Xw,Yw,Zw,1]T represents the X, Y, and Z coordinates in the world coordinate system. [Xi,Yi,1]T represents the coordinates of the radar point in the image coordinate system. R2 and T2 represent the rotation and translation matrices.

[0084] The above process maps the raw radar data onto an image plane. In this image, radar features are stored as pixel values ​​in the projected image. However, since the radar data is projected onto the image plane as individual pixels, information about the object's height and width is lacking. This poses a challenge for extracting radar features and, consequently, impacts the performance of the network input. Therefore, to address this issue, we can assume that the 3D coordinates detected by the radar are returned from the ground, extend the radar projection along a direction perpendicular to the ground, extend the radar height to 2.5m, and map it onto the image plane with a pixel width of 1.

[0085] Step 103: Obtain the target millimeter-wave radar image features and target camera image features corresponding to the millimeter-wave radar image and the camera image, respectively;

[0086] It should be noted that traditional GANs consist of two adversarial models: a generator and a discriminator. Therefore, the embodiments in this application are based on a dual-discriminator multi-adversarial learning model using multimodal fusion.

[0087] To achieve full fusion of millimeter-wave radar and camera image features, the fusion detection model must consider the complementarity of the two source information and be able to adaptively adjust or reweight the provided information. Therefore, we first construct a modal similarity generation module in the generator to obtain the target millimeter-wave radar image features and target camera image features corresponding to the millimeter-wave radar image and the target camera image, respectively. The structure is as follows: Figure 7 As shown.

[0088] Step 104: Perform feature fusion on the target millimeter-wave radar image features and the target camera image features to obtain target image fusion features;

[0089] It should be noted that after obtaining the target millimeter-wave radar image features and target camera image features corresponding to the millimeter-wave radar image and the camera image respectively in step 103, feature fusion needs to be performed based on the correlation and these features in step 104.

[0090] Step 105: Construct a multi-adversarial learning model based on the original millimeter-wave radar data, the camera images, and the fusion features of the target images;

[0091] It should be noted that for the multimodal image fusion task, the original millimeter-wave radar data I1 containing different targets and structures and camera images I2 are used as inputs to the generator G to construct two discriminators, namely the first discriminator D1 and the second discriminator D2, and then a multi-adversarial learning model is constructed.

[0092] Step 106: Determine whether there are obstacles around the vehicle based on the multi-adversarial learning model.

[0093] It should be noted that, in the embodiments of this application, after constructing multiple adversarial learning models, adversarial learning can be performed based on the multiple adversarial learning models. The generated images can retain the rich features of the original images while optimizing the generation of fused images, and then the final fused image can be used to identify obstacles around the vehicle.

[0094] Specifically, the generator attempts to produce realistic images or data of obstacles, while the discriminator tries to distinguish between real and generated data. In adversarial training, the model tries to learn how to recognize obstacles, while the adversarial network attempts to generate obstacle data that is difficult to detect.

[0095] Determining the presence of obstacles around a vehicle using a multi-adversarial learning model can improve the robustness and accuracy of obstacle detection systems.

[0096] In this embodiment, millimeter-wave radar and camera images are acquired around the vehicle, respectively. The original millimeter-wave radar data is preprocessed to obtain millimeter-wave radar images. Target millimeter-wave radar image features and target camera image features corresponding to the millimeter-wave radar images and camera images are obtained, respectively. The target millimeter-wave radar image features and target camera image features are fused to obtain target image fusion features. A multi-adversarial learning model is constructed based on the original millimeter-wave radar data, the camera images, and the target image fusion features. The presence of obstacles around the vehicle is determined based on the multi-adversarial learning model. In other words, in this embodiment, feature extraction and fusion between the original millimeter-wave radar data and camera images are achieved through preprocessing, resulting in efficient extraction of dual-modal fusion features. Furthermore, a dual-discriminator multi-adversarial learning model is constructed based on the original millimeter-wave radar data, camera images, and target image fusion features to ensure that the fused image simultaneously retains the most important feature information of each modality, ultimately achieving accurate obstacle identification under complex conditions.

[0097] Figure 2 This is a flowchart of another obstacle detection method provided in an embodiment of this application, as follows: Figure 2 As shown, the method may include:

[0098] Step 201: Collect raw millimeter-wave radar data and camera images of the area around the vehicle using millimeter-wave radar and camera respectively.

[0099] Step 202: Preprocess the raw millimeter-wave radar data to obtain a millimeter-wave radar image;

[0100] It should be noted that steps 201-202 above are based on the previous discussion and will not be repeated here.

[0101] Step 203: Obtain the correlation of features between the millimeter-wave radar image and the camera image;

[0102] Furthermore, the correlation of features between the millimeter-wave radar image and the camera image includes:

[0103] The millimeter-wave radar image and the camera image are respectively extracted using the same preset backbone network to obtain millimeter-wave radar feature map and camera image feature map;

[0104] The millimeter-wave radar feature map and the camera image feature map are input into the modal similarity generation module to obtain the correlation between the features of the millimeter-wave radar image and the camera image.

[0105] Step 204: Assign weights according to the correlation to obtain the target millimeter-wave radar image features and the target camera image features;

[0106] Further, the correlation includes a first correlation based on the millimeter-wave radar image and a second correlation based on the camera image, and the step of assigning weights according to the correlation to obtain the target millimeter-wave radar image features and the target camera image features includes:

[0107] Assign a first weight to the millimeter-wave radar feature map based on the first correlation, and assign a second weight to the camera image feature map based on the second correlation;

[0108] The target millimeter-wave radar image features are obtained based on the first weight and the millimeter-wave radar feature map, and the target camera image features are obtained based on the second weight and the camera image feature map.

[0109] It should be noted that, in the embodiments of this application, in the generator, the first step is to use the same backbone network to extract millimeter-wave radar features and camera image features to improve the efficiency of the model.

[0110] Specifically, VGG19 can be used to extract features from the input millimeter-wave radar image and camera image respectively, resulting in a millimeter-wave radar feature map H1 and a camera image feature map H2, both with a size of (C×H×W). This leverages the ability of deep networks to extract semantic information from the input data. H1 and H2 are then input into a modality similarity generation module, treating each channel of H1 and H2 as a node, with (H×W) representing the node's features and information about that channel within the entire image.

[0111] Then, graph convolution operations are used to mine the topological relationships between channels, which are used to update the feature representations between channels to obtain X and Y. The feature map size is set to (C×H×W).

[0112] The updated features are then used to calculate the similarity between the two modalities (X and Y). The calculation process is as follows:

[0113]

[0114] In formulas 3 and 4 above, S1 and S2 represent the correlation scores between the two modalities. The mean values ​​of the feature maps for the two modalities are denoted by μx and μy, respectively. The L2 norm is represented by the symbol ||||.

[0115] Then, assigning more weight to modalities with high relevance scores and increasing the weight of important features helps improve obstacle recognition accuracy. X For target millimeter-wave radar image features and A Y The target camera image features, i.e., A X and A Y This is the final refined output. The calculation process is as follows:

[0116] A X =XS1(Formula 5)

[0117] A Y =YS2(Formula 6)

[0118] The modal similarity generation module improves the single-modal representation of information and uncovers the similarities between millimeter-wave radar and camera modes.

[0119] Step 205: Input the target millimeter-wave radar image features and the target camera image features into the dual-layer hybrid pooling attention module, and perform feature fusion through dual-layer parallel processing to obtain the target image fusion features;

[0120] It should be noted that, in this embodiment of the application, a two-layer hybrid pooling attention module is constructed, the structure of which is as follows: Figure 8 As shown.

[0121] The generated A X and AY As input to the dual-layer hybrid pooling attention module, the feature map size is (C×H×W). The dual-modal features are further interacted and fused in a dual-layer parallel manner.

[0122] Specifically, the first layer extracts A. X Features are used to guide A Y The interaction and fusion of features, taking into account both global max pooling and global average pooling, for A X The global features are aggregated, and then a 1DCNN network is used to reduce the number of parameters while effectively realizing local cross-channel interaction. The generated attention coefficients are of size (C×1×1) and are used to guide A. Y Feature extraction.

[0123] Second layer extraction A Y Features are used to guide A X Feature interaction fusion, using the same processing method for A Y The global features are aggregated, and then a 1DCNN network is used to perform local cross-channel interactions to generate attention coefficients of size (C×1×1), which are used to guide A. X Feature extraction. Employing a two-layer parallel approach enhances the interaction and information transfer between different modalities, enabling bimodal features to better complement and collaborate. This helps extract richer and more discriminative feature representations.

[0124] Furthermore, the features generated by the first and second layers are then added and fused to obtain the fused image feature A. XY .

[0125] Step 206: Construct a multi-adversarial learning model based on the original millimeter-wave radar data, the camera images, and the fusion features of the target images;

[0126] Step 207: Determine whether there are obstacles around the vehicle based on the multi-adversarial learning model.

[0127] It should be noted that steps 206-207 above are based on the previous discussion and will not be repeated here.

[0128] In this embodiment, feature extraction and fusion between the original millimeter-wave radar data and camera images are achieved through preprocessing. The dual-modal fusion features are efficiently extracted, and then a multi-adversarial learning model with dual discriminators is constructed based on the fusion features of the original millimeter-wave radar data, camera images, and target images. This ensures that the fused image can simultaneously retain the most important feature information of each modality image, ultimately achieving accurate identification of obstacles under complex conditions.

[0129] In addition, in this embodiment, a modal similarity generation module is designed in the generator of the generative adversarial network to mine the similarities between the two modes. Then, a two-layer hybrid pooling attention module is constructed to extract the interaction and information transmission between modal information in parallel, so that the dual modal features can better complement each other and work together.

[0130] Figure 3 This is a flowchart of another obstacle detection method provided in an embodiment of this application, as follows: Figure 3 As shown, the method may include:

[0131] Step 301: Collect raw millimeter-wave radar data and camera images of the area around the vehicle using millimeter-wave radar and camera respectively.

[0132] Step 302: Preprocess the raw millimeter-wave radar data to obtain a millimeter-wave radar image;

[0133] Step 303: Obtain the target millimeter-wave radar image features and target camera image features corresponding to the millimeter-wave radar image and the camera image, respectively;

[0134] Step 304: Perform feature fusion on the target millimeter-wave radar image features and the target camera image features to obtain target image fusion features;

[0135] It should be noted that steps 301-304 above are based on the previous discussion and will not be repeated here.

[0136] Step 305: Use the raw millimeter-wave radar data and the camera image as input to the generator;

[0137] Step 306: Use the original millimeter-wave radar data and the target image fusion features as input to the first discriminator;

[0138] Step 307: Use the fused features of the camera image and the target image as input to the second discriminator;

[0139] Step 308: Construct a multi-adversarial learning model based on the generator, the first discriminator, and the second discriminator;

[0140] It should be noted that, in the embodiments of this application, in steps 305-308 above, the construction of a multi-adversarial learning model with dual discriminators is carried out as follows: Figure 9 As shown.

[0141] For the multimodal image fusion task, the original millimeter-wave radar data I1 containing different targets and structures and camera images I2 are used as inputs to the generator G to construct two discriminators, D1 and D2, namely the first discriminator and the second discriminator.

[0142] The target image fusion features generated in the preceding step 304, which allow discriminators D1 and D2 to correctly identify the target image fusion feature, i.e., the fused image A, are used to generate the target image fusion feature. XY Using raw millimeter-wave radar images and A XY As input to the first discriminator D1, the camera image and A XY As input to the second discriminator D2, two probabilities are generated that reflect whether the input data comes from real data or generated data.

[0143] Through the adversarial learning described above, the generated images can retain the rich features of the original images while optimizing the generation of fused images.

[0144] The objective function for a dual-discriminator GAN can be expressed as:

[0145]

[0146] In Formula 7 above, G is the generator, and D1 and D2 are two discriminators. The parameter optimization process of the discriminator is a process of maximizing the objective function, that is, using... This means that all optimizations of the generator are a process of minimizing an objective function, that is, using... express.

[0147] Step 309: Determine whether there are obstacles around the vehicle based on the multi-adversarial learning model.

[0148] In this embodiment, feature extraction and fusion between the original millimeter-wave radar data and camera images are achieved through preprocessing. The dual-modal fusion features are efficiently extracted, and then a multi-adversarial learning model with dual discriminators is constructed based on the fusion features of the original millimeter-wave radar data, camera images, and target images. This ensures that the fused image can simultaneously retain the most important feature information of each modality image, ultimately achieving accurate identification of obstacles under complex conditions.

[0149] Furthermore, the advantage of constructing a multi-adversarial learning model lies in its ability to comprehensively utilize data from different modalities (such as millimeter-wave radar data and camera images). Through adversarial training of the generator and discriminator, it improves the accuracy and robustness of obstacle detection. This model can learn richer feature representations because it considers not only data from a single modality but also the interaction and fusion between modalities. Moreover, by setting up two discriminators, the model can separately evaluate the contributions of radar data and image data to obstacle detection, thereby better understanding the relationships between different modalities and optimizing feature fusion strategies. This multi-adversarial learning model can improve the system's adaptability to complex environments, enhance detection performance under different lighting and weather conditions, and help reduce false positives and false negatives.

[0150] Corresponding to the method provided in the above-described obstacle detection method embodiments of this application, see [link to relevant documentation]. Figure 4 This application also provides a block diagram of an obstacle detection device. In this embodiment, the device includes:

[0151] The acquisition module 401 is used to acquire raw millimeter-wave radar data and camera images of the vehicle's surroundings via millimeter-wave radar and camera, respectively.

[0152] Preprocessing module 402 is used to preprocess the raw millimeter-wave radar data to obtain a millimeter-wave radar image;

[0153] The acquisition module 403 is used to acquire the target millimeter-wave radar image features and the target camera image features corresponding to the millimeter-wave radar image and the camera image, respectively.

[0154] The fusion module 404 is used to fuse the target millimeter-wave radar image features and the target camera image features to obtain target image fusion features;

[0155] Module 405 is used to construct a multi-adversarial learning model based on the original millimeter-wave radar data, the camera images, and the fusion features of the target image;

[0156] The obstacle determination module 406 is used to determine whether there are obstacles around the vehicle based on the multi-adversarial learning model.

[0157] Optionally, the acquisition module includes:

[0158] The first acquisition submodule is used to acquire the correlation of features between the millimeter-wave radar image and the camera image;

[0159] The second acquisition submodule is used to assign weights according to the correlation to obtain the target millimeter-wave radar image features and the target camera image features.

[0160] Optionally, the first acquisition submodule includes:

[0161] The first acquisition unit is used to extract features from the millimeter-wave radar image and the camera image respectively through the same preset backbone network to obtain millimeter-wave radar feature map and camera image feature map;

[0162] The second acquisition unit is used to input the millimeter-wave radar feature map and the camera image feature map into the modal similarity generation module to obtain the correlation between the features of the millimeter-wave radar image and the camera image.

[0163] Optionally, the correlation includes a first correlation based on millimeter-wave radar images and a second correlation based on the camera images, wherein the second acquisition submodule includes:

[0164] The third acquisition unit is configured to assign a first weight to the millimeter-wave radar feature map based on the first correlation, and to assign a second weight to the camera image feature map based on the second correlation.

[0165] The fourth acquisition unit is used to obtain target millimeter-wave radar image features based on the first weight and the millimeter-wave radar feature map, and to obtain target camera image features based on the second weight and the camera image feature map.

[0166] Optionally, the building module includes:

[0167] The first construction submodule is used to take the raw millimeter-wave radar data and the camera image as input to the generator;

[0168] The second construction submodule is used to take the original millimeter-wave radar data and the target image fusion features as input to the first discriminator;

[0169] The third construction submodule is used to use the fused features of the camera image and the target image as input to the second discriminator;

[0170] The fourth construction submodule is used to construct a multi-adversarial learning model based on the generator, the first discriminator, and the second discriminator.

[0171] Optionally, the preprocessing module includes:

[0172] The preprocessing submodule is used to map the raw millimeter-wave radar data onto the image plane to obtain a millimeter-wave radar image.

[0173] Optionally, the fusion module includes:

[0174] The fusion submodule is used to input the target millimeter-wave radar image features and the target camera image features into the two-layer hybrid pooling attention module, and perform feature fusion through two-layer parallel processing to obtain the target image fusion features.

[0175] In summary, the obstacle detection device provided in this application acquires raw millimeter-wave radar data and camera images corresponding to the area around a vehicle using millimeter-wave radar and a camera, respectively; preprocesses the raw millimeter-wave radar data to obtain millimeter-wave radar images; acquires target millimeter-wave radar image features and target camera image features corresponding to the millimeter-wave radar images and camera images, respectively; fuses the target millimeter-wave radar image features and target camera image features to obtain target image fusion features; constructs a multi-adversarial learning model based on the raw millimeter-wave radar data, the camera images, and the target image fusion features; and determines whether an obstacle exists around the vehicle based on the multi-adversarial learning model. That is, in this application embodiment, feature extraction and fusion between the raw millimeter-wave radar data and camera images are achieved through preprocessing, efficient extraction of dual-modal fusion features is performed, and then a dual-discriminator multi-adversarial learning model is constructed based on the raw millimeter-wave radar data, camera images, and target image fusion features to ensure that the fused image can simultaneously retain the most important feature information of each modality image, ultimately achieving accurate obstacle identification under complex conditions.

[0176] Figure 5 This is a structural diagram of an electronic device M00 provided in an embodiment of this application. In the diagram, the electronic device M00 includes a processor M01 and a memory M02. The electronic device includes: a processor; and a memory for storing executable instructions of the processor. The processor is configured to execute the instructions to implement any of the obstacle detection methods described above and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0177] In embodiments of this application, the memory M02 can be used to store software programs and various data. The memory M02 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, applications or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory M02 may include volatile memory or non-volatile memory, or the memory x09 may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory M02 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0178] The processor M01 may include one or more processing units; optionally, the processor M01 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into the processor M01.

[0179] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the obstacle detection method embodiments described above and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0180] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0181] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the obstacle detection method embodiments described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0182] It should be understood that the chip involved in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0183] This application also provides a storage medium that, when the instructions in the storage medium are executed by the processor of an electronic device, enables the electronic device to perform any of the obstacle detection methods described above.

[0184] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the obstacle detection method embodiments described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0185] This application also provides a vehicle that includes the obstacle detection device described above.

[0186] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0187] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the related technology, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0188] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An obstacle detection method, characterized in that, The method includes: The vehicle's surroundings are captured using millimeter-wave radar and camera data, respectively. The raw millimeter-wave radar data is preprocessed to obtain a millimeter-wave radar image. Obtain the target millimeter-wave radar image features and the target camera image features corresponding to the millimeter-wave radar image and the camera image, respectively; The target millimeter-wave radar image features and the target camera image features are fused to obtain the target image fusion features; A multi-adversarial learning model is constructed based on the original millimeter-wave radar data, the camera images, and the fusion features of the target images. The multi-adversarial learning model is used to determine whether there are obstacles around the vehicle. The construction of the multi-adversarial learning model based on the original millimeter-wave radar data, the camera images, and the fused features of the target image includes: The raw millimeter-wave radar data and the camera images are used as inputs to the generator; The original millimeter-wave radar data and the target image fusion features are used as inputs to the first discriminator; The fused features of the camera image and the target image are used as input to the second discriminator; A multi-adversarial learning model is constructed based on the generator, the first discriminator, and the second discriminator.

2. The method according to claim 1, characterized in that, The step of acquiring the target millimeter-wave radar image features and target camera image features corresponding to the millimeter-wave radar image and the camera image, respectively, includes: Obtain the correlation of features between the millimeter-wave radar image and the camera image; Based on the correlation, weights are assigned to obtain the target millimeter-wave radar image features and the target camera image features.

3. The method according to claim 2, characterized in that, The correlation between the features obtained from the millimeter-wave radar image and the camera image includes: The millimeter-wave radar image and the camera image are respectively extracted using the same preset backbone network to obtain millimeter-wave radar feature map and camera image feature map; The millimeter-wave radar feature map and the camera image feature map are input into the modal similarity generation module to obtain the correlation between the features of the millimeter-wave radar image and the camera image.

4. The method according to claim 3, characterized in that, The correlation includes a first correlation based on the millimeter-wave radar image and a second correlation based on the camera image. The step of assigning weights according to the correlation to obtain the target millimeter-wave radar image features and the target camera image features includes: Assign a first weight to the millimeter-wave radar feature map based on the first correlation, and assign a second weight to the camera image feature map based on the second correlation; The target millimeter-wave radar image features are obtained based on the first weight and the millimeter-wave radar feature map, and the target camera image features are obtained based on the second weight and the camera image feature map.

5. The method according to claim 1, characterized in that, The step of preprocessing the raw millimeter-wave radar data to obtain a millimeter-wave radar image includes: The raw millimeter-wave radar data is mapped onto the image plane to obtain a millimeter-wave radar image.

6. The method according to claim 1, characterized in that, The step of fusing the target millimeter-wave radar image features and the target camera image features to obtain the target image fusion features includes: The target millimeter-wave radar image features and the target camera image features are input into a two-layer hybrid pooling attention module, and feature fusion is performed in parallel at two layers to obtain target image fusion features.

7. An obstacle detection device, characterized in that, The device includes: The acquisition module is used to acquire raw millimeter-wave radar data and camera images of the vehicle's surroundings using millimeter-wave radar and camera, respectively. The preprocessing module is used to preprocess the raw millimeter-wave radar data to obtain a millimeter-wave radar image. The acquisition module is used to acquire the target millimeter-wave radar image features and target camera image features corresponding to the millimeter-wave radar image and the camera image, respectively. The fusion module is used to fuse the target millimeter-wave radar image features and the target camera image features to obtain target image fusion features; The construction module is used to build a multi-adversarial learning model based on the original millimeter-wave radar data, the camera images, and the fusion features of the target image; An obstacle determination module is used to determine whether there are obstacles around the vehicle based on the multi-adversarial learning model. The building module includes: The first construction submodule is used to take the raw millimeter-wave radar data and the camera image as input to the generator; The second construction submodule is used to take the original millimeter-wave radar data and the target image fusion features as input to the first discriminator; The third construction submodule is used to use the fused features of the camera image and the target image as input to the second discriminator; The fourth construction submodule is used to construct a multi-adversarial learning model based on the generator, the first discriminator, and the second discriminator.

8. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the obstacle detection method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the obstacle detection method as described in any one of claims 1 to 6.

Images