Multi-modal three-dimensional target detection method and device based on diffusion knowledge distillation

Abstract

The present application relates to the technical field of target detection, and particularly relates to a multi-modal three-dimensional target detection method and device based on diffusion knowledge distillation, wherein the method comprises the following steps: acquiring point cloud data, image data and map data of a three-dimensional target, and aligning the point cloud data, the image data and the map data; performing diffusion structure distillation and explicit semantic distillation on the point cloud data and the image data by using the map data; and generating a detection result of the three-dimensional target based on the map data and the distilled point cloud data and image data. Thus, the problem of poor detection accuracy caused by geometric misplacement and semantic deviation between modalities in related technologies is solved.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a multimodal three-dimensional target detection method and apparatus based on diffusion-based knowledge distillation. Background Technology

[0002] With the rapid development of artificial intelligence, computer vision, and sensor technology, autonomous driving systems have made significant progress in recent years. Among these technologies, 3D object detection methods typically rely on heterogeneous sensors such as LiDAR and cameras for environmental perception. However, due to the different working principles and data representation methods of various sensors, the perceived data exhibits spatial misalignment and inconsistent semantic representation. This intermodal difference limits the effective fusion of multimodal data, affecting the accuracy and robustness of the perception system, and resulting in poor performance when handling complex scenes such as distant targets, occlusion, sparse point clouds, or small target detection. Summary of the Invention

[0003] This invention provides a multimodal three-dimensional target detection method and apparatus based on diffusion-based knowledge distillation, in order to solve the problem of poor detection accuracy caused by geometric misalignment and semantic offset between modalities in related technologies.

[0004] The first aspect of the present invention provides a multimodal three-dimensional target detection method based on diffusion-based knowledge distillation, comprising the following steps: acquiring point cloud data, image data, and map data of a three-dimensional target; aligning the point cloud data, image data, and map data; performing diffusion-based structural distillation and explicit semantic distillation on the point cloud data and image data using the map data; and generating a detection result of the three-dimensional target based on the map data and the distilled point cloud data and image data.

[0005] Optionally, diffusion-based structural distillation is performed on point cloud data and image data using map data, including: extracting first bird's-eye view features from map data, second bird's-eye view features from point cloud data, and third bird's-eye view features from image data; adding noise to the first bird's-eye view features through a forward diffusion process; removing noise from the noisy first bird's-eye view features through any denoising pipeline of the back diffusion process, while simultaneously denoising the second and third bird's-eye view features; and optimizing the denoised second and third bird's-eye view features using a first loss function to complete the diffusion-based structural distillation.

[0006] Optionally, explicit semantic distillation is performed on point cloud data and image data using map data, including: obtaining semantic information from the first bird's-eye view features; and using the semantic information as semantic labels to optimize the denoised second bird's-eye view features and the denoised third bird's-eye view features using a second loss function to complete the explicit semantic distillation.

[0007] Alternatively, the formula for the forward diffusion process for any t=1,…,T is:

[0008]

[0009] in, Characteristics of the diffusion time, Let t be the initial feature and t be the diffusion time step. The step size noise intensity and (0,1), To represent the standard normal distribution Noise sampled in (0,1), To represent the cumulative noise variance;

[0010] The formula for the reverse diffusion process is:

[0011]

[0012] in, This is the mean of the reverse prediction. Indicates a normal distribution. Let covariance matrix be the variance matrix. The features after denoising. To represent the features at a given current time step Next, predict the features of the previous time step. The probability distribution.

[0013] Optionally, the diffusion-structured distillation is optimized using a first loss function, which is:

[0014]

[0015] in, For the first loss function, Location in image data The characteristics of the place, Location in map data The characteristics of the location.

[0016] Optionally, explicit semantic distillation is optimized using a second loss function, which is:

[0017]

[0018] in, For the second loss function, For location in map data Semantic label probability distribution at the location, To position in image data The predicted probability distribution.

[0019] Optionally, the detection result of the 3D target is generated based on map data, distilled point cloud data, and image data, including: fusing first bird's-eye view features from the map data with second and third bird's-eye view features after distillation; and generating the detection result of the 3D target using the fused bird's-eye view features, wherein the formula for fusing the first to third bird's-eye view features is:

[0020]

[0021] in, Features of the merged bird's-eye view The first bird's-eye view feature of the map data, The second bird's-eye view features after distillation. The third bird's-eye view features are obtained after distillation, and FusionLayer is a convolutional layer used for channel alignment and feature reconstruction.

[0022] A second aspect of the present invention provides a multimodal three-dimensional target detection device based on diffusion-based knowledge distillation, comprising: an acquisition module for acquiring point cloud data, image data, and map data of a three-dimensional target, and aligning the point cloud data, image data, and map data; a processing module for performing diffusion-based structural distillation and explicit semantic distillation on the point cloud data and image data using the map data; and a detection module for generating detection results of the three-dimensional target based on the map data and the distilled point cloud data and image data.

[0023] Optionally, the processing module is further configured to extract the first bird's-eye view feature from the map data, the second bird's-eye view feature from the point cloud data, and the third bird's-eye view feature from the image data, respectively; add noise to the first bird's-eye view feature through a forward diffusion process; remove noise from the noisy first bird's-eye view feature through any one of the denoising pipes in the back diffusion process, while the denoising pipes denoise the second and third bird's-eye view features; and perform a first loss function optimization on the denoised second and third bird's-eye view features to complete the diffusion-based structural distillation.

[0024] Optionally, the processing module is further used to obtain semantic information from the first bird's-eye view features; and to use the semantic information as semantic labels to perform a second loss function optimization on the denoised second bird's-eye view features and the denoised third bird's-eye view features to complete explicit semantic distillation.

[0025] Alternatively, the formula for the forward diffusion process for any t=1,…,T is:

[0026]

[0027] in, Characteristics of the diffusion time, Let t be the initial feature and t be the diffusion time step. The step size noise intensity and (0,1), To represent the standard normal distribution Noise sampled in (0,1), To represent the cumulative noise variance;

[0028] The formula for the reverse diffusion process is:

[0029]

[0030] in, This is the mean of the reverse prediction. Indicates a normal distribution. Let covariance matrix be the variance matrix. The features after denoising. To represent the features at a given current time step Next, predict the features of the previous time step. The probability distribution.

[0031] Optionally, the diffusion-structured distillation is optimized using a first loss function, which is:

[0032]

[0033] in, For the first loss function, Location in image data The characteristics of the place, Location in map data The characteristics of the location.

[0034] Optionally, explicit semantic distillation is optimized using a second loss function, which is:

[0035]

[0036] in, For the second loss function, For location in map data Semantic label probability distribution at the location, To position in image data The predicted probability distribution.

[0037] Optionally, the detection module is further configured to: fuse the first bird's-eye view features from the map data with the distilled second and third bird's-eye view features; and generate a detection result for the three-dimensional target using the fused bird's-eye view features, wherein the formula for fusing the first to third bird's-eye view features is:

[0038]

[0039] in, Features of the merged bird's-eye view The first bird's-eye view feature of the map data, The second bird's-eye view features after distillation. The third bird's-eye view features are obtained after distillation, and FusionLayer is a convolutional layer used for channel alignment and feature reconstruction.

[0040] A third aspect of the present invention provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multimodal three-dimensional target detection method based on diffusion-based knowledge distillation as described in the above embodiments.

[0041] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the multimodal three-dimensional target detection method based on diffusion-based knowledge distillation as described in the above embodiments.

[0042] Therefore, the present invention has the following beneficial effects:

[0043] This invention, through a combination of diffuse structural distillation and explicit semantic distillation mechanisms, deeply integrates the spatial topology and semantic information of map data to achieve comprehensive perception. This significantly improves the target detection capabilities of image and point cloud data in complex scenes, fully leveraging the potential of maps in 3D target detection. Furthermore, through innovative distillation mechanisms and fusion strategies, it enhances perception capabilities in complex scenes with occlusion, sparse point clouds, and small targets. Thus, it solves the problem of poor detection accuracy caused by geometric misalignment and semantic offset between modalities in related technologies.

[0044] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0045] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0046] Figure 1 This is a flowchart of a multimodal three-dimensional target detection method based on diffusion-based knowledge distillation according to an embodiment of the present invention;

[0047] Figure 2 A schematic diagram of a diffusion structure distillation principle provided according to an embodiment of the present invention;

[0048] Figure 3This is an example diagram of a multimodal three-dimensional target detection device based on diffusion-based knowledge distillation according to an embodiment of the present invention;

[0049] Figure 4 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present invention. Detailed Implementation

[0050] Embodiments of the present invention are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0051] The following description, with reference to the accompanying drawings, describes a multimodal 3D target detection method, electronic device, vehicle, and storage medium based on diffusion-based knowledge distillation according to embodiments of the present invention. Addressing the problems mentioned in the background art, the present invention provides a multimodal 3D target detection method based on diffusion-based knowledge distillation. This method, by combining diffusion-based structural distillation and explicit semantic distillation mechanisms, deeply integrates the spatial topology and semantic information of map data to achieve comprehensive perception, significantly improving the target detection capabilities of image data and point cloud data in complex scenes, fully leveraging the potential of maps in 3D target detection, and enhancing perception capabilities in complex scenes such as occlusion, sparse point clouds, and small targets through innovative distillation mechanisms and fusion strategies. Thus, it solves the problem of poor detection accuracy caused by geometric misalignment and semantic offset between modalities in related technologies.

[0052] Specifically, Figure 1 This is a flowchart illustrating a multimodal three-dimensional target detection method based on diffusion-based knowledge distillation, provided in an embodiment of the present invention.

[0053] like Figure 1 As shown, the multimodal 3D target detection method based on diffusion-based knowledge distillation includes the following steps:

[0054] In step S101, point cloud data, image data, and map data of the three-dimensional target are acquired, and the point cloud data, image data, and map data are aligned.

[0055] It is understood that embodiments of the present invention can receive data from multiple sensors, including LiDAR data, camera image data, and high-precision map data. The high-precision map, as a structured spatial prior information, guides the fusion of multimodal sensor data and contains multiple semantic layers, such as lane lines, passable areas, sidewalks, and parking areas. These layers provide additional spatial and semantic constraints to the sensors through their unique semantic information, thereby improving the accuracy of the perception model. Each layer is represented using a binary raster, where each raster value indicates whether the location contains a certain type of map feature.

[0056] This invention embodiment can align a high-precision map into the sensor's observation space, and extract the first bird's-eye view features of the high-precision map using a map encoder. In actual implementation, to align the high-precision map with the sensor data, this invention embodiment first adjusts the sensor's observation range... Mapping to a global coordinate system and cropping map features within the corresponding range, a specially designed map encoder is then used to extract the first bird's-eye view feature representation of the map. Specifically, the map encoder consists of an unsampled path (containing dilated convolution) and a shallow feature extraction path, which work together to improve the semantic expressiveness of the map in sensor space. Simultaneously, the original semantic map is retained as a supervisory label for explicit distillation, used in subsequent semantic distillation processes.

[0057] In step S102, the point cloud data and image data are subjected to diffusion structural distillation and explicit semantic distillation using map data.

[0058] The knowledge distillation method of this invention is divided into two parts: denoising and semantic label alignment. The first step is called diffuse structural distillation, and the second step is called explicit semantic distillation. Diffuse structural distillation utilizes map data as a "structural teacher" to guide feature learning in point cloud data and image data, thereby significantly improving the perceptual modality's spatial understanding of complex scenes. Diffuse structural distillation implicitly transfers the topological structure and semantic information of map data to point cloud data and image data through diffusion modeling and anti-diffusion processes. The diffusion process includes two stages: forward diffusion (noise injection) and backward diffusion (denoising).

[0059] In one embodiment of the present invention, diffusion-based structural distillation is performed on point cloud data and image data using map data, including: extracting a first bird's-eye view feature from the map data, a second bird's-eye view feature from the point cloud data, and a third bird's-eye view feature from the image data; adding noise to the first bird's-eye view feature through a forward diffusion process; removing noise from the noisy first bird's-eye view feature through any denoising pipeline of the back diffusion process, while simultaneously denoising the second and third bird's-eye view features through a denoising pipeline to achieve knowledge transfer between the first and second / third bird's-eye view features; and optimizing the denoised second and third bird's-eye view features using a first loss function to complete the diffusion-based structural distillation.

[0060] It is understandable that, such as Figure 2 As shown, the map features are the first bird's-eye view features extracted from high-precision map data, containing rich spatial topology and semantic information. The sensor features are the second and third bird's-eye view features extracted from LiDAR point cloud data and camera image data, respectively. During the forward diffusion process, the sensor features are... Add noise to generate noisy features, and optimize them through the following process:

[0061]

[0062] in, The scheduling function is used to control the noise level. This refers to noise sampled from a standard normal distribution. This process guides the perception model to learn structural information from high-precision maps. The key objective of the diffusion process is to enable the model to progressively learn how to extract higher-quality sensor features from the knowledge conveyed by high-precision maps, particularly improving robustness in the face of occlusion, sparse data, or small targets.

[0063] The forward diffusion process includes the following steps:

[0064]

[0065] in, Characteristics of the diffusion time, Let t be the initial feature and t be the diffusion time step. The step size noise intensity and (0,1), To represent the standard normal distribution Noise sampled in (0,1), To represent the cumulative noise variance;

[0066] In this embodiment of the invention, by adding noise to the teacher model in the forward pass, the noisy map can be considered approximating the student modality. In actual implementation, the diffusion-structured distillation then learns a function. To estimate the noise added during the forward process. The network parameters are obtained by minimizing the following formula:

[0067]

[0068] in, This represents the joint expectation of the data samples, standard Gaussian noise, and time step. After training, a denoising pipeline can be used to remove the random noise through the following back-diffusion process. samples Gradual denoising is performed to generate denoised samples. During the backdiffusion process, noise is removed and a refined feature representation is generated using the following formula:

[0069]

[0070] in, This is the mean of the reverse prediction. Indicates a normal distribution. Let covariance matrix be the variance matrix. The features after denoising. To represent the features at a given current time step Next, predict the features of the previous time step. The probability distribution.

[0071] Diffusion-based structural distillation can be optimized using the following loss function:

[0072]

[0073] in, and Image data and map data in location, respectively. The feature representation at each location is used, and the loss function is used to measure the difference between the perception model and the map output. This loss function ensures semantic consistency between the sensor data and the map data by minimizing the feature difference.

[0074] In one embodiment of the present invention, explicit semantic distillation is performed on point cloud data and image data using map data, including: obtaining semantic information from the first bird's-eye view features; and using the semantic information as semantic labels to optimize the denoised second bird's-eye view features and the denoised third bird's-eye view features using a second loss function to complete the explicit semantic distillation.

[0075] This invention enhances the model's ability to perceive semantic regions of a map by introducing semantic labels to perform explicit semantic distillation on point cloud and image data. This strengthens the learning of boundaries and structures of different semantic categories (such as lanes, sidewalks, etc.), especially under challenging conditions such as occlusion and sparse point clouds. Specifically, the explicit semantic distillation mechanism explicitly supervises the denoised second and third bird's-eye view features by introducing semantic labels from the map. The map semantic label for each location contains multiple binary channels, representing different map layers (such as navigable areas, lane lines, sidewalks, etc.). These semantic labels undergo nearest-neighbor interpolation and scaling to maintain a resolution consistent with sensor features, serving as labels to supervise the semantic interpretation process of the features. This mechanism is optimized using the following explicit semantic classification loss function:

[0076]

[0077] in, The semantic category probability distribution of the map. This represents the predicted probability distribution for the perception branch. In this way, the explicit semantic distillation mechanism can further enhance the model's fine-grained understanding of geographical scenes, especially in complex scenes (such as occlusion, sparse point clouds, small targets, etc.), accurately distinguishing and perceiving different types of roads and obstacles, significantly improving the accuracy of target detection.

[0078] In some embodiments, the explicit semantic distillation mechanism further includes scaling and interpolating the semantic information of the map to adjust it to the same resolution as the sensor features, and using it as a label for training supervision. This processing ensures that the map semantic information can accurately match the sensor features, thereby improving the recognition capability in complex scenes.

[0079] In step S103, the detection results of the three-dimensional target are generated based on the map data, the distilled point cloud data, and the image data.

[0080] Specifically, after completing diffuse structural distillation and explicit semantic distillation, this embodiment of the invention can fuse the first bird's-eye view features from the map data with the distilled second and third bird's-eye view features. The fused bird's-eye view features are then used to generate 3D target detection results, further enhancing perception capabilities by constructing a unified 3D target detection representation. In actual execution, these features can be aligned and reconstructed using convolutional layers to generate optimized perception features. Finally, based on the fused bird's-eye view features, target classification, location regression, and target size estimation are performed to generate detection boxes and classification labels, completing the target detection task.

[0081] Specifically, the third-view features from image data, the second-view features from point cloud data, and the first-view features from map data are fused using a lightweight convolutional module (such as a 1×1 convolutional layer). This module not only achieves channel alignment between different modalities but also reconstructs the fused features to generate optimized perceptual features.

[0082]

[0083] in, The formula for integrating features from the first to third bird's-eye view images, The first bird's-eye view feature of the map data, This is the second bird's-eye view feature of the point cloud data. The third bird's-eye view feature of the image data is represented by FusionLayer, which is a convolutional layer used for channel alignment and feature reconstruction.

[0084] Therefore, by integrating three types of bird's-eye view features, the model's ability to understand distant targets and complex scene topology is improved. Finally, the fused features are fed into a unified target detection head for target classification, location regression, and target size estimation. This unified detection head outputs target bounding boxes and classification labels, ensuring that the multimodal perception system can accurately identify and locate various targets in 3D space and provide detailed target information.

[0085] The multimodal 3D target detection method based on diffusion-based knowledge distillation proposed in this invention combines diffusion-based structural distillation and explicit semantic distillation mechanisms to deeply integrate the spatial topology and semantic information of map data to achieve comprehensive perception. This significantly improves the target detection capabilities of image data and point cloud data in complex scenes, fully leverages the potential of maps in 3D target detection, and enhances the perception capabilities in complex scenes such as occlusion, sparse point clouds, and small targets through innovative distillation mechanisms and fusion strategies.

[0086] Next, referring to the accompanying drawings, a multimodal three-dimensional target detection device based on diffusion-based knowledge distillation according to an embodiment of the present invention is described.

[0087] Figure 3 This is a block diagram of a multimodal three-dimensional target detection device based on diffusion-based knowledge distillation, according to an embodiment of the present invention.

[0088] like Figure 3 As shown, the multimodal three-dimensional target detection device 10 based on diffusion-based knowledge distillation includes: an acquisition module 100, a processing module 200, and a detection module 300.

[0089] The acquisition module 100 is used to acquire point cloud data, image data and map data of the 3D target, and align the point cloud data, image data and map data; the processing module 200 is used to perform diffusion structural distillation and explicit semantic distillation on the point cloud data and image data using map data; the detection module 300 is used to generate detection results of the 3D target based on map data and distilled point cloud data and image data.

[0090] In one embodiment of the present invention, the processing module 200 is further configured to extract a first bird's-eye view feature from the map data, a second bird's-eye view feature from the point cloud data, and a third bird's-eye view feature from the image data; add noise to the first bird's-eye view feature through a forward diffusion process; remove noise from the noisy first bird's-eye view feature through any one of the denoising pipes of the back diffusion process, while the denoising pipes denoise the second and third bird's-eye view features; and perform a first loss function optimization on the denoised second and third bird's-eye view features to complete the diffusion-type structural distillation.

[0091] In one embodiment of the present invention, the processing module 200 is further used to obtain semantic information in the first bird's-eye view features; and to use the semantic information as semantic labels to perform a second loss function optimization on the denoised second bird's-eye view features and the denoised third bird's-eye view features to complete explicit semantic distillation.

[0092] In one embodiment of the present invention, the formula for the forward diffusion process for any t=1,…,T is:

[0093]

[0094] in, Characteristics of the diffusion time, Let t be the initial feature and t be the diffusion time step. The step size noise intensity and (0,1), To represent noise sampled from a standard normal distribution N(0,1), To represent the cumulative noise variance;

[0095] The formula for the reverse diffusion process is:

[0096]

[0097] in, This is the mean of the reverse prediction. Indicates a normal distribution. Let covariance matrix be the variance matrix. The features after denoising. To represent the features at a given current time step Next, predict the features of the previous time step. The probability distribution.

[0098] In one embodiment of the present invention, the diffusion-structured distillation is optimized using a first loss function, wherein the first loss function is:

[0099]

[0100] in, For the first loss function, Location in image data The characteristics of the place, Location in map data The characteristics of the location.

[0101] In one embodiment of the present invention, explicit semantic distillation is optimized by a second loss function, wherein the second loss function is:

[0102]

[0103] in, For the second loss function, For location in map data Semantic label probability distribution at the location, To position in image data The predicted probability distribution.

[0104] In one embodiment of the present invention, the detection module 300 is further configured to: fuse a first bird's-eye view feature from the map data with a second and a third bird's-eye view feature after distillation; and generate a detection result of a three-dimensional target using the fused bird's-eye view feature, wherein the formula for fusing the first to third bird's-eye view features is:

[0105]

[0106] in, Features of the merged bird's-eye view The first bird's-eye view feature of the map data, The second bird's-eye view features after distillation. The third bird's-eye view features are obtained after distillation, and FusionLayer is a convolutional layer used for channel alignment and feature reconstruction.

[0107] It should be noted that the foregoing explanation of the embodiment of the multimodal 3D target detection method based on diffusion-based knowledge distillation also applies to the multimodal 3D target detection device based on diffusion-based knowledge distillation in this embodiment, and will not be repeated here.

[0108] The multimodal 3D target detection device based on diffusion-based knowledge distillation proposed in this embodiment of the invention combines diffusion-based structural distillation and explicit semantic distillation mechanisms to deeply integrate the spatial topology and semantic information of map data to achieve comprehensive perception. It significantly improves the target detection capability of image data and point cloud data in complex scenes, fully leverages the potential of maps in 3D target detection, and enhances the perception capability in complex scenes such as occlusion, sparse point clouds, and small targets through innovative distillation mechanisms and fusion strategies.

[0109] Figure 4 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. The electronic device may include:

[0110] The memory 401, the processor 402, and the computer program stored on the memory 401 and capable of running on the processor 402.

[0111] When the processor 402 executes the program, it implements the multimodal three-dimensional target detection method based on diffusion-based knowledge distillation provided in the above embodiments.

[0112] Furthermore, electronic devices also include:

[0113] Communication interface 403 is used for communication between memory 401 and processor 402.

[0114] The memory 401 is used to store computer programs that can run on the processor 402.

[0115] The memory 401 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.

[0116] If the memory 401, processor 402, and communication interface 403 are implemented independently, then the communication interface 403, memory 401, and processor 402 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0117] Optionally, in a specific implementation, if the memory 401, processor 402, and communication interface 403 are integrated on a single chip, then the memory 401, processor 402, and communication interface 403 can communicate with each other through an internal interface.

[0118] Processor 402 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement embodiments of the present invention.

[0119] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described multimodal three-dimensional target detection method based on diffusion-based knowledge distillation.

[0120] This invention also provides a computer program product, including a computer program or instructions, which, when executed, implement the above-described multimodal three-dimensional target detection method based on diffusion-based knowledge distillation.

[0121] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0122] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0123] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0124] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any of the following techniques known in the art, or a combination thereof: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.

[0125] Those skilled in the art will understand that all or part of the steps of the methods implementing the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0126] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A multimodal three-dimensional target detection method based on diffusion-based knowledge distillation, characterized in that, Includes the following steps: Acquire point cloud data, image data, and map data of a 3D target, and align the point cloud data, image data, and map data; Using the map data, both the point cloud data and the image data are simultaneously subjected to diffusion structural distillation and explicit semantic distillation, wherein... The method of simultaneously performing diffusion-based structural distillation on the point cloud data and the image data using the map data includes: extracting a first bird's-eye view feature from the map data, a second bird's-eye view feature from the point cloud data, and a third bird's-eye view feature from the image data; adding noise to the first bird's-eye view feature through a forward diffusion process; removing noise from the noisy first bird's-eye view feature through any denoising pipeline of the back diffusion process, while the denoising pipeline denoises the second and third bird's-eye view features; and optimizing the denoised second and third bird's-eye view features using a first loss function to complete the diffusion-based structural distillation. The explicit semantic distillation of the point cloud data and the image data is performed simultaneously using the map data, including: obtaining semantic information from the first bird's-eye view features; and using the semantic information as semantic labels to optimize the denoised second bird's-eye view features and the denoised third bird's-eye view features using a second loss function to complete the explicit semantic distillation. The detection results of the three-dimensional target are generated based on the map data, the distilled point cloud data, and the image data.

2. The multimodal three-dimensional target detection method based on diffusion-based knowledge distillation according to claim 1, characterized in that, The formula for the forward diffusion process for any t=1,…,T is: in, Characteristics of the diffusion time, Let t be the initial feature and t be the diffusion time step. The step size noise intensity and (0,1), To represent the standard normal distribution Noise sampled in (0,1), To represent the cumulative noise variance; The formula for the reverse diffusion process is: in, This is the mean of the reverse prediction. Indicates a normal distribution. Let covariance matrix be the variance matrix. The features after denoising. To represent the features at a given current time step Next, predict the features of the previous time step. The probability distribution.

3. The multimodal three-dimensional target detection method based on diffusion-based knowledge distillation according to any one of claims 1 or 2, characterized in that, The diffusion-structured distillation is optimized using a first loss function, wherein the first loss function is: in, For the first loss function, Location in image data The characteristics of the place, Location in map data The characteristics of the location.

4. The multimodal three-dimensional target detection method based on diffusion-based knowledge distillation according to claim 1, characterized in that, The explicit semantic distillation is optimized using a second loss function, wherein the second loss function is: in, For the second loss function, For location in map data Semantic label probability distribution at location, To position in image data The predicted probability distribution.

5. The multimodal three-dimensional target detection method based on diffusion-based knowledge distillation according to claim 4, characterized in that, Based on the map data, distilled point cloud data, and image data, the detection results of the three-dimensional target are generated, including: The first bird's-eye view features in the map data are fused with the second and third bird's-eye view features after distillation. The detection result of the three-dimensional target is generated using the fused bird's-eye view features, wherein the formula for fusing the features of the first to third bird's-eye views is: in, Features of the merged bird's-eye view The first bird's-eye view feature of map data, The second bird's-eye view features after distillation. The third bird's-eye view features are obtained after distillation, and FusionLayer is a convolutional layer used for channel alignment and feature reconstruction.

6. A multimodal three-dimensional target detection device based on diffusion-based knowledge distillation, characterized in that, include: The acquisition module is used to acquire point cloud data, image data, and map data of a 3D target, and align the point cloud data, image data, and map data. The distillation module is used to simultaneously perform diffusion-based structural distillation and explicit semantic distillation on the point cloud data and the image data using the map data, wherein... The method of simultaneously performing diffusion-based structural distillation on the point cloud data and the image data using the map data includes: extracting a first bird's-eye view feature from the map data, a second bird's-eye view feature from the point cloud data, and a third bird's-eye view feature from the image data; adding noise to the first bird's-eye view feature through a forward diffusion process; removing noise from the noisy first bird's-eye view feature through any denoising pipeline of the back diffusion process, while the denoising pipeline denoises the second and third bird's-eye view features; and optimizing the denoised second and third bird's-eye view features using a first loss function to complete the diffusion-based structural distillation. The explicit semantic distillation of the point cloud data and the image data is performed simultaneously using the map data, including: obtaining semantic information from the first bird's-eye view features; and using the semantic information as semantic labels to optimize the denoised second bird's-eye view features and the denoised third bird's-eye view features using a second loss function to complete the explicit semantic distillation. The detection module is used to generate the detection result of the three-dimensional target based on the map data, the distilled point cloud data, and the image data.

7. An electronic device, characterized in that, include: The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multimodal three-dimensional target detection method based on diffusion-based knowledge distillation as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed, they implement the multimodal three-dimensional target detection method based on diffusion-based knowledge distillation as described in any one of claims 1-5.

Images