Model training method, target detection method, electronic equipment and storage medium

Abstract

The invention discloses a model training method and device, a target detection method and device, electronic equipment and a storage medium. The training method comprises the following steps: inputting a sample image and sample point cloud data collected for a sample region into a teacher model for target detection to obtain first feature information and a first detection result; inputting the sample image into a student model for target detection to obtain second feature information and a second detection result; determining target loss according to at least two of the first distillation loss, the second distillation loss and the third distillation loss; and at least adjusting parameters of the student model according to the target loss until a training ending condition is met, and taking the student detection model meeting the training ending condition as a target detection model. According to the method provided by the invention, the accuracy of target detection is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and more particularly, to a model training method, a target detection method, an apparatus, an electronic device, and a storage medium. BACKGROUND

[0002] Three-dimensional target detection is an important application of computer vision, which is used to identify the pose information of a target in a three-dimensional space, the category to which the target belongs, and the size of the target, etc. from two-dimensional images and point cloud data, or from two-dimensional images. However, there is a problem of low accuracy of three-dimensional target detection in the related art. SUMMARY

[0003] Therefore, the embodiments of the present application provide a model training method, a target detection method, an apparatus, an electronic device, and a storage medium.

[0004] In a first aspect, the embodiments of the present application provide a model training method, which comprises: inputting sample images and sample point cloud data collected for a sample region into a teacher model for target detection to obtain first feature information and a first detection result, the first feature information comprising first image bird's eye view features, point cloud bird's eye view features, and fusion bird's eye view features obtained by fusing and encoding the first image bird's eye view features and the point cloud bird's eye view features; inputting the sample images into a student model for target detection to obtain second feature information and a second detection result; the second feature information comprising second image bird's eye view features and refined bird's eye view features obtained by encoding the second image bird's eye view features; determining a target loss according to at least two of a first distillation loss, a second distillation loss, and a third distillation loss; the first distillation loss reflecting differences between the second image bird's eye view features and the first image bird's eye view features and the point cloud bird's eye view features respectively; the second distillation loss reflecting a difference between the fusion bird's eye view features and the refined bird's eye view features; the third distillation loss reflecting a difference between the first detection result and the second detection result; adjusting parameters of the student model according to the target loss until a training end condition is reached, and taking the student detection model after the training end condition is reached as a target detection model.

[0005] In a second aspect, the embodiments of the present application provide a target detection method, which comprises: obtaining a to-be-detected image; performing three-dimensional target detection on the to-be-detected image by using a target detection model to obtain a target detection result; and the target detection model is trained according to the method of the first aspect.

[0006] Thirdly, embodiments of this application provide a model training apparatus, comprising: a first detection module, configured to input sample images and sample point cloud data collected from a sample region into a teacher model for target detection, obtaining first feature information and a first detection result; the first feature information includes a first image bird's-eye view feature, a point cloud bird's-eye view feature, and a fused bird's-eye view feature obtained by fusing and encoding the first image bird's-eye view feature and the point cloud bird's-eye view feature; and a second detection module, configured to input sample images into a student model for target detection, obtaining second feature information and a second detection result; the second feature information includes a second image bird's-eye view feature and a fused bird's-eye view feature obtained by fusing and encoding the second image bird's-eye view feature and the point cloud bird's-eye view feature. The system includes a refined bird's-eye view feature obtained by encoding the features; a loss determination module for determining the target loss based on at least two of the first distillation loss, the second distillation loss, and the third distillation loss; the first distillation loss reflects the difference between the second image bird's-eye view feature and the first image bird's-eye view feature and the point cloud bird's-eye view feature, respectively; the second distillation loss reflects the difference between the fused bird's-eye view feature and the refined bird's-eye view feature; and the third distillation loss reflects the difference between the first detection result and the second detection result; and an adjustment module for adjusting the parameters of the student model based on the target loss until the training termination condition is met, and using the student detection model after the training termination condition is met as the target detection model.

[0007] Optionally, the first distillation loss includes a first loss and a second loss; the loss determination module is further configured to determine the first loss based on the bird's-eye view features of the second image and the bird's-eye view features of the first image; and to determine the second loss based on the bird's-eye view features of the second image and the bird's-eye view features of the point cloud.

[0008] Optionally, the loss determination module is further configured to acquire target attribute annotation information corresponding to the sample image and sample point cloud data, the target attribute annotation information including the annotation location information of the sample target; generate a region mask in the bird's-eye view based on the annotation location information of each sample target; the region mask is used to indicate the position of the center point of each sample target in the bird's-eye view; perform masking processing on the bird's-eye view features of the sample point cloud and the bird's-eye view features of the second image based on the region mask, to obtain the bird's-eye view features of the masked sample point cloud and the bird's-eye view features of the masked image; and determine the second loss based on the bird's-eye view features of the masked sample point cloud and the bird's-eye view features of the masked image.

[0009] Optionally, the first detection result includes first target attribute information, which includes first semantic recognition information and the first pose information of the sample target indicated by the first semantic recognition information; the second detection result includes second target attribute information; the second target attribute information includes second semantic recognition information and the second pose information of the sample target indicated by the second semantic recognition information; the loss determination module is further configured to determine a third distillation loss based on the difference between the first semantic recognition information and the second semantic recognition information, and the difference between the first pose information of the sample target indicated by the first semantic recognition information and the second pose information of the target indicated by the first semantic recognition information.

[0010] Optionally, the loss determination module is also used to obtain the target supervision loss, which is determined based on the differences between the target attribute annotation information corresponding to the sample image and the first detection result and the second detection result, respectively; at least two of the first distillation loss, the second distillation loss and the third distillation loss are weighted and calculated with the target supervision loss to obtain the target loss.

[0011] Optionally, the loss determination module is also used to perform weighted calculation of the first distillation loss, the second distillation loss, the third distillation loss, and the target supervision loss to obtain the target loss.

[0012] Optionally, the first detection module is further configured to encode the sample point cloud data by the point cloud feature encoder in the teacher model to obtain sample point cloud features; transform the sample point cloud features to a bird's-eye view to obtain point cloud bird's-eye view features; encode the sample image by the first image feature encoder in the teacher model to obtain first image features; transform the first image features to a bird's-eye view by the first view transformation network in the teacher model to obtain first image bird's-eye view features; fuse the point cloud bird's-eye view features and the first image bird's-eye view features by the first bird's-eye view feature encoder in the teacher model to obtain fused bird's-eye view features; and perform three-dimensional target prediction based on the fused bird's-eye view features by the first detection head network in the teacher model to obtain a first detection result.

[0013] Optionally, the first detection head network includes a first semantic branch, a first centrality branch, a first depth branch, a first detection box size branch, and a first heading angle branch; the first detection result includes a first semantic classification result, a first centrality prediction result, a first center depth prediction result, a first size prediction result, and a first heading angle prediction result; the first detection module is further configured to perform semantic classification based on fused bird's-eye view features by the first semantic branch to obtain a first semantic classification result; perform center point prediction based on fused bird's-eye view features by the first centrality branch to obtain a first centrality prediction result; perform center point depth prediction based on fused bird's-eye view features by the first depth branch to obtain a first center depth prediction result; perform size prediction based on fused bird's-eye view features by the first detection box size branch to obtain a first size prediction result; and perform heading angle prediction based on fused bird's-eye view features by the first heading angle branch to obtain a first heading angle prediction result.

[0014] Optionally, the first detection module is further configured to have the first view transformation network process the sample image features according to the following process: determining image view pseudo-point cloud features based on pixel features in the first image features; transforming the image view pseudo-point cloud features to the device coordinate system of the reference device according to the camera intrinsic and extrinsic parameters of the camera from which the sample image originates, to obtain intermediate pseudo-point cloud features, wherein the camera from which the sample image originates is located on the reference device; dividing the intermediate pseudo-point cloud features into multiple voxel grids and aggregating the pseudo-point cloud features located in the same voxel grid to obtain four-dimensional pseudo-voxel features; and reducing the dimensionality of the four-dimensional pseudo-voxel features to obtain the first image bird's-eye view features.

[0015] Optionally, the first detection module is further configured to predict the probability that each pixel in the sample image is dispersed to multiple discrete depths based on the pixel features in the sample image features; determine the discrete depth with the highest probability as the target depth of the pixel based on the probability that each pixel in the sample image is dispersed to multiple discrete depths; and determine the pseudo point cloud features of the image view based on the target depth of each pixel in the sample image and the pixel features of each pixel in the sample image features.

[0016] Optionally, the second detection module is further configured to: extract features from the sample image using the second image feature encoder in the student model to obtain second image features; transform the second image features to a bird's-eye view using the second view transformation network in the student model to obtain second image bird's-eye view features; encode the second image bird's-eye view features using the second bird's-eye view feature encoder in the student model to obtain refined bird's-eye view features; and perform three-dimensional target prediction based on the refined bird's-eye view features using the second detection head network in the student model to obtain the second detection result.

[0017] Fourthly, embodiments of this application provide a target detection device, which includes: an acquisition module for acquiring an image to be detected; and a target detection module for performing three-dimensional target detection on the image to be detected using a target detection model to obtain a target detection result; the target detection model is trained according to the method described in the first aspect above.

[0018] Fifthly, embodiments of this application provide an electronic device, including: a processor and a memory; the memory stores computer-readable instructions, which, when executed by the processor, implement the above-described method.

[0019] Sixthly, embodiments of this application provide a computer-readable storage medium storing computer-readable instructions that, when executed by a processor, implement the above-described method.

[0020] In a seventh aspect, embodiments of this application provide a computer program product including computer-readable instructions that, when executed by a processor, implement the method described above.

[0021] In this application, the teacher model first determines the first feature information and the first detection result based on sample images and sample point cloud data, and the student model determines the second feature information and the second detection result based on sample images. Then, based on the first feature information, the first detection result, the second feature information, and the second detection result, the first distillation loss, the second distillation loss, and the third distillation loss are determined. Then, the target loss is determined by at least one of the first distillation loss, the second distillation loss, and the third distillation loss. The parameters of the student model are adjusted by the target loss so that the teacher model is not limited by modality. At the same time, the complementary knowledge of two modal data, namely point cloud data obtained by radar and image data obtained by camera, is integrated and transferred to the student model to improve the detection capability of the student model. Furthermore, the first distillation loss is determined based on shallow-level first image bird's-eye view features, second image bird's-eye view features, and point cloud bird's-eye view features. Therefore, when training the student model based on the first distillation loss, the student model, in addition to learning visual knowledge, can also learn the 3D features of the sample point cloud data, effectively reducing feature differences between different modalities, thus achieving coarse feature distillation across modalities. The second distillation loss is determined based on deep-level fused bird's-eye view features and refined bird's-eye view features. Therefore, when training the student model based on the second distillation loss, the student model can learn the multimodal feature extraction capabilities of the teacher model, thus achieving refined feature distillation across modalities. The third distillation loss is based on a more refined second prediction result. And the first prediction result is determined. Thus, when training the student model based on the third distillation loss, the gap between the prediction response of the teacher model and the student model is narrowed, and more refined prediction distillation is achieved. Therefore, at least two of the first distillation loss, the second distillation loss and the third distillation loss are combined to determine the target loss. The target loss involves distillation loss with different degrees of refinement. This enables multi-level distillation supervision from coarse to fine when training the student model based on the target loss. It guides the student model to learn better 3D detection feature representations and improves the detection capability of the target detection model after training based on 2D images for 3D target detection, thereby improving the accuracy of the target detection model for 3D target detection. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A schematic diagram illustrating an application scenario applicable to embodiments of this application is shown;

[0024] Figure 2A flowchart of a model training method according to an embodiment of this application is shown;

[0025] Figure 3 It shows Figure 2 A flowchart of step S110 in one embodiment is shown in the corresponding example;

[0026] Figure 4 It shows Figure 3 A flowchart of step S240 in one embodiment is shown in the corresponding example;

[0027] Figure 5 A schematic diagram of a first bird's-eye view feature encoder in one embodiment of this application is shown;

[0028] Figure 6 It shows Figure 2 A flowchart of step S120 in one embodiment is shown in the corresponding example;

[0029] Figure 7 A schematic diagram of the model training process in one embodiment of this application is shown;

[0030] Figure 8 A flowchart of a target detection method according to an embodiment of this application is shown;

[0031] Figure 9 A block diagram of a model training apparatus according to an embodiment of this application is shown;

[0032] Figure 10 A block diagram of a target detection device according to an embodiment of this application is shown;

[0033] Figure 11 A structural block diagram of an electronic device for performing a model training method according to an embodiment of this application is shown. Detailed Implementation

[0034] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0035] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" can be interchanged in a specific order or sequence where permissible, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein. In the following description, references to "some embodiments or some ways of embodiments" describe a subset of all possible embodiments; however, it is understood that "some embodiments" can be the same subset or different subsets of all possible embodiments and can be combined with each other without conflict.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit the application. It should be noted that "multiple" as used herein refers to two or more. "And / or" describes the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0037] The abbreviations or custom terms used in the embodiments of this application are explained below:

[0038] Multimodal (or multi-source): In this application, it specifically refers to different modal data acquired from two different types of sensors: cameras and lidar. For example, a camera captures image data in a two-dimensional perspective view, while a lidar captures point cloud data in a three-dimensional view.

[0039] Knowledge distillation is an emerging model compression method that employs a teacher-student training structure. Typically, a pre-trained teacher model provides knowledge, and the student model acquires this knowledge through distillation training, transferring knowledge from the complex teacher model to the simpler student model.

[0040] Bird's-eye view (BEV) is an elevated view of an object or location from a very steep angle, creating the perspective as if the observer were a bird in flight looking down.

[0041] 3D target detection refers to predicting the 3D attributes of a target based on acquired 2D images and point cloud data, or one of them. The 3D attributes include semantic category, size, and the target's pose information in the 3D coordinate system (such as position, heading angle, etc.).

[0042] See Figure 1 , Figure 1A schematic diagram of an application scenario applicable to an embodiment of this application is shown. The application scenario includes a database 500, a terminal device 400, and a server 200. The terminal device 400 is connected to the server 200 through a network 300, wherein the network 300 can be a wide area network, a local area network, or a combination of both.

[0043] In some embodiments, the model training method and object detection method provided in this application can be implemented by a server. For example, server 200 obtains training data from database 500. The training data includes sample images and sample point cloud data collected for sample regions. Subsequently, server 200 processes the sample images and sample point cloud data based on the teacher model to obtain first feature information and first detection result. The student model processes the sample images to obtain second feature information and second detection result. Then, server 200 determines a first distillation loss, a second distillation loss, and a third distillation loss based on the aforementioned first feature information, first detection result, second feature information, and second detection result. Server 200 determines a target loss based on at least two of the first distillation loss, second distillation loss, and third distillation loss, and adjusts the parameters of the student model at least by means of the target loss until the training termination condition is met. The student detection model after the training termination condition is met is used as the object detection model.

[0044] Server 200 can receive the image to be detected sent by terminal device 400, and perform three-dimensional object detection on the image to be detected using the object detection model trained above, obtain the object detection result, and then return the object detection result to terminal device 400 so that terminal device 400 can output the object detection result.

[0045] In some other embodiments, the model training method and object detection method provided in this application can be implemented by a terminal device.

[0046] In some other embodiments, the model training method provided in this application can be implemented by a server, and the object detection method can be implemented by a terminal device.

[0047] In some embodiments, the terminal device 400 or server 200 can implement the model training method and object detection method provided in this application embodiment by running a computer program. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), i.e., a program that needs to be installed in the operating system to run; it can also be a small program, i.e., a program that only needs to be downloaded to a browser environment to run; or it can be a small program that can be embedded in any APP, and the small program can be controlled by the user to run or close. In short, the above-mentioned computer program can be any form of application, module or plugin.

[0048] In some embodiments, server 200 may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0049] The terminal device 400 can be a smartphone, tablet computer, laptop computer, desktop computer, intelligent voice interaction device, smart home appliance, vehicle terminal, smart TV, intelligent service terminal, etc., but is not limited to these. The terminal device and the server can be directly or indirectly connected through wired or wireless communication methods, which is not limited in this embodiment.

[0050] For ease of description, electronic devices are used as the subject of execution of the method in this application.

[0051] Please see Figure 2 , Figure 2 This application illustrates a flowchart of a model training method according to an embodiment of the present application. The method can be executed by an electronic device, such as... Figure 1 In other embodiments, the server 200 or terminal device 400 shown may also implement the method of this application through interaction between the server 200 and the terminal device 400. This method may include S110-S140, which are described in detail below:

[0052] S110. Input the sample images and sample point cloud data collected for the sample region into the teacher model for target detection to obtain the first feature information and the first detection result.

[0053] The first feature information includes the first image bird's-eye view feature, the point cloud bird's-eye view feature, and the fused bird's-eye view feature obtained by fusing and encoding the first image bird's-eye view feature and the point cloud bird's-eye view feature.

[0054] A sample region can refer to any area in the actual physical environment. A sample image is an image obtained by acquiring images of a sample region, while sample point cloud data is point cloud data obtained by acquiring point cloud data of a sample region; the point cloud data can be acquired using radar (such as lidar).

[0055] In one possible implementation, the sample images and sample point cloud data acquired for the sample region can be obtained by a camera and radar installed on the same reference device. This reference device can be a vehicle, robot, industrial equipment, or medical equipment, etc. Since the sample point cloud data and sample images are acquired for the sample region, the sample images present the sample targets within the sample region, and the point cloud data also presents the sample targets within that sample region; that is, the sample point cloud data includes three-dimensional point cloud data of the sample targets.

[0056] In this application, there can be multiple sample images for the same sample area: images of the same sample area are acquired from different perspectives to obtain sample images of the same sample area from different perspectives. For example, images can be obtained by multiple cameras installed on the vehicle to acquire images of the vehicle's external environment. The different cameras have different perspectives so that the acquired multiple sample images can accurately indicate the external environment of the vehicle.

[0057] In some implementation examples, the teacher model may include a first image bird's-eye view feature encoder, a point cloud bird's-eye view feature encoder, a first bird's-eye view feature encoder, and a first detection head network.

[0058] The first image bird's-eye view feature encoder is used to encode the features of the sample image to obtain the first image bird's-eye view features from a bird's-eye view perspective. That is, the input of the first image bird's-eye view feature encoder is the image, and the output is the image bird's-eye view features from a bird's-eye view perspective. This first image bird's-eye view feature encoder can be constructed using convolutional neural networks, pooling neural networks, etc.

[0059] A point cloud bird's-eye view feature encoder is used to extract point cloud bird's-eye view features from sample point cloud data. The input of the point cloud bird's-eye view feature encoder is point cloud data, and the output is the point cloud bird's-eye view features. The point cloud bird's-eye view feature encoder can be constructed from one or more neural networks, such as convolutional neural networks, fully connected neural networks, pooling neural networks, recurrent neural networks, etc.

[0060] The first bird's-eye view feature encoder is used to fuse and encode the features output by the first image bird's-eye view feature encoder (e.g., the first image bird's-eye view features) and the features output by the point cloud bird's-eye view feature encoder (e.g., the point cloud bird's-eye view features) to obtain fused bird's-eye view features from a bird's-eye view perspective, such as the fused bird's-eye view features mentioned above. This first bird's-eye view feature encoder can be constructed from one or more neural networks, such as convolutional neural networks, fully connected neural networks, pooling neural networks, recurrent neural networks, and convolutional neural networks with residual designs.

[0061] The first detection head network is used to fuse bird's-eye view features to predict three-dimensional attributes in order to obtain detection results (e.g., the first detection result). The detection result includes the three-dimensional attribute information of each predicted target, such as semantic category, pose information in the three-dimensional coordinate system, size, etc.

[0062] The three-dimensional coordinate system corresponding to the sample point cloud data is the device coordinate system of the LiDAR from which the sample point cloud data originates. For ease of description, the device of the LiDAR from which the sample point cloud data originates is referred to as the reference device. Assume that the three coordinate axes in the reference device's device coordinate system are the X0 axis, Y0 axis, and Z0 axis. Based on this, the coordinates of each point in the sample point cloud data are the coordinates in the device coordinate system of this reference device.

[0063] In some embodiments, to improve data processing efficiency and reduce the interference of invalid data on subsequent processing, a three-dimensional sensing range can be preset. This three-dimensional sensing range can be a range of three-dimensional coordinates specified in the device coordinate system of the reference device. Then, the sample point cloud data is filtered according to the three-dimensional sensing range to filter out points whose coordinates are not within the three-dimensional sensing range. Subsequently, the points in the sample point cloud data whose coordinates are within the three-dimensional sensing range are encoded with point cloud features.

[0064] The three-dimensional perception range can be specified according to actual needs. For example, if the sample area is the external environment around the vehicle, the three-dimensional perception range can be defined as X0:[-54.0,54.0], Y0:[-54.0,54.0], Z0:[-5.0,3.0].

[0065] The first detection head network refers to the detection head network deployed in the teacher model to predict the three-dimensional attribute information of the output target. The three-dimensional attribute information of the target includes the semantic category of the sample target, the pose information (i.e., position and orientation) of the sample target in three-dimensional space (three-dimensional coordinate system), and the size of the target. The input of the first detection head network is the fused bird's-eye view features output by the first bird's-eye view feature encoder, and the output is the three-dimensional detection result, which is the identified three-dimensional attribute information of the sample target. This three-dimensional attribute information serves as the first detection result.

[0066] In some implementations, such as Figure 3 As shown, S110 may include S210-S260, as follows:

[0067] S210. The point cloud feature encoder in the teacher model encodes the sample point cloud data to obtain the sample point cloud features.

[0068] S220. Transform the sample point cloud features to a bird's-eye view to obtain the point cloud bird's-eye view features.

[0069] A point cloud feature encoder is a neural network used to encode features from sample point cloud data to obtain the features of the sample point cloud. This point cloud feature encoder can be constructed from one or more neural networks, such as convolutional neural networks, fully connected neural networks, pooling neural networks, and recurrent neural networks.

[0070] The point cloud feature encoder can be the backbone network used to encode the sample point cloud data from PointPillars (a deep learning model for 3D object detection based on sample point cloud data), CenterPoint (a model for 3D object detection), or VoxelNet (a deep learning model for 3D object detection based on sample point cloud data).

[0071] In some embodiments, the point cloud feature encoder can voxelize the point cloud in the sample point cloud data according to a predefined voxel separation gap (e.g., a voxel separation gap of 0.075m on the X0 axis, a voxel separation gap of 0.075m on the Y0 axis, and a voxel separation gap of 0.2m on the Z0 axis).

[0072] The point cloud feature encoder extracts the features of points within each voxel grid, and then aggregates the features of points within each voxel grid using the same encoder to obtain the sample point cloud features. The sample point cloud features include the features of each voxel grid. Then, the sample point cloud features are subjected to elevation compression (i.e., compression along the Z0 axis) and flattened along the Z0 axis. Finally, a point cloud feature dimensionality reduction network composed of sparse 3D convolutions is used to effectively generate the point cloud features in the BEV space. (where X and Y represent the width and height of the feature, respectively, C) Lidar (representing feature dimension), the resulting features in the BEV (Bird's-Eye View) space. This refers to the bird's-eye view characteristics of point clouds.

[0073] S230. The first image feature encoder in the teacher model encodes the sample image to obtain the first image feature.

[0074] S240. The first image features are transformed into a bird's-eye view by the first view transformation network in the teacher model to obtain the first image bird's-eye view features.

[0075] In this embodiment, the first image bird's-eye view feature encoder may include a first image feature encoder and a first view transformation network cascaded in sequence.

[0076] The first image feature encoder refers to a neural network used to encode features of a two-dimensional sample image to obtain two-dimensional image features; the first image feature encoder can be constructed by convolutional neural networks, pooling neural networks, etc.

[0077] In some embodiments, the first image feature encoder may include an image backbone network and a feature fusion network. The image backbone network is used to extract shallow features of the image, and the feature fusion network is used to fuse the shallow features extracted by the image backbone network. The image backbone network may be a visual geometric group network, a U-shaped network with residual structure, or a window splitter network, and the feature fusion network may be a feature pyramid network.

[0078] In some embodiments, the first image feature encoder can be a network obtained by combining a window splitter network and a feature pyramid network. That is, the window splitter network can first extract features from the sample image, and then the feature pyramid network can fuse features at multiple scales to effectively utilize shallow features with more resolution information and deep features with more semantic information. The window splitter network can reduce the computational cost of image feature encoding through window attention and transferred window attention, while also supporting flexible processing of feature information at different scales.

[0079] The size of the feature output by the first image feature encoder can be the same as the size of the image input to the first image feature encoder. For example, the size of the feature output by the first image feature encoder can be 1 / 8 of the size of the input sample image.

[0080] When there are multiple sample images input to the first image feature encoder, the first image feature encoder outputs the first image features. (in, N represents the sample image features of the i-th sample image. cams This refers to the number of sample images (for example, if the identified sample area is obstacles around a vehicle, this number is the number of cameras installed on the vehicle to collect images of the surrounding environment; the default is 6 cameras for surround view). H and W represent the height and width of the sample image features, respectively, and C represents the channel dimension, which defaults to 256 channels.

[0081] Then, the first view transformation network uses the first image features corresponding to the sample regions. Switch to a bird's-eye view to obtain image features from that perspective.

[0082] In some embodiments, such as Figure 4As shown, the first view transformation network processes the sample image features according to the following process S310-S340:

[0083] S310. Based on the pixel features in the first image features, determine the pseudo point cloud features of the image view.

[0084] The first image feature includes the pixel features of each pixel in the sample image.

[0085] In some embodiments, the depth value of each pixel in the sample image can be estimated using a depth estimation network based on the pixel features of each pixel in the first image features. Then, using the camera intrinsic parameters of the camera that acquired the sample image and the depth value of each pixel, the pixels in the sample image can be mapped into a three-dimensional space to obtain a three-dimensional pseudo-point cloud, thereby obtaining the pseudo-point cloud features of the image view. The depth estimation network can be a neural network used to predict the depth value of pixels based on the pixel features of the image pixels.

[0086] In some embodiments, step 310 includes: predicting the probability that each pixel in the sample image is distributed across multiple discrete depths based on pixel features in the sample image features; determining the discrete depth with the highest probability as the target depth of the pixel based on the probability that each pixel in the sample image is distributed across multiple discrete depths; and determining the pseudo point cloud features of the image view based on the target depth of each pixel in the sample image and the pixel features of each pixel in the sample image features.

[0087] Specifically, the depth estimation network in the first view transformation network can be used to predict the probability of each pixel corresponding to a discrete depth in multiple discrete depths based on the pixel features of that pixel. The number of discrete depths is D, which can be a value greater than 1. For example, if the depth values ​​of the multiple discrete depths range from 1 meter to 60 meters, and the depth interval is 1 meter (or 0.5 meters), the determined D discrete depths can be 59 or 118.

[0088] By using a depth estimation network, the probability of each pixel in the sample image at each discrete depth can be obtained. It should be understood that the sum of the probabilities of each pixel scattered at D discrete depths is 1. That is, for a pixel i, the probability of the pixel at the j-th discrete depth is Pij, then Pi1+Pi2+...+Pij...PiD=1. After obtaining the probability of each pixel at D discrete depths, the discrete depth with the highest probability is determined as the target depth of the pixel.

[0089] Then, the first image features and the target depth of each pixel are combined to obtain the image view pseudo point cloud features, that is, the image view pseudo point cloud features include the pixel features of each pixel in the first image features and the target depth of each pixel.

[0090] S320. Based on the camera intrinsic and extrinsic parameters of the camera from which the sample image originates, transform the pseudo-point cloud features of the image view to the device coordinate system of the reference device to obtain intermediate pseudo-point cloud features.

[0091] In this context, the camera from which the sample image originates is positioned on a reference device. The process described above is equivalent to performing a coordinate transformation using the camera's intrinsic and extrinsic parameters to transform the image view pseudo-point cloud features to the reference device's coordinate system. The result of this transformation is called the intermediate pseudo-point cloud features. It's understandable that both the image view pseudo-point cloud features and the intermediate pseudo-point cloud features are three-dimensional features.

[0092] S330. Divide the intermediate pseudo-point cloud features into multiple voxel grids and aggregate the pseudo-point cloud features located in the same voxel grid to obtain four-dimensional pseudo-voxel features.

[0093] Multiple voxel grids can be pre-divided, and the positions of multiple voxel grids are known. Each point in the intermediate pseudo-point cloud feature is located in the voxel grid. Then, the pseudo-point cloud features in the same voxel grid are aggregated to obtain a four-dimensional pseudo-voxel feature. The four-dimensional pseudo-voxel feature involves four dimensions, including three coordinate dimensions (the spatial dimensions corresponding to the X0, Y0, and Z0 axes of the device coordinate system) and a channel dimension of the feature. The channel dimension of the feature is used to represent the feature vector in the voxel grid.

[0094] In some embodiments, CUDA (Compute Unified Device Architecture) multi-threaded accelerated efficient BEV (Bird's Eye View) pooling operations can be used to aggregate pseudo-point cloud features in the same voxel mesh to accelerate the acquisition of four-dimensional pseudo-voxel features.

[0095] S340. Dimensionality reduction is performed on the four-dimensional pseudo-voxel features to obtain the first image bird's-eye view features.

[0096] In one possible implementation, when reducing the dimensionality of the four-dimensional pseudo-voxel features, the features in each voxel grid can be projected onto the plane of the bird's-eye view to obtain the first image bird's-eye view features. At this time, the spatial dimensions corresponding to the X0 axis, Y0 axis and Z0 axis are reduced to the spatial dimensions corresponding to the X0 axis and Y0 axis respectively.

[0097] In one possible implementation, step S340 may be: reducing the dimensionality of the four-dimensional pseudo-voxel features in the channel dimension to obtain the first image bird's-eye view features.

[0098] In this approach, max pooling, average pooling, and self-attention mechanisms can be used to reduce dimensionality in the channel dimension, thereby obtaining the first image's bird's-eye view features.

[0099] For example, a four-dimensional pseudo-voxel feature V can be represented as V∈R XxYxZxC C represents the channel dimension. In obtaining the four-dimensional pseudo-voxel features V∈R... XxYxZxC Then, the four-dimensional pseudo-voxel features V∈R XxYxZxC Reconstructed into a three-dimensional tensor V'∈R XxYx (ZC) The channel dimension of this 3D tensor is ZC. Subsequently, four 3x3 convolutional layers can be used to reduce the channel dimension of the 3D tensor, for example, from ZC to C. camera The final first image bird's-eye view features are obtained. The first image's bird's-eye view features simultaneously preserve both semantic and spatial information.

[0100] By employing the steps S310-S340 described above, the two-dimensional first image features are transformed into the BEV space to obtain the first image bird's-eye view features from a bird's-eye view perspective.

[0101] S250. The first bird's-eye view feature encoder in the teacher model performs fusion encoding on the point cloud bird's-eye view features and the first image bird's-eye view features to obtain fused bird's-eye view features.

[0102] The first bird's-eye view feature encoder can perform operations such as stitching, matrix summation, and matrix multiplication on the point cloud bird's-eye view features and the first image bird's-eye view features to achieve the fusion of the point cloud bird's-eye view features and the first image bird's-eye view features, and obtain the fused bird's-eye view features.

[0103] In one possible implementation, step S250 may include: stitching together the first image bird's-eye view features and the point cloud bird's-eye view features along the channel direction to obtain stitched bird's-eye view features; performing convolution processing on the stitched bird's-eye view features by the first convolutional network in the first bird's-eye view feature encoder to obtain a first intermediate feature; performing convolution processing on the first intermediate feature by the second convolutional network in the first bird's-eye view feature encoder to obtain a second intermediate feature; fusing the first intermediate feature and the second intermediate feature to obtain a third intermediate feature; and performing convolution processing on the third intermediate feature by the third convolutional network in the first bird's-eye view feature encoder to obtain a fused bird's-eye view feature.

[0104] In some embodiments, such as Figure 5As shown, the first convolutional network includes a 3x3 convolutional layer with the ReLU activation function, the second convolutional network includes two 3x3 convolutional layers with the ReLU activation function, and the third convolutional network may include three 3x3 convolutional layers with the ReLU activation function. The outputs of the first and second convolutional networks can be connected via residual jumps, as shown below. Figure 5 As shown, convolutional layers are denoted by Conv. In other embodiments, the size of the convolutional kernels deployed in the first, second, and third convolutional networks may not be limited to... Figure 5 As shown, other sizes of convolution kernels can be used, which can be selected according to actual needs. In related technologies, during the conversion of two-dimensional image features to a bird's-eye view, the estimated depth distribution may not be accurate enough, and there is a certain degree of local spatial misalignment between the converted image bird's-eye view features and the point cloud bird's-eye view features converted from three-dimensional point cloud features to a bird's-eye view.

[0105] In the above embodiments, by using a first bird's-eye view feature encoder with residual design to fuse and encode the point cloud bird's-eye view features and the first image bird's-eye view features, this local spatial misalignment can be compensated, and the accurate fusion of the point cloud bird's-eye view features and the first image bird's-eye view features can be achieved, thereby improving the accuracy of the fusion of the point cloud bird's-eye view features and the first image bird's-eye view features and making the obtained fused bird's-eye view features more accurate.

[0106] S260. The first detection head network in the teacher model performs 3D target prediction based on fused bird's-eye view features to obtain the first detection result.

[0107] In some embodiments, the first detection head network includes a first semantic branch, a first centrality branch, a first depth branch, a first detection box size branch, and a first heading angle branch; the first detection result includes a first semantic classification result, a first centrality prediction result, a first centrality depth prediction result, a first size prediction result, and a first heading angle prediction result; thus, the above step S260 may include the following a1-a5:

[0108] a1. The first semantic branch performs semantic classification based on the fused bird's-eye view features to obtain the first semantic classification result.

[0109] The first semantic branch is responsible for predicting the probability (also known as the classification score) of a target (e.g., a sample target during training) belonging to each semantic category. The first semantic classification result represents the predicted semantic category for the sample target, such as pedestrian, vehicle, house, tree, animal, etc. The semantic category of the sample target indicated by the first semantic classification result can be the semantic category with the highest probability among the predicted probabilities of belonging to each semantic category.

[0110] a2. The first centrality branch predicts the center point based on the features of the sample image to obtain the first centrality prediction result.

[0111] The first centrality branch is responsible for predicting the centrality heatmap of the target's 3D bounding box projected onto the pixel coordinate system. The target's 3D bounding box refers to the 3D bounding box that encloses the target; the 3D bounding box can be a cuboid.

[0112] The first centrality prediction result includes a heatmap of the center points of the 3D detection bounding boxes of the sample target projected onto the coarse center points in the pixel coordinate system. In other words, the heatmap of the first centrality prediction result reflects the probability that each coarse center point is the center point of the 3D detection bounding box of the sample target.

[0113] Each coarse center point can be at least one pixel in the corresponding sample image. For example, if the sample image is a 1000×1000 image, and each pixel region consisting of 2×2 pixels corresponds to a coarse center point, then the heatmap of the first centrality prediction result includes the probability that 500×500 coarse center points are the center points of the 3D detection boxes of the sample target.

[0114] a3. The first depth branch performs center point depth prediction based on the fused bird's-eye view features to obtain the first depth prediction result.

[0115] The first depth prediction result includes the predicted depth to the center point of the 3D detection box of the sample target, which can be understood as the distance between the center point of the 3D detection box of the sample target and the origin of the coordinate system.

[0116] a4. The size prediction is performed by the first detection box size branch based on the fused bird's-eye view features to obtain the first size prediction result.

[0117] The first detection box size branch is responsible for predicting the size of the 3D detection box of the sample target, including its length, width, and height. The size of this 3D detection box can be the actual physical size in meters.

[0118] The first size prediction result includes the length, width, and height of the predicted 3D detection box of the sample target in the camera coordinate system.

[0119] a5. The first heading angle prediction result is obtained by predicting the heading angle based on the fused bird's-eye view features from the first heading angle branch.

[0120] The first heading angle branch is responsible for predicting the angle between the target's direction of travel and the X-axis in the three-dimensional coordinate system.

[0121] The first heading angle prediction result includes the angle between the forward direction of the sample target and the X-axis of the three-dimensional coordinate system. The first heading angle prediction result is expressed in radians and the value range is [-π, π], with the clockwise direction being positive.

[0122] In some embodiments, the first semantic classification result, the first depth prediction result, the first size prediction result, and the first heading angle prediction result each include the prediction result of each pixel in the sample image (i.e., semantic classification result, centrality prediction result, central depth prediction result, size prediction result, and heading angle prediction result), that is, the first detection head network performs three-dimensional attribute prediction pixel by pixel.

[0123] In some implementations, the first semantic classification result output by the first semantic branch of the first detection head network can be a classification heatmap. Among them, C Nums The total number of semantic categories representing the sample target, together with the first center depth prediction result, first size prediction result, and first heading angle prediction result output by the first detection head network, are represented as: Wherein, T represents the total number of feature attributes, the first center depth prediction result is one feature attribute, the first size prediction result is three feature attributes (the length, width and height of the three-dimensional detection box of the sample target are each one feature attribute), and the first heading angle prediction result is two attributes (the sine and cosine values ​​of the angle between the forward direction of the sample target and the X-axis of the three-dimensional coordinate system are each one feature attribute).

[0124] S120. Input the sample image into the student model for target detection to obtain the second feature information and the second detection result.

[0125] The second feature information includes the second image bird's-eye view feature and the refined bird's-eye view feature obtained by encoding the second image bird's-eye view feature.

[0126] As mentioned earlier, the input to the teacher model includes sample point cloud data and sample images, while the input to the student model is sample images, excluding point cloud data. Therefore, the inputs to the teacher and student models are different. Furthermore, the overall number of model parameters in the teacher model is greater than that in the student model.

[0127] In this application, the student model includes a feature encoder for extracting image features from an image from a bird's-eye view (referred to as the second image bird's-eye view feature encoder), an encoder for refining the encoding of the second image bird's-eye view feature encoder (referred to as the second bird's-eye view feature encoder), and a detection head network for predicting three-dimensional attributes based on the second bird's-eye view feature encoder (referred to as the second detection head network).

[0128] It can be seen that the basic network architecture of the student model is similar to that of the teacher model. Compared to the teacher model, the student model does not involve a point cloud bird's-eye view feature encoder.

[0129] Furthermore, for neural network modules with similar functions in the student model and the teacher model (e.g., the first image bird's-eye view feature encoder in the teacher model and the second image bird's-eye view feature encoder in the student model, the first bird's-eye view feature encoder in the teacher model and the second bird's-eye view feature encoder in the student model, the first detector head network in the teacher model and the second detector head network in the student model), the neural network module in the student model has fewer parameters than its corresponding neural network module with similar functions in the teacher model. That is, the second image bird's-eye view feature encoder in the student model has fewer parameters than the first image bird's-eye view feature encoder in the teacher model, the second bird's-eye view feature encoder in the student model has fewer parameters than the first bird's-eye view feature encoder in the teacher model, and the second detector head network in the student model has fewer parameters than the first detector head network in the teacher model.

[0130] In some implementations, such as Figure 6 As shown, S120 may include S410-S440, as follows:

[0131] S410. The second image feature encoder in the student model extracts features from the sample image to obtain the second image features.

[0132] S420. The second image features are transformed into a bird's-eye view by the second view transformation network in the student model to obtain the second image bird's-eye view features.

[0133] In this embodiment, the second image bird's-eye view feature encoder includes a second image feature encoder and a second view transformation network cascaded in sequence.

[0134] The second image feature encoder extracts features from the sample images to obtain the second image features. It should be understood that the sample images collected for each sample region can be one or more. When multiple sample images are collected for each sample region, the second image feature encoder can encode the features of each sample image separately to obtain the corresponding second image features.

[0135] Then, the second view transformation network uses the second image features corresponding to the sample regions. Switch to a bird's-eye view to obtain image features from that perspective.

[0136] The data processing process of the second view transformation network is the same as that of the first view transformation network, and will not be repeated here.

[0137] S430. The second bird's-eye view feature encoder in the student model encodes the bird's-eye view features of the second image to obtain the refined bird's-eye view features.

[0138] In one possible implementation, the second bird's-eye view feature encoder has a similar structure to the first bird's-eye view feature encoder, and the above step S430 may include: performing convolution processing on the bird's-eye view features of the second image by the fourth convolutional network in the second bird's-eye view feature encoder to obtain a fourth intermediate feature; performing convolution processing on the first intermediate feature by the fifth convolutional network in the second bird's-eye view feature encoder to obtain a fifth intermediate feature; fusing the fifth intermediate feature and the fourth intermediate feature to obtain a sixth intermediate feature; and performing convolution processing on the sixth intermediate feature by the sixth convolutional network in the second bird's-eye view feature encoder to obtain a refined bird's-eye view feature.

[0139] S440. The second detection head network in the student model performs 3D target prediction based on refined bird's-eye view features to obtain the second detection result.

[0140] In some embodiments, the second detection head network includes a second semantic branch, a second centrality branch, a second depth branch, a second detection box size branch, and a second heading angle branch; the second detection result includes a second semantic classification result, a second centrality prediction result, a second centrality depth prediction result, a second size prediction result, and a second heading angle prediction result; the above step S440 may include the following b1-b5:

[0141] b1. The second semantic branch performs semantic classification based on the fused bird's-eye view features to obtain the second semantic classification result.

[0142] The second semantic branch is responsible for predicting the probability (also known as the classification score) of a target (e.g., a sample target during training) belonging to each semantic category. The semantic category of the sample target indicated by the second semantic classification result can be the semantic category with the highest probability among the predicted probabilities of belonging to each semantic category.

[0143] b2. The second centrality branch predicts the center point based on the features of the sample image to obtain the second centrality prediction result.

[0144] The second centrality branch is responsible for predicting the target's 3D bounding box and projecting it onto a centrality heatmap in the pixel coordinate system using the corresponding camera intrinsics.

[0145] The second centrality prediction result includes a heatmap of the center points of the 3D detection bounding boxes of the sample target projected onto the coarse center points in the pixel coordinate system. In other words, the heatmap of the second centrality prediction result reflects the probability that each coarse center point is the center point of the 3D detection bounding box of the sample target.

[0146] b3. The second depth branch performs center point depth prediction based on the fused bird's-eye view features to obtain the second depth prediction result.

[0147] The second depth prediction result includes the predicted depth to the center point of the 3D detection box of the sample target, which can be understood as the distance between the center point of the 3D detection box of the sample target and the origin of the camera coordinate system.

[0148] b4. The second detection box size branch performs size prediction based on the fused bird's-eye view features to obtain the second size prediction result.

[0149] The second detection box size branch is responsible for predicting the dimensions of the 3D detection box of the sample target in the 3D coordinate system, including length, width, and height. The second size prediction result includes the predicted length, width, and height of the 3D detection box of the sample target in the 3D coordinate system.

[0150] b5. The second heading angle prediction result is obtained by predicting the heading angle based on the fused bird's-eye view features by the second heading angle branch.

[0151] The second heading angle branch is responsible for predicting the angle between the target's direction of travel and the X-axis in the three-dimensional coordinate system. The second heading angle prediction result includes the angle between the target's direction of travel and the X-axis of the three-dimensional coordinate system. The second heading angle prediction result is expressed in radians and ranges from -π to π, with clockwise being positive.

[0152] In some embodiments, the second semantic classification result, the second centrality prediction result, the second depth prediction result, the second size prediction result, and the second heading angle prediction result each include the prediction result of each pixel in the sample image (i.e., the semantic classification result, the centrality prediction result, the central depth prediction result, the size prediction result, and the heading angle prediction result), that is, the second detection head network performs three-dimensional attribute prediction pixel by pixel.

[0153] In some implementations, the second semantic classification result output by the second semantic branch of the second detection head network can be a classification heatmap. Among them, C Nums The total number of semantic categories representing the sample target, together with the second center depth prediction result, second size prediction result, and second heading angle prediction result output by the second detection head network, are represented as:

[0154] S130. Determine the target loss based on at least two of the first distillation loss, the second distillation loss, and the third distillation loss.

[0155] Among them, the first distillation loss reflects the difference between the second image bird's-eye view features and the first image bird's-eye view features and the point cloud bird's-eye view features, respectively; the second distillation loss reflects the difference between the fused bird's-eye view features and the refined bird's-eye view features; and the third distillation loss reflects the difference between the first detection result and the second detection result.

[0156] In some embodiments, the first distillation loss includes a first loss and a second loss; the first loss indicates the difference between the second image bird's-eye view features and the first image bird's-eye view features, and the second loss indicates the difference between the second image bird's-eye view features and the point cloud bird's-eye view features; at this time, before S130, the method further includes: determining the first loss based on the second image bird's-eye view features and the first image bird's-eye view features; and determining the second loss based on the second image bird's-eye view features and the point cloud bird's-eye view features.

[0157] The first loss can be calculated based on the bird's-eye view features of the second and first images using various loss functions, including cross-entropy loss, IoU loss, weighted cross-entropy loss, mean squared error loss, Manhattan distance loss, and cosine similarity loss. For example, when determining the mean squared error loss as the first loss based on the difference between the second and first image bird's-eye view features, the calculation process for the first loss follows the formula in Formula 1, as follows:

[0158]

[0159] Among them, L IMD This is the first loss.

[0160] Student model output And the output of the teacher model Since both modes are the same, the first loss can be reflected in the intramodal distillation under the image mode, so that the student model can learn useful visual semantic knowledge, thereby improving the detection performance of the object detection model.

[0161] However, relying solely on images may not provide sufficient positional and geometric information to help detect 3D targets. To address this limitation, this application also introduces a second loss to perform cross-modal distillation on the coarse camera BEV features in the student model, enabling it to extract 3D spatial knowledge, such as depth distribution and target position and orientation, from the point cloud modality of the teacher model.

[0162] In some embodiments, a second loss can be obtained by calculating the loss between the second image bird's-eye view features and the point cloud bird's-eye view features based on a preset loss function. Preset loss functions include, for example, the weighted cross-entropy loss function and the mean squared error loss function mentioned above.

[0163] Since the second image bird's-eye view features are derived from the image and the point cloud bird's-eye view features are derived from the point cloud data, the second loss is a cross-modal distillation loss.

[0164] In some implementations, determining the second loss based on the second image bird's-eye view features and the point cloud bird's-eye view features includes: acquiring target attribute annotation information corresponding to the sample image and sample point cloud data, the target attribute annotation information including the annotation position information of the sample targets; generating a region mask in the bird's-eye view based on the annotation position information of each sample target; the region mask is used to indicate the position of the center point of each sample target in the bird's-eye view; performing masking processing on the sample point cloud bird's-eye view features and the second image bird's-eye view features respectively based on the region mask to obtain masked sample point cloud bird's-eye view features and masked image bird's-eye view features; and determining the second loss based on the masked sample point cloud bird's-eye view features and the masked image bird's-eye view features.

[0165] The labeled location information is used to indicate the true three-dimensional location of the sample target in the sample area.

[0166] In some implementations, the labeled location information may include the location information of the reference point of the 3D detection box of each sample target in the sample area. The reference point may be the center point of the 3D detection box of the sample target. Thus, the spatial range occupied by the 3D detection box of the sample target in the sample area can be determined based on the location information of the center point of the 3D detection box of the sample target and the size of the 3D detection box of the sample target.

[0167] The center point of the 3D detection bounding box of the sample target can be divided into corresponding voxel meshes, resulting in the voxel mesh containing the center point of the sample target's 3D detection bounding box. This voxel mesh is then used as the target voxel mesh corresponding to the sample target. The mask value of the target voxel mesh corresponding to the sample target is set to 1, while the mask values ​​of other voxel meshes are set to 0. Next, the target voxel mesh corresponding to the sample target is transformed to a bird's-eye view to obtain the region mask in the bird's-eye view. In the region mask in the bird's-eye view, the mask value in the region containing the sample target can be 1, while the mask value in other regions outside the sample target region is 0.

[0168] Next, the region mask of the sample target in the bird's-eye view is multiplied element-wise with the bird's-eye view features of the sample point cloud to extract the features of the sample target from the bird's-eye view features of the sample point cloud, without needing to focus on the features of other objects (such as the background) other than the sample target, to obtain the bird's-eye view features of the mask sample point cloud. Then, the region mask in the bird's-eye view is multiplied element-wise with the bird's-eye view features of the second image to extract the features of the sample target from the bird's-eye view features of the second image, to obtain the bird's-eye view features of the mask image. After that, based on the bird's-eye view features of the mask sample point cloud and the bird's-eye view features of the mask image, the loss can be calculated using the cross-entropy loss function, the IoU loss function, the weighted cross-entropy loss function, the mean squared error loss function, the Manhattan distance loss function, and the cosine similarity loss function, as the second loss.

[0169] For example, when the loss function used to calculate the second loss is the error loss function, the aforementioned calculation process for determining the second loss refers to Formula 2, which is as follows:

[0170]

[0171] Among them, L CMD For the second loss, ⊙ is multiplied element by element, and M is the region mask.

[0172] Compared to the method described above, which calculates the loss using the bird's-eye view features of the second image and the bird's-eye view features of the point cloud, this embodiment generates a center point region mask M for each sample target in the BEV space based on the labeled location information of each sample target. The region of interest (i.e., the region where each sample target is located) is then extracted using the region mask. Finally, the loss is calculated on the features of different modalities within the region of interest to obtain the second loss. Since this second loss is based on the features of the sample target in the image and the features of the sample target in the aggregated point cloud data, and does not involve the features of other objects besides the sample target in the two modalities, this second loss allows distillation to focus more intently on the target, and the student model can learn cross-modal features, thereby narrowing the cross-modal gap.

[0173] In this application, the loss can be calculated based on the fusion of bird's-eye view features and refined bird's-eye view features, using cross-entropy loss function, IoU loss function, weighted cross-entropy loss function, mean squared error loss function, Manhattan distance loss function, and cosine similarity loss function as the second distillation loss.

[0174] For example, when the loss function for the second distillation loss is determined to be the mean square error loss function, the calculation process for the second distillation loss refers to Formula 3, which is as follows:

[0175]

[0176] Among them, LMMD_F This is the loss from the second distillation. To incorporate bird's-eye view features, To refine the aerial view features.

[0177] As mentioned above, the first bird's-eye view feature encoder in the teacher model and the second bird's-eye view feature encoder in the student model have similar structures. The teacher model outputs a deeply refined fused bird's-eye view feature. The student model outputs a more detailed bird's-eye view feature after further refinement. The two types of features will then be fed into their respective detection head networks for prediction. And to enable the student model... Feature learning acquires multimodal knowledge (including 3D position and geometric information, etc.). Therefore, a second distillation loss is designed to determine the difference between the fused bird's-eye view features of the teacher model and the refined bird's-eye view features of the student model. When training the student model with the second distillation loss, the fused bird's-eye view features of the teacher model and the refined bird's-eye view features of the student model are aligned, thereby improving the student model's ability to learn multimodal knowledge.

[0178] Based on the first and second detection results, the loss can be calculated using the cross-entropy loss function, IoU loss function, weighted cross-entropy loss function, mean squared error loss function, Manhattan distance loss function, and cosine similarity loss function, and used as the third distillation loss.

[0179] For example, when determining the loss function SmoothL1 for the third distillation loss, the calculation process for the third distillation loss refers to Formula 4, which is as follows:

[0180]

[0181] Among them, L MMD―F This is the loss from the third distillation.

[0182] In some implementations, the first detection result includes first target attribute information, which includes first semantic recognition information and the first pose information of the sample target indicated by the first semantic recognition information; the second detection result includes second target attribute information; the second target attribute information includes second semantic recognition information and the second pose information of the sample target indicated by the second semantic recognition information; before S130, the method further includes: determining a third distillation loss based on the difference between the first semantic recognition information and the second semantic recognition information, and the difference between the first pose information of the sample target indicated by the first semantic recognition information and the second pose information of the target indicated by the first semantic recognition information.

[0183] The first semantic recognition information may include the aforementioned first semantic classification result and first size prediction result; the first pose information may include the first center depth prediction result and first heading angle prediction result; correspondingly, the second semantic recognition information may include the aforementioned second semantic classification result and second size prediction result; and the second pose information may include the second center depth prediction result and second heading angle prediction result. Therefore, a first semantic classification loss can be obtained by calculating the loss based on the first and second semantic classification results; a first centrality prediction loss can be obtained by calculating the loss based on the first and second centrality prediction results; a first size prediction loss can be obtained by calculating the loss value based on the first and second size prediction results; a first center depth prediction loss can be obtained by calculating the loss based on the first and second center depth prediction results; and a first heading angle prediction loss can be obtained by calculating the loss based on the first and second heading angle prediction results. Then, the first semantic classification loss, the first centrality prediction loss, the first size prediction loss, the first center depth prediction loss, and the first heading angle prediction loss are weighted and summed to obtain the third distillation loss.

[0184] It should be understood that the loss functions used for attribute prediction loss for the aforementioned different attributes can be different or the same, and can be selected according to actual needs.

[0185] After determining the first distillation loss, the second distillation loss, and the third distillation loss, at least two of these three distillation losses can be selected. The target loss is obtained by weighted summation of the selected at least two distillation losses.

[0186] In some embodiments, S130 may include: obtaining a target supervision loss, which is determined based on the differences between the target attribute annotation information corresponding to the sample image and the first detection result and the second detection result, respectively; and weighting at least two of the first distillation loss, the second distillation loss and the third distillation loss with the target supervision loss to obtain a target loss.

[0187] Target attribute annotation information refers to the supervision information used to train student models. Target attribute annotation information can include the true semantic category of the sample target and the true pose information of the sample target in the three-dimensional coordinate system.

[0188] In some embodiments, if the first detection result includes a first semantic classification result, a first centrality prediction result, a first centrality depth prediction result, a first size prediction result, and a first heading angle prediction result, then the target attribute annotation information may include semantic ground truth, centrality ground truth, centrality depth ground truth, size ground truth, and heading angle ground truth.

[0189] Therefore, the second semantic classification loss can be obtained by calculating the loss based on the first semantic classification result and the semantic ground truth; the second size prediction loss can be obtained by calculating the loss value based on the first size prediction result and the size ground truth; the second centrality prediction loss can be obtained by calculating the loss value based on the first center depth prediction result and the centrality ground truth; the second center depth prediction loss can be obtained by calculating the loss value based on the first center depth prediction result and the center depth ground truth; the second heading angle prediction loss can be obtained by calculating the loss value based on the first heading angle prediction result and the heading angle ground truth; and then the second semantic classification loss, the second size prediction loss, the second center depth prediction loss, the second centrality prediction loss, and the second heading angle prediction loss are weighted and summed to obtain the third loss.

[0190] Accordingly, the third semantic classification loss can be obtained by calculating the loss based on the second semantic classification result and the semantic ground truth; the third size prediction loss can be obtained by calculating the loss based on the second size prediction result and the size ground truth; the third centrality prediction loss can be obtained by calculating the loss based on the second center depth prediction result and the centrality ground truth; the second center depth prediction loss can be obtained by calculating the loss based on the second center depth prediction result and the center depth ground truth; the third heading angle prediction loss can be obtained by calculating the loss based on the second heading angle prediction result and the heading angle ground truth; and then the third semantic classification loss, the third size prediction loss, the third centrality prediction loss, the third center depth prediction loss, and the third heading angle prediction loss are weighted and summed to obtain the fourth loss.

[0191] Then, the third and fourth losses are summed to obtain the target supervision loss. It should be understood that, from a perspective perspective, the loss functions used for attribute prediction losses of different attributes can be different or the same; the choice depends on the actual needs.

[0192] Then, the third and fourth losses are summed to obtain the target supervision loss, which is used to indicate the difference between the detection results of the teacher model and the student model, so that when the student model is trained with the target supervision loss, the student model learns the predictive ability of the teacher model.

[0193] After determining the target monitoring loss, at least two of the first distillation loss, second distillation loss, and third distillation loss are weighted and calculated together with the target monitoring loss to obtain the target loss. For example, the target loss can be obtained by weighting the second distillation loss, third distillation loss, and target monitoring loss together; alternatively, the target loss can be obtained by weighting the first distillation loss, second distillation loss, and target monitoring loss together; or, all three can be weighted and calculated together.

[0194] For example, if the target loss is obtained by weighting the first distillation loss, the second distillation loss, the third distillation loss, and the target monitoring loss, the calculation process of the target loss refers to Formula 5, which is as follows:

[0195] L Total =L Det +W IMD ·L IMD +W CMD ·L CMD +W MMD_F ·L MMD_F +W MMD_P ·L MMD_P ;(five)

[0196] Among them, L Total For the target loss, L Det To monitor the loss of the target, W IMD W CMD W MMD_F W MMD_P The weights W represent the first loss, the second loss, the second distillation loss, and the third distillation loss, respectively. IMD W CMD W MMD_F W MMD_P It can be set according to needs; this application does not impose any restrictions.

[0197] S140. Adjust the parameters of the student model at least according to the target loss until the training termination condition is met. Use the student detection model after the training termination condition is met as the target detection model.

[0198] In some embodiments, since the teacher model may have been pre-trained, in S140, the parameters of the teacher model can be fixed, and the parameters of the student model can be adjusted according to the target loss to minimize the target loss. The training termination condition may be that the number of iterations of the student model parameters reaches a preset number, or that the target loss is less than a preset loss threshold.

[0199] In some embodiments, the parameters of both the student model and the teacher model can be adjusted simultaneously based on the target loss to minimize the target loss.

[0200] In this application, the teacher model first determines the first feature information and the first detection result based on sample images and sample point cloud data, and the student model determines the second feature information and the second detection result based on sample images. Then, based on the first feature information, the first detection result, the second feature information, and the second detection result, the first distillation loss, the second distillation loss, and the third distillation loss are determined. Then, the target loss is determined by at least one of the first distillation loss, the second distillation loss, and the third distillation loss. The parameters of the student model are adjusted by the target loss so that the teacher model is not limited by modality. At the same time, the complementary knowledge of two modal data, namely point cloud data obtained by radar and image data obtained by camera, is integrated and transferred to the student model to improve the detection capability of the student model. Furthermore, the first distillation loss is determined based on shallow-level first image bird's-eye view features, second image bird's-eye view features, and point cloud bird's-eye view features. Therefore, when training the student model based on the first distillation loss, the student model, in addition to learning visual knowledge, can also learn the 3D features of the sample point cloud data, effectively reducing feature differences between different modalities, thus achieving coarse feature distillation across modalities. The second distillation loss is determined based on deep-level fused bird's-eye view features and refined bird's-eye view features. Therefore, when training the student model based on the second distillation loss, the student model can learn the multimodal feature extraction capabilities of the teacher model, thus achieving refined feature distillation across modalities. The third distillation loss is based on a more refined second prediction result. And the first prediction result is determined. Thus, when training the student model based on the third distillation loss, the gap between the prediction response of the teacher model and the student model is narrowed, and more refined prediction distillation is achieved. Therefore, at least two of the first distillation loss, the second distillation loss and the third distillation loss are combined to determine the target loss. The target loss involves distillation loss with different degrees of refinement. This enables multi-level distillation supervision from coarse to fine when training the student model based on the target loss. It guides the student model to learn better 3D detection feature representations and improves the detection capability of the target detection model after training based on 2D images for 3D target detection, thereby improving the accuracy of the target detection model for 3D target detection.

[0201] Furthermore, the first distillation loss is determined based on the shallow-level first image bird's-eye view features, second image bird's-eye view features, and point cloud bird's-eye view features output from the front-end network parts of the teacher and student models. Since the first image bird's-eye view features, second image bird's-eye view features, and point cloud bird's-eye view features obtained by the teacher and student models are relatively coarse, the first loss indicates the distillation loss of coarse features within the camera modality, while the second loss indicates the distillation loss of coarse features across modalities under the bird's-eye view perspective. Similarly, fusing bird's-eye view features is obtained by further processing the first image bird's-eye view features and point cloud bird's-eye view features, and refining the bird's-eye view features is also achieved by processing the second image bird's-eye view features. Therefore, the second distillation loss determined based on the deep, refined bird's-eye view features and the fused bird's-eye view features indicates the distillation loss of the refined features across modalities. The first detection result is obtained by further processing the fused bird's-eye view features, and the second detection result is obtained by further processing the refined bird's-eye view features. Therefore, the third distillation loss determined based on the first and second detection results is a more refined predictive distillation loss. Thus, when training the student model by fusing the first, second, and third distillation losses, a multi-level distillation from coarse to fine is achieved, which can ensure the accuracy of the target detection model in 3D target detection based on images after training.

[0202] For example, the training process of an object detection model is as follows: Figure 7 As shown. The teacher model takes sample point cloud data and sample images from multiple cameras as input, and includes a "point cloud / first image feature encoder" + "camera view transformation module" + "first bird's-eye view feature encoder" + "first detection head network". The student model takes sample images as input, and includes a "second image feature encoder" + "camera view transformation module" + "second bird's-eye view feature encoder" + "second detection head network".

[0203] The teacher model's point cloud feature encoder encodes the sample point cloud data to obtain sample point cloud features. A bird's-eye view transformation is then performed on the sample point cloud features to obtain point cloud bird's-eye view features. Simultaneously, the teacher model's first image feature encoder encodes the sample image to obtain first image features. A bird's-eye view transformation is then performed on the first image features to obtain first image bird's-eye view features. Then, the teacher model's first bird's-eye view feature encoder fuses and encodes the point cloud bird's-eye view features and the first image bird's-eye view features to obtain fused bird's-eye view features. Finally, the teacher model's first detection head network performs predictions based on the fused bird's-eye view features to obtain the first detection result.

[0204] The sample image is encoded by the second image feature encoder of the student model to obtain the second image feature. The second image feature is then transformed by a bird's-eye view to obtain the second image bird's-eye view feature. Then, the second image bird's-eye view feature is further refined by the second bird's-eye view feature encoder of the student model to obtain the refined bird's-eye view feature. Finally, the second detection head network of the student model makes a prediction based on the fused bird's-eye view feature to obtain the second detection result.

[0205] A first loss can be determined based on the first image bird's-eye view features and the second image bird's-eye view features, indicating the distillation loss within the image modality; a second loss can be determined based on the point cloud bird's-eye view features and the second image bird's-eye view features, indicating the distillation loss across modalities (across image and point cloud data), and the first and second losses are used as the first distillation loss; a second distillation loss can be determined based on the fused bird's-eye view features and the refined bird's-eye view features, indicating the distillation loss across modalities (across camera and radar); a third distillation loss can be determined based on the first and second detection results, indicating the distillation loss of the prediction response between the teacher model and the student model, and a detection loss is determined separately for the teacher model as the third loss, and a detection loss is determined separately for the student model as the fourth loss. The third loss and the fourth loss are summed as the target supervision loss, and then the parameters of the student model and the teacher model are adjusted through the target supervision loss, and the parameter-adjusted student model is obtained as the target detection model.

[0206] Please see Figure 8 , Figure 8 This application illustrates a flowchart of a target detection method according to an embodiment of the present application. The method can be executed by an electronic device, such as... Figure 1 The method, which includes S510-S520, can be further described in detail below, using the server 200 or terminal device 400 shown.

[0207] S510. Acquire the image to be detected.

[0208] In one possible implementation, the image to be detected can be an image captured by a camera mounted on a target device. The target device can be a vehicle, robot, industrial equipment, or medical equipment, etc. Similarly, the captured image to be detected can be multiple images from multiple viewpoints.

[0209] The image to be detected and the target point cloud image can also be acquired for the same target scene. The positional relationship between the acquisition devices of the image to be detected and the target point cloud image is not specifically limited in this embodiment.

[0210] S520. Perform 3D target detection on the image to be detected using the target detection model to obtain the target detection result.

[0211] The target detection model was trained using the training method described in the aforementioned embodiments.

[0212] The specific process and implementation principle of the target detection model performing three-dimensional target detection based on the image to be detected to obtain the target detection result can be referred to in the specific description of the student model determining the second detection result in the aforementioned embodiment, and will not be repeated in this embodiment.

[0213] In one possible implementation, the image to be detected is acquired by the vehicle. In this case, the target detection result includes the three-dimensional attribute information of obstacles around the vehicle. For example, six cameras are arranged around the vehicle to acquire images from the left front, front, right front, left rear, front rear, and right rear views. These six images are used as the images to be detected, and the target detection model determines the target detection result based on these six images.

[0214] By adopting the above method, the target detection method of this application can be applied to vehicles equipped with surround-view cameras, such as autonomous vehicles and unmanned delivery vehicles, so as to realize accurate and reliable 3D position detection and size estimation of obstacle targets (large vehicles, small vehicles, tricycles, two-wheeled vehicles, pedestrians, cones, etc.) in road scenes, so as to assist the vehicle in performing subsequent path planning and decision control.

[0215] In this embodiment, the 3D target detection process no longer requires point cloud data as input, enabling pure camera deployment and reducing the cost of radar deployment. Simultaneously, the target detection process no longer needs to process point cloud data, significantly reducing the amount of data that the target detection model needs to process and improving detection efficiency.

[0216] Please see Figure 9 , Figure 9 A block diagram of a model training apparatus 1000 according to an embodiment of this application is shown. The model training apparatus 1000 includes:

[0217] The first detection module 1010 is used to input sample images and sample point cloud data collected for the sample area into the teacher model for target detection, and obtain first feature information and first detection result. The first feature information includes first image bird's-eye view features, point cloud bird's-eye view features, and fused bird's-eye view features obtained by fusing and encoding the first image bird's-eye view features and point cloud bird's-eye view features.

[0218] The second detection module 1020 is used to input the sample image into the student model for target detection, and obtain the second feature information and the second detection result; the second feature information includes the second image bird's-eye view feature and the refined bird's-eye view feature obtained by encoding the second image bird's-eye view feature;

[0219] The loss determination module 1030 is used to determine the target loss based on at least two of the first distillation loss, the second distillation loss, and the third distillation loss; the first distillation loss reflects the difference between the second image bird's-eye view feature and the first image bird's-eye view feature and the point cloud bird's-eye view feature, respectively; the second distillation loss reflects the difference between the fused bird's-eye view feature and the refined bird's-eye view feature; and the third distillation loss reflects the difference between the first detection result and the second detection result.

[0220] The adjustment module 1040 is used to adjust the parameters of the student model at least according to the target loss until the training termination condition is met, and the student detection model after the training termination condition is met is used as the target detection model.

[0221] Optionally, the first distillation loss includes a first loss and a second loss; the loss determination module 1030 is further configured to determine the first loss based on the bird's-eye view features of the second image and the bird's-eye view features of the first image; and to determine the second loss based on the bird's-eye view features of the second image and the bird's-eye view features of the point cloud.

[0222] Optionally, the loss determination module 1030 is further configured to acquire target attribute annotation information corresponding to the sample image and sample point cloud data, the target attribute annotation information including the annotation position information of the sample target; generate a region mask in the bird's-eye view based on the annotation position information of each sample target; the region mask is used to indicate the position of the center point of each sample target in the bird's-eye view; perform masking processing on the bird's-eye view features of the sample point cloud and the bird's-eye view features of the second image based on the region mask, to obtain the bird's-eye view features of the masked sample point cloud and the bird's-eye view features of the masked image; and determine the second loss based on the bird's-eye view features of the masked sample point cloud and the bird's-eye view features of the masked image.

[0223] Optionally, the first detection result includes first target attribute information, which includes first semantic recognition information and the first pose information of the sample target indicated by the first semantic recognition information; the second detection result includes second target attribute information; the second target attribute information includes second semantic recognition information and the second pose information of the sample target indicated by the second semantic recognition information; the loss determination module 1030 is further configured to determine the third distillation loss based on the difference between the first semantic recognition information and the second semantic recognition information, and the difference between the first pose information of the sample target indicated by the first semantic recognition information and the second pose information of the target indicated by the first semantic recognition information.

[0224] Optionally, the loss determination module 1030 is further configured to obtain the target supervision loss, which is determined based on the differences between the target attribute annotation information corresponding to the sample image and the first detection result and the second detection result, respectively; and to calculate the target loss by weighting at least two of the first distillation loss, the second distillation loss and the third distillation loss with the target supervision loss.

[0225] Optionally, the loss determination module 1030 is also used to perform weighted calculation of the first distillation loss, the second distillation loss, the third distillation loss and the target monitoring loss to obtain the target loss.

[0226] Optionally, the first detection module 1010 is further configured to encode the sample point cloud data by the point cloud feature encoder in the teacher model to obtain sample point cloud features; transform the sample point cloud features to a bird's-eye view to obtain point cloud bird's-eye view features; encode the sample image by the first image feature encoder in the teacher model to obtain first image features; transform the first image features to a bird's-eye view by the first view transformation network in the teacher model to obtain first image bird's-eye view features; fuse the point cloud bird's-eye view features and the first image bird's-eye view features by the first bird's-eye view feature encoder in the teacher model to obtain fused bird's-eye view features; and perform three-dimensional target prediction based on the fused bird's-eye view features by the first detection head network in the teacher model to obtain a first detection result.

[0227] Optionally, the first detection head network includes a first semantic branch, a first centrality branch, a first depth branch, a first detection box size branch, and a first heading angle branch; the first detection result includes a first semantic classification result, a first centrality prediction result, a first center depth prediction result, a first size prediction result, and a first heading angle prediction result; the first detection module 1010 is further configured to perform semantic classification based on fused bird's-eye view features by the first semantic branch to obtain a first semantic classification result; perform center point prediction based on fused bird's-eye view features by the first centrality branch to obtain a first centrality prediction result; perform center point depth prediction based on fused bird's-eye view features by the first depth branch to obtain a first center depth prediction result; perform size prediction based on fused bird's-eye view features by the first detection box size branch to obtain a first size prediction result; and perform heading angle prediction based on fused bird's-eye view features by the first heading angle branch to obtain a first heading angle prediction result.

[0228] Optionally, the first detection module 1010 is further configured to process the sample image features by the first view transformation network according to the following process: determining the image view pseudo-point cloud features based on the pixel features in the first image features; transforming the image view pseudo-point cloud features to the device coordinate system of the reference device according to the camera intrinsic and extrinsic parameters of the camera from which the sample image originates, to obtain intermediate pseudo-point cloud features, wherein the camera from which the sample image originates is located on the reference device; dividing the intermediate pseudo-point cloud features into multiple voxel grids, and aggregating the pseudo-point cloud features located in the same voxel grid to obtain four-dimensional pseudo-voxel features; and reducing the dimensionality of the four-dimensional pseudo-voxel features to obtain the first image bird's-eye view features.

[0229] Optionally, the first detection module 1010 is further configured to predict the probability that each pixel in the sample image is dispersed to multiple discrete depths based on the pixel features in the sample image features; determine the discrete depth with the highest probability as the target depth of the pixel based on the probability that each pixel in the sample image is dispersed to multiple discrete depths; and determine the pseudo point cloud features of the image view based on the target depth of each pixel in the sample image and the pixel features of each pixel in the sample image features.

[0230] Optionally, the second detection module 1020 is further configured to extract features from the sample image by the second image feature encoder in the student model to obtain second image features; transform the second image features to a bird's-eye view by the second view transformation network in the student model to obtain second image bird's-eye view features; encode the second image bird's-eye view features by the second bird's-eye view feature encoder in the student model to obtain refined bird's-eye view features; and perform three-dimensional target prediction based on the refined bird's-eye view features by the second detection head network in the student model to obtain the second detection result.

[0231] Please see Figure 10 , Figure 10 This illustration shows a block diagram of a target detection apparatus according to an embodiment of the present application. The target detection apparatus 1100 includes:

[0232] The acquisition module 1110 is used to acquire the image to be detected;

[0233] The target detection module 1120 is used to perform three-dimensional target detection on the image to be detected through the target detection model and obtain the target detection result; the target detection model is trained according to the method described in the first aspect above.

[0234] It should be noted that the device embodiments in this application correspond to the aforementioned method embodiments. The specific principles in the device embodiments can be found in the content of the aforementioned method embodiments, and will not be repeated here.

[0235] Please see Figure 11 , Figure 11 A structural block diagram of an electronic device for performing a model training method according to an embodiment of this application is shown. The electronic device may be... Figure 1 The terminal device 400 or server 200 mentioned should be noted. Figure 11 The computer system 1200 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0236] like Figure 11As shown, the computer system 1200 includes a Central Processing Unit (CPU) 1201, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on programs stored in Read-Only Memory (ROM) 1202 or programs loaded from storage portion 1208 into Random Access Memory (RAM) 1203. The RAM 1203 also stores various programs and data required for system operation. The CPU 1201, ROM 1202, and RAM 1203 are interconnected via a bus 1204. An Input / Output (I / O) interface 1205 is also connected to the bus 1204.

[0237] The following components are connected to I / O interface 1205: an input section 1206 including a keyboard, mouse, etc.; an output section 1207 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to I / O interface 1205 as needed. Removable media 1211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1210 as needed so that computer programs read from them can be installed into storage section 1208 as needed.

[0238] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1209, and / or installed from removable medium 1211. When the computer program is executed by central processing unit (CPU) 1201, it performs various functions defined in the system of this application.

[0239] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0240] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer-readable instructions.

[0241] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0242] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable storage medium carries computer-readable instructions that, when executed by a processor, implement the methods in any of the above embodiments.

[0243] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including computer-readable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-readable instructions from the computer-readable storage medium, and the processor executes the computer-readable instructions, causing the electronic device to perform the methods of any of the above embodiments.

[0244] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal. It can be implemented wholly or partially using software, hardware (e.g., processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that functions as a whole.

[0245] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0246] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause an electronic device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the methods according to the embodiments of this application.

[0247] Other embodiments of this application will readily conceive of by those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. It should be understood that this application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

[0248] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A model training method, characterized in that, The method comprises: inputting a sample image and sample point cloud data collected towards a sample area into a teacher model for target detection to obtain first feature information and a first detection result, the first feature information comprising first image bird's eye view features, point cloud bird's eye view features, and fusion bird's eye view features obtained by fusing and encoding the first image bird's eye view features and the point cloud bird's eye view features; inputting the sample image into a student model for target detection to obtain second feature information and a second detection result; the second feature information comprising second image bird's eye view features and refined bird's eye view features obtained by encoding the second image bird's eye view features; determining a target loss according to at least two of a first distillation loss, a second distillation loss, and a third distillation loss; the first distillation loss reflecting differences between the second image bird's eye view features and the first image bird's eye view features and the point cloud bird's eye view features respectively; the second distillation loss reflecting a difference between the fusion bird's eye view features and the refined bird's eye view features; and the third distillation loss reflecting a difference between the first detection result and the second detection result; adjusting parameters of the student model according to the target loss until a training end condition is reached, and taking the student detection model after the training end condition is reached as a target detection model.

2. The method of claim 1, wherein, The first distillation loss comprises a first loss and a second loss; Before the target loss is determined according to at least two of the first distillation loss, the second distillation loss, and the third distillation loss, the method further comprises: determining the first loss according to the second image bird's eye view features and the first image bird's eye view features; determining the second loss according to the second image bird's eye view features and the point cloud bird's eye view features.

3. The method of claim 2, wherein, The determining of the second loss according to the second image bird's eye view features and the point cloud bird's eye view features comprises: obtaining target attribute label information corresponding to the sample image and sample point cloud data, the target attribute label information comprising label position information of sample targets; generating a region mask in a bird's eye view perspective according to the label position information of the sample targets; the region mask is used to indicate positions of center points of the sample targets in the bird's eye view perspective; respectively performing mask processing on the sample point cloud bird's eye view features and the second image bird's eye view features according to the region mask to obtain mask sample point cloud bird's eye view features and mask image bird's eye view features; determining the second loss based on the mask sample point cloud bird's eye view features and the mask image bird's eye view features.

4. The method of claim 1, wherein, The first detection result comprises first target attribute information, the first target attribute information comprising first semantic recognition information and first pose information of a sample target indicated by the first semantic recognition information; the second detection result comprises second target attribute information; the second target attribute information comprises second semantic recognition information and second pose information of a sample target indicated by the second semantic recognition information; Before the target loss is determined according to at least two of the first distillation loss, the second distillation loss, and the third distillation loss, the method further comprises: Determine a third distillation loss according to a difference between the first semantic recognition information and the second semantic recognition information, and a difference between first pose information of a sample target indicated by the first semantic recognition information and second pose information of the target indicated by the first semantic recognition information.

5. The method according to any one of claims 1 to 4, characterized in that, The determining the target loss according to at least two of the first distillation loss, the second distillation loss and the third distillation loss comprises: Obtaining a target supervision loss determined according to a difference between target attribute label information corresponding to the sample image and the first detection result and a difference between the target attribute label information and the second detection result; The target loss is obtained by performing weighted calculation on at least two of the first distillation loss, the second distillation loss and the third distillation loss and the target supervision loss.

6. The method of claim 5, wherein, The target loss is obtained by performing weighted calculation on at least two of the first distillation loss, the second distillation loss and the third distillation loss and the target supervision loss. The target loss is obtained by performing weighted calculation on the first distillation loss, the second distillation loss, the third distillation loss and the target supervision loss.

7. The method of claim 1, wherein, The target loss is obtained by performing weighted calculation on the first distillation loss, the second distillation loss, the third distillation loss and the target supervision loss. The target loss is obtained by performing weighted calculation on the first distillation loss, the second distillation loss, the third distillation loss and the target supervision loss. The sample point cloud data is encoded by a point cloud feature encoder in the teacher model to obtain sample point cloud features; The sample point cloud features are transformed to a bird's eye view to obtain point cloud bird's eye view features; The sample image is encoded by a first image feature encoder in the teacher model to obtain first image features; The first image features are converted to a bird's eye view by a first view conversion network in the teacher model to obtain first image bird's eye view features; The point cloud bird's eye view features and the first image bird's eye view features are fused and encoded by a first bird's eye view feature encoder in the teacher model to obtain fused bird's eye view features; 8. The method of claim 7, wherein, The first detection result is obtained by performing three-dimensional target prediction based on the fused bird's eye view features by a first detection head network in the teacher model. The first detection head network comprises a first semantic branch, a first center degree branch, a first depth branch, a first detection frame size branch and a first heading angle branch. The first detection result comprises a first semantic classification result, a first center degree prediction result, a first center depth prediction result, a first size prediction result and a first heading angle prediction result. The first detection result is obtained by performing three-dimensional target prediction based on the fused bird's eye view features by a first detection head network in the teacher model. The first semantic classification result is obtained by performing semantic classification based on the fused bird's eye view features by the first semantic branch; The first center degree prediction result is obtained by performing center point prediction based on the fused bird's eye view features by the first center degree branch; The first center depth prediction result is obtained by performing center point depth prediction based on the fused bird's eye view features by the first depth branch; perform size prediction based on the fused bird's eye view feature, to obtain the first size prediction result; perform heading angle prediction based on the fused bird's eye view feature, to obtain the first heading angle prediction result.

9. The method of claim 7, wherein, The first view conversion network in the teacher model converts the first image feature to a bird's eye view, to obtain a first image bird's eye view feature, including: The first view conversion network processes the sample image feature according to the following process: determine an image view pseudo point cloud feature based on the pixel feature in the first image feature; transform the image view pseudo point cloud feature to a device coordinate system of a reference device according to camera intrinsic parameters and camera extrinsic parameters of a camera from which the sample image is derived, to obtain an intermediate pseudo point cloud feature, the camera being arranged on the reference device; divide the intermediate pseudo point cloud feature into a plurality of voxel grids, and aggregate pseudo point cloud features located in the same voxel grid, to obtain a four-dimensional pseudo voxel feature; perform dimension reduction on the four-dimensional pseudo voxel feature, to obtain the first image bird's eye view feature.

10. The method of claim 9, wherein, The method includes: predict the probability of each pixel in the sample image being dispersed to a plurality of discrete depths based on the pixel feature in the sample image feature; determine the discrete depth with the maximum probability as the target depth of the pixel according to the probability of each pixel in the sample image being dispersed to a plurality of discrete depths; determine an image view pseudo point cloud feature based on the target depth of each pixel in the sample image and the pixel feature of each pixel in the sample image feature.

11. The method of claim 1, wherein, The method includes: extract a second image feature from the sample image by a second image feature encoder in the student model; transform the second image feature to a bird's eye view by a second view conversion network in the student model, to obtain a second image bird's eye view feature; encode the second image bird's eye view feature by a second bird's eye view feature encoder in the student model, to obtain a refined bird's eye view feature; perform three-dimensional target prediction based on the refined bird's eye view feature by a second detection head network in the student model, to obtain a second detection result.

12. A target detection method characterized by, The method includes: obtain a to-be-detected image; perform three-dimensional target detection on the to-be-detected image by a target detection model, to obtain a target detection result, the target detection model being trained according to the method in any one of claims 1-11.

13. A model training apparatus, comprising: The apparatus includes: a first detection module configured to input a sample image and sample point cloud data facing a sample region into a teacher model to perform target detection, to obtain first feature information and a first detection result, the first feature information including a first image bird's eye view feature, a point cloud bird's eye view feature, and a fused bird's eye view feature obtained by fusing and encoding the first image bird's eye view feature and the point cloud bird's eye view feature; The second detection module is configured to input the sample image into the student model to perform target detection, and obtain second feature information and a second detection result. The second feature information includes second image bird's eye view features and refined bird's eye view features obtained by encoding the second image bird's eye view features. The loss determination module is configured to determine a target loss according to at least two of the first distillation loss, the second distillation loss, and the third distillation loss. The first distillation loss reflects differences between the second image bird's eye view features and the first image bird's eye view features and the point cloud bird's eye view features, respectively. The second distillation loss reflects a difference between the fusion bird's eye view features and the refined bird's eye view features. The third distillation loss reflects a difference between the first detection result and the second detection result. The adjustment module is configured to adjust at least parameters of the student model according to the target loss until a training end condition is reached, and to use the student detection model after the training end condition is reached as a target detection model.

14. A target detection apparatus characterized by comprising: The device comprises: An acquisition module configured to acquire a to-be-detected image. A target detection module configured to perform three-dimensional target detection on the to-be-detected image by using a target detection model, and obtain a target detection result. The target detection model is trained according to the method in any one of claims 1-11.

15. An electronic device, comprising: Comprise: A processor; A memory having computer readable instructions stored thereon, wherein the computer readable instructions, when executed by the processor, implement the method in any one of claims 1-12.

16. A computer readable storage medium, characterized in that, Having computer readable instructions stored thereon, wherein the computer readable instructions, when executed by a processor, implement the method in any one of claims 1-12.

17. A computer program product, characterised in that, Comprise computer readable instructions, wherein the computer readable instructions, when executed by a processor, implement the method in any one of claims 1-12.

Images