BEV target detection method, device, equipment, storage medium and program product

Abstract

The present application relates to a BEV target detection method, device, equipment, storage medium and program product. The method includes: extracting features from surround-view multi-camera image samples, performing perspective transformation on the extracted first multi-scale image features and second multi-scale image features to obtain first sparse BEV features and second sparse BEV features; extracting features from the first sparse BEV features to obtain first dense BEV features; extracting features from the second sparse BEV features to obtain second dense BEV features; constructing a first multi-granularity distillation loss based on the first multi-scale image features and the second multi-scale image features; constructing a second multi-granularity distillation loss based on the first dense BEV features and the second dense BEV features; optimizing the parameters of the student model based on the first multi-granularity distillation loss and the second multi-granularity distillation loss; and performing target detection on surround-view multi-camera images using the obtained target student model. The use of this method can improve the accuracy of target detection.

Description

Technical Field

[0001] The present application relates to the field of image technology, and in particular to a BEV target detection method, device, equipment, storage medium and program product. Background Art

[0002] BEV target detection refers to target detection under the bird's-eye view (BEV) perspective. It has the advantages of greatly reducing the complexity of post-processing and facilitating multimodal information fusion, and has become the mainstream trend in the field of autonomous driving.

[0003] In the traditional BEV target detection scheme, the target detection model is usually trained, and then the trained template detection model is used to transform the image features of the perspective view into the features under the BEV view, and then the target detection is performed. However, when the traditional BEV target detection scheme is used for target detection, the accuracy of target detection is not high. Summary of the invention

[0004] Based on this, it is necessary to provide a BEV target detection method, device, equipment, storage medium and program product to improve the accuracy of target detection in response to the above technical problems.

[0005] In a first aspect, the present application provides a BEV target detection method, the method comprising:

[0006] Performing feature extraction on surround-view multi-camera image samples through an image encoder of a teacher model and an image encoder of a student model to obtain first multi-scale image features of the teacher model and second multi-scale image features of the student model;

[0007] Performing a perspective transformation on the first multi-scale image feature through the perspective transformation network of the teacher model to obtain a first sparse BEV feature; and performing a perspective transformation on the second multi-scale image feature through the perspective transformation network of the student model to obtain a second sparse BEV feature;

[0008] Performing feature extraction on the first sparse BEV feature through the BEV encoder of the teacher model to obtain a first dense BEV feature; and performing feature extraction on the second sparse BEV feature through the BEV encoder of the student model to obtain a second dense BEV feature;

[0009] Constructing a first multi-granularity distillation loss of a perspective view according to the first multi-scale image features and the second multi-scale image features; and constructing a second multi-granularity distillation loss of a BEV perspective according to the first dense BEV features and the second dense BEV features;

[0010] Optimizing the parameters of the student model based on the first multi-granularity distillation loss and the second multi-granularity distillation loss to obtain a target student model;

[0011] The target student model is used to perform target detection on the surround multi-camera image to obtain a detection result.

[0012] In a second aspect, the present application further provides a BEV target detection device, the device comprising:

[0013] A first extraction module is used to extract features from surround-view multi-camera image samples through an image encoder of a teacher model and an image encoder of a student model to obtain first multi-scale image features of the teacher model and second multi-scale image features of the student model;

[0014] A transformation module, configured to perform a perspective transformation on the first multi-scale image feature through the perspective transformation network of the teacher model to obtain a first sparse BEV feature; and perform a perspective transformation on the second multi-scale image feature through the perspective transformation network of the student model to obtain a second sparse BEV feature;

[0015] A second extraction module, configured to extract the first sparse BEV feature through the BEV encoder of the teacher model to obtain a first dense BEV feature; and to extract the second sparse BEV feature through the BEV encoder of the student model to obtain a second dense BEV feature;

[0016] A loss calculation module, configured to construct a first multi-granularity distillation loss of a perspective view according to the first multi-scale image features and the second multi-scale image features; and to construct a second multi-granularity distillation loss of a BEV perspective according to the first dense BEV features and the second dense BEV features;

[0017] an optimization module, configured to optimize the parameters of the student model based on the first multi-granularity distillation loss and the second multi-granularity distillation loss to obtain a target student model;

[0018] The detection module is used to perform target detection on the surround multi-camera image through the target student model to obtain a detection result.

[0019] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the BEV target detection method when executing the computer program.

[0020] In a fourth aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the BEV target detection method are implemented.

[0021] In a fifth aspect, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the BEV target detection method are implemented.

[0022] The above-mentioned BEV target detection method, device, equipment, storage medium and program product extract features of surround multi-camera image samples through the image encoder of the teacher model and the image encoder of the student model to obtain the first multi-scale image features of the teacher model and the second multi-scale image features of the student model; perform perspective transformation on the first multi-scale image features through the perspective transformation network of the teacher model to obtain the first sparse BEV features; and, perform perspective transformation on the second multi-scale image features through the perspective transformation network of the student model to obtain the second sparse BEV features; perform feature extraction on the first sparse BEV features through the BEV encoder of the teacher model to obtain the first dense BEV features; and, perform feature extraction on the second sparse BEV features through the BEV encoder of the student model to obtain to the second dense BEV feature; constructing the first multi-granularity distillation loss of the perspective view according to the first multi-scale image feature and the second multi-scale image feature; and constructing the second multi-granularity distillation loss of the BEV view according to the first dense BEV feature and the second dense BEV feature. Therefore, during training, not only the BEV view is considered for knowledge distillation, but also the knowledge distillation between the perspective view of the image, and the knowledge distillation between multi-granularity features is considered, which can better transfer the capabilities of the teacher model to the student model. Therefore, by optimizing the parameters of the student model based on the first multi-granularity distillation loss and the second multi-granularity distillation loss, a target student model with better performance can be obtained. When the target student model is used to perform target detection on surround multi-camera images, the accuracy of target detection can be effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is an application environment diagram of a BEV target detection method in an embodiment;

[0024] Figure 2 Schematic diagram of a flow chart of a BEV target detection method in one embodiment;

[0025] Figure 3 A schematic diagram of the structure of a teacher model and a student model during a training process in one embodiment;

[0026] Figure 4 A schematic diagram of the structure of a student model after training in one embodiment;

[0027] Figure 5 is a structural block diagram of a BEV target detection device in one embodiment;

[0028] Figure 6 is a structural block diagram of a BEV target detection device in another embodiment;

[0029] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0031] It should be noted that in the following description, the terms "first, second and third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first, second and third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0032] The BEV target detection method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 through a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers.

[0033] It should be noted that the data storage system can store data that needs to be processed by the server 104. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers.

[0034] The terminal 102 may be a motor vehicle (including a fuel vehicle and a new energy vehicle) or an onboard device in a motor vehicle, or may be a smart phone, a tablet computer, a laptop computer, a desktop computer, an Internet of Things device, and a portable wearable device.

[0035] Server 104 can be an independent physical server or a service node in a blockchain system. Each service node in the blockchain system forms a peer-to-peer network. The peer-to-peer protocol is an application layer protocol running on the Transmission Control Protocol (TCP). In addition, server 104 can also be a server cluster composed of multiple physical servers, which can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0036] In one embodiment, Figure 2 As shown, a BEV target detection method is provided, which can be Figure 1 The method is performed by a server or a terminal in the embodiment, or by a server and a terminal in collaboration. Figure 1 The following steps are used as an example to illustrate the terminal execution in:

[0037] S202, extracting features from the surround-view multi-camera image samples through the image encoder of the teacher model and the image encoder of the student model to obtain first multi-scale image features of the teacher model and second multi-scale image features of the student model.

[0038] The teacher model can be a target detection model, such as the YOLO series models, the Faster R-CNN model that combines the region proposal network and the convolutional neural network, the Single Shot MultiBox Detector, the FCOS model based on the full convolution design, and other target detection models. The student model can be a target detection model with fewer parameters and network layers than the teacher model.

[0039] The surround-view multi-camera image sample may be a surround-view multi-camera image (i.e., a surround-view multi-camera view image) used for model training of the student model. The surround-view multi-camera image may be an image captured by multiple cameras with different viewpoints, that is, the above-mentioned multiple cameras may capture images of various directions at the location of the camera, such as capturing images in the four directions of the front, back, left, and right of the location, or even more directions to ensure that the environment around the location is captured; or, the surround-view multi-camera image may also be an image captured by a camera in different viewpoints during rotation along the target axis. In some actual application scenarios, the surround-view multi-camera image may include at least one of a surround-view multi-camera view driving image or a multi-camera view environment image.

[0040] Both the first multi-scale image feature and the second multi-scale image feature may include image features of multiple different scales.

[0041] In one embodiment, before extracting image features, the terminal may also perform image enhancement processing on the surround view multi-camera image samples, such as denoising, increasing brightness and contrast, etc.; then, the enhanced surround view multi-camera image samples are input into the teacher model and the student model respectively, and the image encoder of the teacher model is used to extract features of the enhanced surround view multi-camera image samples to obtain the first multi-scale image features of the teacher model; and the image encoder of the student model is used to extract features of the enhanced surround view multi-camera image samples to obtain the second multi-scale image features of the student model.

[0042] In one embodiment, the image encoder may include a backbone network and a first pyramid network (Feature Pyramid Network, FPN), so the terminal can input surround multi-camera image samples into the teacher model and the student model respectively, perform feature extraction through the backbone network of the image encoder in the teacher model, and obtain the first multi-scale image features of the teacher model; and perform feature extraction through the backbone network of the image encoder in the student model to obtain the second multi-scale image features of the student model.

[0043] S204, performing perspective transformation on the first multi-scale image feature through the perspective transformation network of the teacher model to obtain a first sparse BEV feature; and performing perspective transformation on the second multi-scale image feature through the perspective transformation network of the student model to obtain a second sparse BEV feature.

[0044] The perspective transformation network may be a lift-splat perspective transformation network. The first sparse BEV feature and the second sparse BEV feature may be single-scale sparse BEV features.

[0045] In one embodiment, the terminal can input the first multi-scale image feature into the perspective transformation network of the teacher model, which performs perspective transformation on the first multi-scale image feature to obtain a first sparse BEV feature; and input the second multi-scale image feature into the perspective transformation network of the student model, which performs perspective transformation on the second multi-scale image feature to obtain a second sparse BEV feature.

[0046] In another embodiment, before performing the perspective conversion, the terminal may also first process the target image feature of the first multi-scale image feature through the first pyramid network of the teacher model to obtain the first processed image feature; and, process the target image feature of the second multi-scale image feature through the first pyramid network of the student model to obtain the second processed image feature. After obtaining the first processed image feature and the second processed image feature, the terminal may perform a perspective conversion on the first processed image feature through the perspective conversion network of the teacher model to obtain the first sparse BEV feature; and, perform a perspective conversion on the second processed image feature through the perspective conversion network of the student model to obtain the second sparse BEV feature.

[0047] In one embodiment, the following method can be used to obtain the target image features: the terminal upsamples the image features in the first multi-scale image features whose scale is smaller than the first target scale, and fuses the up-sampling results with the image features of the corresponding scale until the scale of the obtained first fused image features is equal to the first target scale, and the first fused image features are used as the target image features of the first multi-scale image features; and upsamples the image features in the second multi-scale image features whose scale is smaller than the first target scale, and fuses the up-sampling results with the image features of the corresponding scale until the scale of the obtained second fused image features is equal to the first target scale, and the second fused image features are used as the target image features of the second multi-scale image features.

[0048] Before upsampling, the image features may be subjected to convolution channel transformation, thereby changing the number of feature channels so that the number of channels between image features of different scales remains consistent.

[0049] For example, assuming that the first multi-scale image features include scales and The terminal can convert the image features of the scale into The image features are upsampled, and then the upsampled result is combined with The image features of the first multi-scale image feature are fused, and the obtained fused image features are used as the target image features of the first multi-scale image features. The target image features are input into the first pyramid network of the teacher model and the first pyramid network of the student model for feature processing to obtain the first processed image features. In addition, for obtaining the target image features of the second multi-scale image features, the above method can be referred to. Among them, It can be the number of cameras or the number of camera angles. is the height of the image feature, is the width of the image feature, and are the number of feature channels of image features. Among them, the scale of surround multi-camera image samples can be , 3 is the number of image channels. The scales of the first sparse BEV feature and the second sparse BEV feature obtained using the above features can be , The number of feature channels representing the BEV features, and are the height and width of the BEV feature, respectively.

[0050] S206, extracting features from the first sparse BEV feature through the BEV encoder of the teacher model to obtain a first dense BEV feature; and extracting features from the second sparse BEV feature through the BEV encoder of the student model to obtain a second dense BEV feature.

[0051] The first dense BEV feature and the second dense BEV feature may both include dense BEV features of multiple scales.

[0052] In one embodiment, the BEV encoder may include a backbone network and a second pyramid network, so the terminal may extract the first sparse BEV features through the backbone network of the BEV encoder in the teacher model to obtain the first dense BEV features of each scale, such as the scale of and and extracting the second sparse BEV features through the backbone network of the BEV encoder in the student model to obtain the second dense BEV features of each scale, such as the scale of and It should be noted that the above scales are only examples and not exhaustive, and dense BEV features of other scales may also be included.

[0053] in, , , and These are the feature channel numbers of BEV features.

[0054] S208, constructing a first multi-granularity distillation loss of a perspective view according to the first multi-scale image features and the second multi-scale image features; and constructing a second multi-granularity distillation loss of a BEV perspective according to the first dense BEV features and the second dense BEV features.

[0055] The first multi-granularity distillation loss may refer to the distillation loss of image features of different scales, and the second multi-granularity distillation loss may refer to the distillation loss of BEV features of different scales.

[0056] In one embodiment, for the calculation of the first multi-granularity distillation loss, the following steps can be referred to: the terminal uses each scale image feature in the first multi-scale image feature as a reference to align each scale image feature in the second multi-scale image feature on the channel; subtracts each scale image feature in the first multi-scale image feature from each scale image feature of the aligned second multi-scale image feature to obtain the difference of each scale image feature; sums the difference of each scale image feature to obtain the first multi-granularity distillation loss of the perspective view.

[0057] In addition, in the process of calculating the first multi-granularity distillation loss, the terminal can also upsample the image features in the first multi-scale image features whose scales are smaller than the first target scale, and fuse the upsampling results with the image features of the corresponding scales to obtain the first fused image features of each scale; and, upsample the image features in the second multi-scale image features whose scales are smaller than the first target scale, and fuse the upsampling results with the image features of the corresponding scales to obtain the second fused image features of each scale; therefore, the first fused image features of each scale can be subtracted from the aligned second fused image features of each scale, and the image features in the first multi-scale image features whose scales meet the scale conditions can be subtracted from the image features of the corresponding scales of the aligned second multi-scale image features to obtain the difference of the image features of each scale, and then the difference of the image features of each scale is summed to obtain the first multi-granularity distillation loss of the perspective view.

[0058] Before upsampling, the image features may be subjected to convolution channel transformation, thereby changing the number of feature channels so that the number of channels between image features of different scales remains consistent.

[0059] For example, if the first multi-scale image feature includes image features of N scales, where the image feature with the largest scale is an image feature of the Mth scale, the terminal may store images smaller than the Mth scale (such as ) are upsampled, and then the result of each upsampling is fused with the image features of the corresponding scale to obtain the first fused image features of each scale. Similarly, the second fused image features of each scale are obtained in the same manner as the first fused image features. It should be noted that in the above-mentioned upsampling process, the image features of the smallest scale (such as the image features of the Nth scale) are preferentially upsampled and fused, and then the obtained fused image features of the corresponding scale (such as the N-1th scale) are upsampled and fused, and so on, and finally the fused image features of each scale can be obtained. Finally, the first fused image features of each scale are subtracted from the second fused image features of each scale after alignment, and the image features of the Nth scale in the first multi-scale image features are subtracted from the image features of the Nth scale in the second multi-scale image features to obtain the difference of the image features of each scale.

[0060] For the acquisition of fused image features, let’s take an example to illustrate. Assume that the first multi-scale image features include scales and The terminal can convert the image features of the scale into The image features are upsampled, and then the upsampled result is combined with The image features of In addition, for obtaining the second fused image feature of the second multi-scale image feature, the above method can be referred to.

[0061] In one embodiment, the calculation of the second multi-granularity distillation loss may refer to the following steps: the terminal uses the BEV features of each scale in the first dense BEV feature as a reference to align the BEV features of each scale in the second dense BEV feature on the channel; subtracts the BEV features of each scale in the first dense BEV feature from the aligned BEV features of each scale in the second dense BEV feature to obtain the difference of the BEV features of each scale; sums the differences of the BEV features of each scale to obtain the second multi-granularity distillation loss from the BEV perspective.

[0062] In addition, in the process of calculating the first multi-granularity distillation loss, the terminal can also upsample the BEV features in the first dense BEV features whose scales are smaller than the second target scale, and fuse the upsampling results with the BEV features of the corresponding scales to obtain the first fused BEV features of each scale; and, upsample the BEV features in the second dense BEV features whose scales are smaller than the second target scale, and fuse the upsampling results with the BEV features of the corresponding scales to obtain the second fused BEV features of each scale; therefore, the first fused BEV features of each scale are subtracted from the second fused BEV features of each scale after alignment, and the BEV features in the first dense BEV features whose scales meet the scale conditions are subtracted from the BEV features of the corresponding scales of the aligned second dense BEV features to obtain the difference of the BEV features of each scale.

[0063] Among them, before upsampling, the BEV features can be processed by convolution channel transformation, so as to change the number of feature channels and keep the number of channels between BEV features of different scales consistent.

[0064] S210, optimizing parameters of the student model based on the first multi-granularity distillation loss and the second multi-granularity distillation loss to obtain a target student model.

[0065] In one embodiment, the terminal back-propagates the first multi-granularity distillation loss and the second multi-granularity distillation loss in the student model, and during the back-propagation process, optimizes the parameters of the student model until the model converges to obtain a target student model.

[0066] In another embodiment, the terminal may also optimize the parameters of the student model in combination with the BEV feature distillation loss. Specifically: the terminal fine-tunes the first dense BEV feature through the second pyramid network of the teacher model to obtain the first fine-tuned BEV feature; and, fine-tunes the second dense BEV feature through the second pyramid network of the student model to obtain the second fine-tuned BEV feature; constructs the BEV feature distillation loss based on the first fine-tuned BEV feature and the second fine-tuned BEV feature; optimizes the parameters of the student model based on the first multi-granularity distillation loss, the second multi-granularity distillation loss and the BEV feature distillation loss to obtain the target student model.

[0067] Among them, before using the above-mentioned pyramid network for fine-tuning, the BEV features can be first subjected to convolutional feature transformation to achieve fine-tuning of the BEV features to improve the expression ability.

[0068] S212, performing target detection on the surround multi-camera image through the target student model to obtain a detection result.

[0069] Among them, the surround-view multi-camera image can be a surround-view multi-camera image to be detected, such as an image collected by cameras installed in various directions of the vehicle while the vehicle is driving, so that vehicles, pedestrians, obstacles and lane lines on the road can be detected.

[0070] In one embodiment, the terminal can perform feature extraction on the surround multi-camera image through the image encoder of the target student model to obtain the third multi-scale image feature of the target student model; perform perspective transformation on the third multi-scale image feature through the perspective transformation network of the target student model to obtain the third sparse BEV feature; perform feature extraction on the third sparse BEV feature through the BEV encoder of the target student model to obtain the third dense BEV feature; perform detection and processing on the three dense BEV features through the target detection head of the target student model to obtain the target detection result.

[0071] For example, the terminal can perform feature extraction on the surround multi-camera image through the backbone network of the image encoder in the target student model to obtain the third multi-scale image feature of the target student model; perform perspective transformation on the third multi-scale image feature through the perspective transformation network of the target student model to obtain the third sparse BEV feature; perform feature extraction on the third sparse BEV feature through the backbone network of the BEV encoder in the target student model to obtain the third dense BEV feature; perform detection and processing on the three dense BEV features through the target detection head of the target student model to obtain the target detection result.

[0072] In one embodiment, the terminal can process the target image features in the third multi-scale image features through the first pyramid network of the target student model to obtain a third processed image feature; therefore, the third processed image feature can be transformed in perspective through the perspective transformation network of the target student model to obtain a third sparse BEV feature.

[0073] For obtaining the target image features, the following method can be referred to: the terminal can upsample the image features in the third multi-scale image features whose scale is smaller than the first target scale, and fuse the up-sampling results with the image features of the corresponding scales until the scale of the obtained third fused image features is equal to the first target scale, and use the third fused image features as the target image features of the third multi-scale image features.

[0074] In one embodiment, the terminal can fine-tune the third dense BEV feature through the second pyramid network of the target student model to obtain the third fine-tuned BEV feature; therefore, the third dense image feature can be input into the target detection head of the target student model, so that the target detection head of the target student model performs target detection based on the third fine-tuned BEV feature to obtain the target detection result.

[0075] In the above embodiment, feature extraction is performed on the surround multi-camera image samples by the image encoder of the teacher model and the image encoder of the student model to obtain the first multi-scale image features of the teacher model and the second multi-scale image features of the student model; the first multi-scale image features are subjected to perspective transformation by the perspective transformation network of the teacher model to obtain the first sparse BEV features; and, the second multi-scale image features are subjected to perspective transformation by the perspective transformation network of the student model to obtain the second sparse BEV features; the first sparse BEV features are subjected to feature extraction by the BEV encoder of the teacher model to obtain the first dense BEV features; and, the second sparse BEV features are subjected to feature extraction by the BEV encoder of the student model to obtain the second dense BEV features; A first multi-granularity distillation loss of a perspective view is constructed according to the first multi-scale image features and the second multi-scale image features; and a second multi-granularity distillation loss of a BEV view is constructed according to the first dense BEV features and the second dense BEV features. Therefore, during training, not only the BEV view is considered for knowledge distillation, but also the knowledge distillation between the perspective viewpoints of the image and the knowledge distillation between multi-granularity features are considered, which can better transfer the capabilities of the teacher model to the student model. Therefore, by optimizing the parameters of the student model based on the first multi-granularity distillation loss and the second multi-granularity distillation loss, a target student model with better performance can be obtained. When the target student model is used to perform target detection on surround multi-camera images, the accuracy of target detection can be effectively improved.

[0076] In order to understand the solution of this application more clearly, you can combine Figure 3 and Figure 4 The specific contents are as follows:

[0077] For the training process, please refer to Figure 3 , the specific steps are as follows:

[0078] 1. The surround-view multi-camera images are input into the Teacher model. The backbone network of the Teacher model image encoder extracts features from the surround-view multi-camera images to obtain multi-scale image features.

[0079] 2. The Teacher model uses the FPN network to process multi-scale image features and obtain the output features of the FPN network;

[0080] 3. The Teacher model uses the lift-splat perspective transformation network to transform the output features of the FPN network to obtain sparse original BEV features;

[0081] 4. The Teacher model uses the backbone network of the BEV encoder to extract the sparse original BEV features and obtain multi-scale dense BEV features;

[0082] 5. The Teacher model uses the FPN network of the BEV encoder to fine-tune the multi-scale BEV features to obtain the fine-tuned dense BEV features;

[0083] 6. The Teacher model uses the 3D object detection head to detect the fine-tuned dense BEV features and obtain the detection results;

[0084] 7. Input the surround multi-camera image into the Student model, which uses the backbone network of the image encoder to extract features and obtain multi-scale image features;

[0085] 8. The Student model uses the FPN network to process the multi-scale image features and obtain the output features of the FPN network;

[0086] 9. The Student model uses the lift-splat perspective transformation network to transform the output features of the FPN network to obtain sparse original BEV features;

[0087] 10. The Student model uses the backbone network of the BEV encoder to process the sparse original BEV features to obtain multi-scale dense BEV features;

[0088] 11. The Student model uses the FPN network of the BEV encoder to fine-tune the multi-scale dense BEV features to obtain the fine-tuned dense BEV features;

[0089] 12. The Student model uses the 3D target detection head to perform target detection on the fine-tuned dense BEV features to obtain the detection results;

[0090] 13. Construct a multi-granularity distillation loss of perspective view using the multi-camera multi-scale image features obtained in step 1 and the multi-camera multi-scale image features obtained in step 7;

[0091] In one embodiment, L2 loss is constructed between the multi-scale image features of the Teacher model and the Student model at each scale (the multi-scale image features of the Student model are first aligned with the multi-scale image features of the Teacher model on the channel through convolution), and then the L2 losses corresponding to all scales are added together to obtain the multi-granularity distillation loss of the perspective view;

[0092] 14. Construct a multi-granularity distillation loss from a BEV perspective by using the multi-scale BEV features obtained in step 4 and the multi-scale BEV features obtained in step 10;

[0093] In one embodiment, L2 loss is constructed between the dense BEV features of the Teacher model and the Student model at each scale (the multi-scale dense BEV features of the Student model are first aligned on the channel with the dense BEV features of the corresponding scale of the Teacher model through convolution), and then the L2 losses corresponding to all scales are added together to obtain the multi-granularity distillation loss from the BEV perspective.

[0094] 15. Construct BEV feature distillation loss by using the fine-tuned dense BEV features obtained in step 5 and the fine-tuned dense BEV features obtained in step 11;

[0095] 16. The multi-granularity distillation loss of perspective view, the multi-granularity distillation loss of BEV view and the BEV feature distillation loss are added to the model training to obtain a student model for target detection.

[0096] After completing the training of the student model, apply the trained student model to target detection. You can refer to Figure 4 , as follows:

[0097] 1. Input the surround multi-camera images to be detected into the image encoder backbone network to obtain multi-scale image features of multiple cameras;

[0098] 2. The Student model uses the FPN network to process the multi-scale image features and obtain the output features of the FPN network;

[0099] 3. Input the output features of the FPN network into the lift-splat perspective transformation network to obtain sparse original BEV features;

[0100] 4. Input the sparse original BEV features into the backbone network of the BEV encoder to obtain multi-scale dense BEV features;

[0101] 5. Input the multi-scale dense BEV features into the FPN network of the BEV encoder for fine-tuning to obtain the fine-tuned dense BEV features;

[0102] 6. Input the fine-tuned dense BEV features into the 3D object detection head to obtain the detection results.

[0103] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0104] Based on the same inventive concept, the embodiment of the present application also provides a BEV target detection device for implementing the BEV target detection method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more BEV target detection device embodiments provided below can refer to the limitations of the BEV target detection method above, and will not be repeated here.

[0105] In one embodiment, Figure 5 As shown, a BEV target detection device is provided, including: a first extraction module 502, a transformation module 504, a second extraction module 506, a loss calculation module 508, an optimization module 510 and a detection module 512, wherein:

[0106] A first extraction module 502 is used to extract features from the surround multi-camera image samples through an image encoder of the teacher model and an image encoder of the student model to obtain first multi-scale image features of the teacher model and second multi-scale image features of the student model;

[0107] A transformation module 504 is used to perform a perspective transformation on the first multi-scale image feature through a perspective transformation network of a teacher model to obtain a first sparse BEV feature; and perform a perspective transformation on the second multi-scale image feature through a perspective transformation network of a student model to obtain a second sparse BEV feature;

[0108] A second extraction module 506 is used to extract the first sparse BEV feature through the BEV encoder of the teacher model to obtain a first dense BEV feature; and to extract the second sparse BEV feature through the BEV encoder of the student model to obtain a second dense BEV feature;

[0109] A loss calculation module 508 is used to construct a first multi-granularity distillation loss of a perspective view according to the first multi-scale image features and the second multi-scale image features; and to construct a second multi-granularity distillation loss of a BEV perspective according to the first dense BEV features and the second dense BEV features;

[0110] An optimization module 510 is used to optimize the parameters of the student model based on the first multi-granularity distillation loss and the second multi-granularity distillation loss to obtain a target student model;

[0111] The detection module 512 is used to perform target detection on the surround multi-camera images through the target student model to obtain a detection result.

[0112] In one embodiment, if Figure 6 As shown, the device also includes:

[0113] A first processing module 514 is used to process the target image feature of the first multi-scale image feature through the first pyramid network of the teacher model to obtain a first processed image feature; and to process the target image feature of the second multi-scale image feature through the first pyramid network of the student model to obtain a second processed image feature;

[0114] The transformation module 504 is also used to perform perspective transformation on the first processed image feature through the perspective transformation network of the teacher model to obtain a first sparse BEV feature; and to perform perspective transformation on the second processed image feature through the perspective transformation network of the student model to obtain a second sparse BEV feature.

[0115] In one embodiment, if Figure 6 As shown, the device also includes:

[0116] A first fine-tuning module 516 is used to fine-tune the first dense BEV feature through the second pyramid network of the teacher model to obtain a first fine-tuned BEV feature; and fine-tune the second dense BEV feature through the second pyramid network of the student model to obtain a second fine-tuned BEV feature;

[0117] The loss calculation module 508 is further used to construct a BEV feature distillation loss according to the first fine-tuned BEV feature and the second fine-tuned BEV feature;

[0118] The optimization module 510 is also used to optimize the parameters of the student model based on the first multi-granularity distillation loss, the second multi-granularity distillation loss and the BEV feature distillation loss to obtain a target student model.

[0119] In one embodiment, the loss calculation module 508 is further used to align each scale image feature in the second multi-scale image feature on the channel with reference to each scale image feature in the first multi-scale image feature; subtract each scale image feature in the first multi-scale image feature from each scale image feature of the aligned second multi-scale image feature to obtain the difference of each scale image feature; and sum the difference of each scale image feature to obtain the first multi-granularity distillation loss of the perspective view.

[0120] In one embodiment, if Figure 6 As shown, the device also includes:

[0121] A first upsampling module 518 is configured to upsample image features of the first multi-scale image features whose scale is smaller than the first target scale, and fuse the upsampling results with image features of corresponding scales to obtain first fused image features of each scale; and to upsample image features of the second multi-scale image features whose scale is smaller than the first target scale, and fuse the upsampling results with image features of corresponding scales to obtain second fused image features of each scale;

[0122] The loss calculation module 508 is also used to subtract the first fused image features of each scale from the aligned second fused image features of each scale, and to subtract the image features of the scale that meets the scale condition in the first multi-scale image features from the image features of the corresponding scale of the aligned second multi-scale image features.

[0123] In one embodiment, the loss calculation module 508 is also used to align the BEV features of each scale in the second dense BEV feature on the channel with reference to the BEV features of each scale in the first dense BEV feature; subtract the BEV features of each scale in the first dense BEV feature from the BEV features of each scale in the aligned second dense BEV feature to obtain the difference of the BEV features of each scale; and sum the differences of the BEV features of each scale to obtain the second multi-granularity distillation loss from the BEV perspective.

[0124] In one embodiment, if Figure 6 As shown, the device also includes:

[0125] The second upsampling module 520 is used to upsample the BEV features whose scale is smaller than the second target scale in the first dense BEV features, and fuse the upsampled results with the BEV features of the corresponding scales to obtain first fused BEV features of each scale; and upsample the BEV features whose scale is smaller than the second target scale in the second dense BEV features, and fuse the upsampled results with the BEV features of the corresponding scales to obtain second fused BEV features of each scale;

[0126] The loss calculation module 508 is also used to subtract the first fused BEV features of each scale from the aligned second fused BEV features of each scale, and to subtract the BEV features of the scale that meets the scale condition in the first dense BEV features from the BEV features of the corresponding scale of the aligned second dense BEV features.

[0127] In one embodiment, the detection module 512 is further used to perform feature extraction on the surround multi-camera image through the image encoder of the target student model to obtain a third multi-scale image feature of the target student model; perform perspective transformation on the third multi-scale image feature through the perspective transformation network of the target student model to obtain a third sparse BEV feature; perform feature extraction on the third sparse BEV feature through the BEV encoder of the target student model to obtain a third dense BEV feature; perform detection processing on the three dense BEV features through the target detection head of the target student model to obtain a target detection result.

[0128] In one embodiment, if Figure 6 As shown, the device also includes:

[0129] A second processing module 522 is used to process the target image feature in the third multi-scale image feature through the first pyramid network of the target student model to obtain a third processed image feature;

[0130] The transformation module 504 is further used to perform perspective transformation on the third processed image feature through the perspective transformation network of the target student model to obtain a third sparse BEV feature.

[0131] In one embodiment, if Figure 6 As shown, the device also includes:

[0132] A second fine-tuning module 524 is used to fine-tune the third dense BEV feature through the second pyramid network of the target student model to obtain a third fine-tuned BEV feature;

[0133] The detection module 512 is further used to input the third dense image feature into the target detection head of the target student model, so that the target detection head of the target student model performs target detection based on the third fine-tuned BEV feature to obtain a target detection result.

[0134] In the above embodiment, feature extraction is performed on the surround multi-camera image samples by the image encoder of the teacher model and the image encoder of the student model to obtain the first multi-scale image features of the teacher model and the second multi-scale image features of the student model; the first multi-scale image features are subjected to perspective transformation by the perspective transformation network of the teacher model to obtain the first sparse BEV features; and, the second multi-scale image features are subjected to perspective transformation by the perspective transformation network of the student model to obtain the second sparse BEV features; the first sparse BEV features are subjected to feature extraction by the BEV encoder of the teacher model to obtain the first dense BEV features; and, the second sparse BEV features are subjected to feature extraction by the BEV encoder of the student model to obtain the second dense BEV features; A first multi-granularity distillation loss of a perspective view is constructed according to the first multi-scale image features and the second multi-scale image features; and a second multi-granularity distillation loss of a BEV view is constructed according to the first dense BEV features and the second dense BEV features. Therefore, during training, not only the BEV view is considered for knowledge distillation, but also the knowledge distillation between the perspective viewpoints of the image and the knowledge distillation between multi-granularity features are considered, which can better transfer the capabilities of the teacher model to the student model. Therefore, by optimizing the parameters of the student model based on the first multi-granularity distillation loss and the second multi-granularity distillation loss, a target student model with better performance can be obtained. When the target student model is used to perform target detection on surround multi-camera images, the accuracy of target detection can be effectively improved.

[0135] Each module in the above-mentioned BEV target detection device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0136] In one embodiment, a computer device is provided, which may be a server, and includes a processor, a memory, an input / output interface (I / O for short) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a BEV target detection method is implemented.

[0137] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a BEV target detection method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0138] Those skilled in the art will understand that Figure 7The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0139] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above-mentioned BEV target detection method when executing the computer program.

[0140] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned BEV target detection method are implemented.

[0141] In one embodiment, a computer program product is provided, comprising a computer program, which implements the steps of the above-mentioned BEV target detection method when executed by a processor.

[0142] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0143] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0144] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0145] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A BEV target detection method, characterized in that: The method comprises: Performing feature extraction on surround-view multi-camera image samples through an image encoder of a teacher model and an image encoder of a student model to obtain first multi-scale image features of the teacher model and second multi-scale image features of the student model; Processing the target image feature of the first multi-scale image feature through the first pyramid network of the teacher model to obtain a first processed image feature; and processing the target image feature of the second multi-scale image feature through the first pyramid network of the student model to obtain a second processed image feature; Performing a perspective transformation on the first processed image feature through the perspective transformation network of the teacher model to obtain a first sparse BEV feature; and performing a perspective transformation on the second processed image feature through the perspective transformation network of the student model to obtain a second sparse BEV feature; Performing feature extraction on the first sparse BEV feature through the BEV encoder of the teacher model to obtain a first dense BEV feature; and performing feature extraction on the second sparse BEV feature through the BEV encoder of the student model to obtain a second dense BEV feature; Taking each scale image feature in the first multi-scale image feature as a reference, aligning each scale image feature in the second multi-scale image feature on the channel; subtracting each scale image feature in the first multi-scale image feature from each scale image feature of the aligned second multi-scale image feature to obtain the difference of each scale image feature; summing the difference of each scale image feature to obtain a first multi-granularity distillation loss of perspective view; and, taking each scale BEV feature in the first dense BEV feature as a reference, aligning each scale BEV feature in the second dense BEV feature on the channel; subtracting each scale BEV feature in the first dense BEV feature from each scale BEV feature of the aligned second dense BEV feature to obtain the difference of each scale BEV feature; summing the difference of each scale BEV feature to obtain a second multi-granularity distillation loss of BEV perspective; Fine-tuning the first dense BEV feature through the second pyramid network of the teacher model to obtain a first fine-tuned BEV feature; and fine-tuning the second dense BEV feature through the second pyramid network of the student model to obtain a second fine-tuned BEV feature; constructing a BEV feature distillation loss according to the first fine-tuned BEV feature and the second fine-tuned BEV feature; Optimizing the parameters of the student model based on the first multi-granularity distillation loss, the second multi-granularity distillation loss, and the BEV feature distillation loss to obtain a target student model; The target student model is used to perform target detection on the surround multi-camera image to obtain a detection result.

2. The method according to claim 1, characterized in that The method further comprises: Upsampling image features whose scales are smaller than the first target scale in the first multi-scale image features, and fusing the upsampling results with image features of corresponding scales to obtain first fused image features of each scale; and upsampling image features whose scales are smaller than the first target scale in the second multi-scale image features, and fusing the upsampling results with image features of corresponding scales to obtain second fused image features of each of the scales; Subtracting each scale image feature in the first multi-scale image feature from each scale image feature after alignment in the second multi-scale image feature comprises: Subtracting the first fused image features of each scale from the aligned second fused image features of each scale, and subtracting the image features of the scale satisfying the scale condition in the first multi-scale image features from the image features of the corresponding scale of the aligned second multi-scale image features.

3. The method according to claim 1, characterized in that The method further comprises: Upsampling the BEV features whose scale is smaller than the second target scale in the first dense BEV features, and fusing the upsampling results with the BEV features of the corresponding scales to obtain first fused BEV features of each scale; and upsampling the BEV features whose scale is smaller than the second target scale in the second dense BEV features, and fusing the upsampling results with the BEV features of the corresponding scales to obtain second fused BEV features of each scale; Subtracting each scale BEV feature in the first dense BEV feature from each scale BEV feature after alignment of the second dense BEV feature comprises: The first fused BEV features of each scale are subtracted from the aligned second fused BEV features of each scale, and the BEV features of the scales satisfying the scale condition in the first dense BEV features are subtracted from the BEV features of the corresponding scales of the aligned second dense BEV features.

4. The method according to any one of claims 1 to 3, characterized in that: The target student model is used to perform target detection on the surround multi-camera image, and the detection result obtained includes: Performing feature extraction on the surround multi-camera image by using the image encoder of the target student model to obtain a third multi-scale image feature of the target student model; Performing perspective transformation on the third multi-scale image feature through the perspective transformation network of the target student model to obtain a third sparse BEV feature; Performing feature extraction on the third sparse BEV feature through the BEV encoder of the target student model to obtain a third dense BEV feature; The target detection head of the target student model detects and processes the third dense BEV feature to obtain a target detection result.

5. The method according to claim 4, characterized in that The method further comprises: Processing the target image features in the third multi-scale image features through the first pyramid network of the target student model to obtain third processed image features; The step of performing perspective transformation on the third multi-scale image feature through the perspective transformation network of the target student model to obtain a third sparse BEV feature includes: The third processed image feature is subjected to perspective transformation through the perspective transformation network of the target student model to obtain a third sparse BEV feature.

6. The method according to claim 4, characterized in that The method further comprises: Fine-tuning the third dense BEV feature through the second pyramid network of the target student model to obtain a third fine-tuned BEV feature; The target detection head of the target student model detects and processes the third dense BEV feature to obtain a target detection result, including: The third dense BEV feature is input into the target detection head of the target student model, so that the target detection head of the target student model performs target detection based on the third fine-tuned BEV feature to obtain a target detection result.

7. A BEV target detection device, characterized in that: The device comprises: A first extraction module is used to extract features from surround-view multi-camera image samples through an image encoder of a teacher model and an image encoder of a student model to obtain first multi-scale image features of the teacher model and second multi-scale image features of the student model; A first processing module is configured to process a target image feature of the first multi-scale image feature through a first pyramid network of the teacher model to obtain a first processed image feature; and to process a target image feature of the second multi-scale image feature through a first pyramid network of the student model to obtain a second processed image feature; a transformation module, configured to perform a perspective transformation on the first processed image feature through the perspective transformation network of the teacher model to obtain a first sparse BEV feature; and perform a perspective transformation on the second processed image feature through the perspective transformation network of the student model to obtain a second sparse BEV feature; A second extraction module, configured to extract the first sparse BEV feature through the BEV encoder of the teacher model to obtain a first dense BEV feature; and extract the second sparse BEV feature through the BEV encoder of the student model to obtain a second dense BEV feature; A loss calculation module, used to align each scale image feature in the second multi-scale image feature on the channel with reference to each scale image feature in the first multi-scale image feature; subtract each scale image feature in the first multi-scale image feature from each scale image feature of the aligned second multi-scale image feature to obtain the difference of each scale image feature; sum the difference of each scale image feature to obtain a first multi-granularity distillation loss of a perspective view; and, using each scale BEV feature in the first dense BEV feature as a reference, align each scale BEV feature in the second dense BEV feature on the channel; subtract each scale BEV feature in the first dense BEV feature from each scale BEV feature of the aligned second dense BEV feature to obtain the difference of each scale BEV feature; sum the difference of each scale BEV feature to obtain a second multi-granularity distillation loss of a BEV perspective; A first fine-tuning module, configured to fine-tune the first dense BEV feature through a second pyramid network of the teacher model to obtain a first fine-tuning BEV feature; and fine-tune the second dense BEV feature through a second pyramid network of the student model to obtain a second fine-tuning BEV feature; The loss calculation module is further used to construct a BEV feature distillation loss according to the first fine-tuned BEV feature and the second fine-tuned BEV feature; an optimization module, configured to optimize the parameters of the student model based on the first multi-granularity distillation loss, the second multi-granularity distillation loss, and the BEV feature distillation loss to obtain a target student model; The detection module is used to perform target detection on the surround multi-camera image through the target student model to obtain a detection result.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images