Occupancy grid prediction method, device, equipment, storage medium and program product

Abstract

The present application relates to an occupancy grid prediction method, device, equipment, storage medium and program product. The method comprises: extracting first image features of different scales from surround multi-camera images; performing dimensionality reduction processing on the first image features of each scale respectively to obtain one-dimensional image features of each scale; performing BEV feature conversion on the one-dimensional image features of each scale respectively through a multi-layer perceptron to obtain one-dimensional BEV features of each scale; performing dimensionality increase processing on the one-dimensional BEV features of each scale respectively to obtain two-dimensional BEV features of each scale; fusing the two-dimensional BEV features of each scale to obtain fused BEV features; performing occupancy grid prediction based on the fused BEV features to obtain a first prediction result. The use of this method can effectively improve the efficiency of occupancy grid prediction.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an occupancy grid prediction method, device, equipment, storage medium and program product. Background Art

[0002] Occupancy grid prediction is a perception technology in autonomous driving. It mainly divides the world into tiny (or ultra-tiny) cubes or voxels and predicts whether each cube or voxel is free or occupied, thereby controlling the vehicle.

[0003] Traditional occupancy grid prediction methods usually use the lift-splat method to perform perspective transformation during the prediction process. However, when performing perspective transformation, operators such as cumsum, gather / gatherND, and scatherND are used. Therefore, a neural network processing unit (NPU) is required to support these operators. However, most NPUs cannot fully support these operators, and even if they support these operators, the execution speed is very slow, which greatly affects the efficiency of occupancy grid prediction. Summary of the invention

[0004] Based on this, it is necessary to provide an occupancy grid prediction method, device, equipment, storage medium and program product to address the above technical problems, which can effectively improve the efficiency of occupancy grid prediction.

[0005] In a first aspect, the present application provides an occupancy grid prediction method, the method comprising:

[0006] Extracting first image features of different scales from surround-view multi-camera images;

[0007] Performing dimensionality reduction processing on the first image features of each scale respectively to obtain one-dimensional image features of each scale;

[0008] Performing BEV feature conversion on the one-dimensional image features of each scale by a multi-layer perceptron to obtain one-dimensional BEV features of each scale;

[0009] Performing dimensionality-upgrading processing on the one-dimensional BEV features of each scale to obtain the two-dimensional BEV features of each scale;

[0010] Fusing the two-dimensional BEV features of each scale to obtain a fused BEV feature;

[0011] Occupancy grid prediction is performed based on the fused BEV features to obtain a first prediction result.

[0012] In one embodiment, the first image features of each scale are subjected to dimensionality reduction processing to obtain one-dimensional image features of each scale, including:

[0013] Performing morphological transformation and dimension rearrangement on the first-scale image features to obtain one-dimensional image features of the first scale;

[0014] The second-scale image features are morphologically transformed and dimensionally rearranged to obtain one-dimensional image features of the second scale.

[0015] In one embodiment, extracting target BEV features from surround-view multi-camera image samples using a teacher model includes:

[0016] Extracting the first image feature from the surround multi-camera image through the backbone network of the teacher model;

[0017] Performing perspective transformation on the first image feature through the perspective transformation network of the teacher model to obtain a preliminary BEV feature;

[0018] The preliminary BEV features are fine-tuned by the BEV encoder of the teacher model to obtain the target BEV features.

[0019] In one embodiment, performing occupancy grid prediction based on the target BEV feature to obtain a second prediction result includes:

[0020] Performing occupancy grid prediction on the target BEV feature through the occupancy prediction head of the teacher model to obtain a second intermediate result of occupancy grid prediction; performing height conversion on the first intermediate result to obtain a second prediction result having length, width and height;

[0021] The occupancy grid prediction is performed based on the fused BEV features of the surround view multi-camera image samples to obtain a third prediction result, including:

[0022] The occupancy grid is predicted by the fused BEV features of the surround view multi-camera image samples through the occupancy prediction head of the student model to obtain a third intermediate result of the occupancy grid prediction; the second intermediate result is height-converted to obtain a third prediction result with length, width and height.

[0023] In a second aspect, the present application further provides an occupancy grid prediction device, the device comprising:

[0024] An extraction module, used for extracting first image features of different scales from the surround multi-camera images;

[0025] A dimension reduction module, used for performing dimension reduction processing on the first image features of each scale respectively to obtain one-dimensional image features of each scale;

[0026] A feature conversion module, used for performing BEV feature conversion on the one-dimensional image features of each scale through a multi-layer perceptron to obtain one-dimensional BEV features of each scale;

[0027] A dimension-increasing module is used to perform dimension-increasing processing on the one-dimensional BEV features of each scale to obtain the two-dimensional BEV features of each scale;

[0028] A fusion module, used for fusing the two-dimensional BEV features of each scale to obtain a fused BEV feature;

[0029] A prediction module is used to perform occupancy grid prediction based on the fused BEV feature to obtain a first prediction result.

[0030] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the occupancy grid prediction method when executing the computer program.

[0031] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the occupancy grid prediction method are implemented.

[0032] In a fifth aspect, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the occupancy grid prediction method are implemented.

[0033] The above-mentioned occupancy grid prediction method, device, computer equipment, storage medium and computer program product extract first image features of different scales from the surround multi-camera image; perform dimensionality reduction processing on the first image features of each scale to obtain one-dimensional image features of each scale; perform BEV feature conversion on the one-dimensional image features of each scale through a multi-layer perceptron to obtain one-dimensional BEV features of each scale; perform dimensionality increase processing on the one-dimensional BEV features of each scale to obtain two-dimensional BEV features of each scale; fuse the two-dimensional BEV features of each scale to obtain fused BEV features; perform occupancy grid prediction based on the fused BEV features, The first prediction result is obtained, and the multi-layer perceptron near-perspective transformation uses a fully connected layer, which is a linear operator. The neural network processing units also support linear operators and have fast execution speed, which improves the conversion efficiency of the bird's-eye view and is beneficial to improving the efficiency of occupied grid prediction. In addition, before the BEV feature conversion, the first image features of each scale are reduced in dimension, and after the BEV feature conversion, the one-dimensional BEV features of each scale are increased in dimension, and the obtained two-dimensional BEV features of each scale are fused, and the fused BEV features are used for occupied grid prediction, which can effectively improve the accuracy of the prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] FIG1 is a diagram of an application environment of an occupancy grid prediction method according to an embodiment;

[0035] FIG2 is a schematic diagram of a flow chart of an occupancy grid prediction method in one embodiment;

[0036] FIG3 is a schematic diagram of the structure of a trained student model in one embodiment;

[0037] FIG4 is a schematic diagram of the structure of a teacher model and a student model during training in one embodiment;

[0038] FIG5 is a block diagram of an occupancy grid prediction device in one embodiment;

[0039] FIG6 is a structural block diagram of an occupied grid prediction device in another embodiment;

[0040] FIG. 7 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0042] It should be noted that in the following description, the terms "first, second and third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first, second and third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0043] The occupancy grid prediction method provided in the embodiment of the present application can be applied in the application environment shown in FIG1. ​​In which, the terminal 102 communicates with the server 104 through a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers.

[0044] It should be noted that the data storage system can store data that needs to be processed by the server 104. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers.

[0045] The terminal 102 may be a motor vehicle (including a fuel vehicle and a new energy vehicle) or an onboard device in a motor vehicle, or may be a smart phone, a tablet computer, a laptop computer, a desktop computer, an Internet of Things device, and a portable wearable device.

[0046] Server 104 can be an independent physical server or a service node in a blockchain system. Each service node in the blockchain system forms a peer-to-peer network. The peer-to-peer protocol is an application layer protocol running on the Transmission Control Protocol (TCP). In addition, server 104 can also be a server cluster composed of multiple physical servers, which can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0047] In one embodiment, as shown in FIG2 , an occupancy grid prediction method is provided. The method may be executed by the server or the terminal in FIG1 , or by the server and the terminal in collaboration. The method is described by taking the terminal in FIG1 as an example, and includes the following steps:

[0048] S202: extract first image features of different scales from the surround multi-camera images.

[0049] The surround multi-camera image may be a surround multi-camera perspective image, which is an image captured by multiple cameras with different perspectives, that is, the multiple cameras mentioned above can capture images of all directions at the location of the camera, such as capturing images in the four directions of the front, back, left, and right of the location, or even more directions to ensure that the environment around the location is captured; or, the surround multi-camera image may also be an image captured by a camera in different perspectives during the rotation along the target axis. In some practical application scenarios, the surround multi-camera image may include at least one of a surround multi-camera perspective driving image or a multi-camera perspective environment image.

[0050] The first image features of different scales may be image features obtained by downsampling at different times, such as 16-fold downsampling image features and 32-fold downsampling image features.

[0051] In one embodiment, the occupancy grid prediction method of the present application can be applied to a target student model, so the terminal can input the surround view multi-camera image into the target student model, perform feature extraction on the surround view multi-camera image through the backbone network in the target student model, and obtain the first image features of each scale.

[0052] In one embodiment, the image features of different scales include first-scale image features and second-scale image features; therefore, the terminal can extract the first-scale image features and the second-scale image features from the surround multi-camera images through the backbone network in the target learning model.

[0053] The scale of the first-scale image feature is greater than the scale of the second-scale image feature.

[0054] For example, as shown in FIG3, the terminal will scale The surround multi-camera images are input to the target student model. The backbone network of the target student model can extract image features of corresponding sizes from the surround multi-camera images, such as The image features and The image features of , and Represents a dimension respectively, where It can be the number of cameras or the number of camera angles. and are the height and width of the surround view multi-camera image, 3 represents the number of image channels, Indicates the number of feature channels.

[0055] In another embodiment, before extracting image features, the terminal may also perform image enhancement processing on the surround view multi-camera image, such as denoising, increasing brightness and contrast, etc.; then, the enhanced surround view multi-camera image is input into the target student model, and features of the enhanced surround view multi-camera image are extracted through the backbone network in the target student model to obtain first image features at each scale.

[0056] S204, performing dimensionality reduction processing on the first image features of each scale respectively to obtain one-dimensional image features of each scale.

[0057] In one embodiment, the terminal may perform morphological transformation and dimension rearrangement on the first-scale image features to obtain one-dimensional image features of the first scale; and perform morphological transformation and dimension rearrangement on the second-scale image features to obtain one-dimensional image features of the second scale.

[0058] Among them, the morphological transformation (reshape) can be to transform the first-scale image features into image features of other forms. For example, the first-scale image features have m dimensions (such as 2×3×4), and they are transformed into n dimensions (such as 6×4), where m is greater than n and both are positive integers.

[0059] Permute means to change the position of a dimension. In the process of dimensionality reduction, the morphological transformation and dimensionality permute can be performed in any order, that is, the morphological transformation can be performed before or after the dimensionality permute.

[0060] S206, performing BEV feature conversion on the one-dimensional image features of each scale through a multi-layer perceptron to obtain one-dimensional BEV features of each scale.

[0061] Among them, the multi-layer perceptron can be composed of fully connected layers, and the multi-layer perceptron can be used to convert one-dimensional image features of each scale into one-dimensional BEV features of each scale, thereby realizing the transformation from perspective view to bird's eye view (BEV) view.

[0062] In one embodiment, the one-dimensional image features of each scale include one-dimensional image features of a first scale and one-dimensional image features of a second scale; therefore, the one-dimensional image features of the first scale are converted into BEV features by the multi-layer perceptron of the first branch in the target learning model to obtain one-dimensional first BEV features; the one-dimensional image features of the second scale are converted into BEV features by the multi-layer perceptron of the second branch in the target learning model to obtain one-dimensional second BEV features.

[0063] S208, performing dimensionality upscaling processing on the one-dimensional BEV features of each scale to obtain two-dimensional BEV features of each scale.

[0064] In one embodiment, the terminal may use morphological transformation to perform dimension-up processing on the one-dimensional BEV features of each scale to obtain two-dimensional BEV features of each scale. The morphological transformation in the dimension-up process may be the inverse operation of the morphological transformation in the dimension-down process.

[0065] S210, fusing the two-dimensional BEV features of each scale to obtain a fused BEV feature.

[0066] In one embodiment, the two-dimensional BEV features at each scale include two-dimensional BEV features corresponding to the first-scale image features and two-dimensional BEV features corresponding to the second-scale image features, and the method further includes:

[0067] Channel transformation is performed on the two-dimensional BEV features corresponding to the first-scale image features and / or the two-dimensional BEV features corresponding to the second-scale image features to obtain the channel-transformed BEV features corresponding to the first-scale image features and / or the channel-transformed BEV features corresponding to the second-scale image features; therefore, when performing feature fusion, the channel-transformed BEV features corresponding to the first-scale image features and the two-dimensional BEV features corresponding to the second-scale image features can be fused, or the two-dimensional BEV features corresponding to the first-scale image features and the channel-transformed BEV features corresponding to the second-scale image features can be fused, or the channel-transformed BEV features corresponding to the first-scale image features and the channel-transformed BEV features corresponding to the second-scale image features can be fused, or the channel-transformed BEV features corresponding to the first-scale image features and the channel-transformed BEV features corresponding to the second-scale image features can be fused to obtain the fused BEV features.

[0068] Among them, channel transformation can be achieved through a channel transformation network, such as through convolution and residual networks.

[0069] In addition, in addition to the BEV features used for occupancy grid prediction obtained by the above process, other linear operators can also be used for perspective conversion. The specific process is as follows: in the process of extracting the first image features, the terminal can also extract depth features from the surround view multi-camera image; for the first image features and depth features of the surround view multi-camera image, key information is extracted in the horizontal and vertical directions respectively to obtain key image features and key depth features in the horizontal direction and key image features and key depth features in the vertical direction; matrix operations are performed on the key image features and key depth features in the horizontal direction to obtain first bird's-eye view features; matrix operations are performed on the key image features and key depth features in the vertical direction to obtain second bird's-eye view features; feature fusion is performed on the first bird's-eye view features and the second bird's-eye view features to obtain the bird's-eye view features of the surround view multi-camera image.

[0070] In one embodiment, the terminal can perform matrix multiplication on the key image features and key depth features in the horizontal direction to obtain horizontal dimension-added features; perform matrix multiplication on the horizontal dimension-added features and the horizontal annular matrix to obtain horizontal intermediate features; and perform element-by-element multiplication on the elements in the horizontal intermediate features and the elements in the corresponding positions in the horizontal radial matrix to obtain the first bird's-eye view features.

[0071] In one embodiment, the terminal can perform matrix multiplication on the key image features and key depth features in the vertical direction to obtain the vertical dimension-upgraded features; perform matrix multiplication on the vertical dimension-upgraded features and the vertical circumferential matrix to obtain the vertical intermediate features; perform arithmetic operations on the elements in the vertical intermediate features and the elements in the corresponding positions in the vertical radial matrix to obtain the second bird's-eye view features.

[0072] In one embodiment, the terminal may perform arithmetic operations on elements in the first bird's-eye view feature and elements at corresponding positions in the second bird's-eye view feature to obtain the bird's-eye view features of the surround multi-camera image.

[0073] In one embodiment, for obtaining key image features and key depth features, the following steps may be specifically referred to: the terminal may compress the first image features of the surround view multi-camera image in the horizontal direction, and select feature values ​​that meet preset conditions in the horizontal direction during the compression process to obtain key image features in the horizontal direction; and, compress the first image features of the surround view multi-camera image in the vertical direction, and select feature values ​​that meet preset conditions in the vertical direction during the compression process to obtain key image features in the vertical direction; predict the compression weights of each element of the first image features of the surround view multi-camera image in the horizontal direction, and the compression weights of each element in the vertical direction; based on the compression weights of each element, perform weighted processing on the depth features of the surround view multi-camera image in the horizontal direction and the vertical direction to obtain key depth features in the horizontal direction and key depth features in the vertical direction.

[0074] S212, performing occupancy grid prediction based on the fused BEV features to obtain a first prediction result.

[0075] The first prediction result includes prediction results of length, width and height.

[0076] In one embodiment, the terminal may first fine-tune the fused BEV features to obtain the fine-tuned BEV features; then, perform occupancy grid prediction on the fused BEV features through the occupancy prediction head of the target student model to obtain a first intermediate result of the occupancy grid prediction; finally, perform height conversion on the first intermediate result to obtain a first prediction result having length, width and height, such as performing a channel-to-height (Channel2Height) operation on the first intermediate result to obtain a first prediction result having length, width and height.

[0077] For the fine-tuning process, you can refer to the following steps: the terminal can upsample and fine-tune the fused BEV features, such as upsampling and fine-tuning the features through upsampling and convolutional networks, to obtain fine-tuned BEV features.

[0078] After obtaining the first prediction result including the length, width and height, the first prediction result can be used to control the direction and speed of the vehicle.

[0079] In the above embodiment, first image features of different scales are extracted from the surround multi-camera image; dimensionality reduction processing is performed on the first image features of each scale to obtain one-dimensional image features of each scale; BEV feature conversion is performed on the one-dimensional image features of each scale through a multi-layer perceptron to obtain one-dimensional BEV features of each scale; the one-dimensional BEV features of each scale are respectively dimensionalized to obtain two-dimensional BEV features of each scale; the two-dimensional BEV features of each scale are fused to obtain fused BEV features; occupancy grid prediction is performed based on the fused BEV features to obtain a first prediction result, and the above multi-camera image features are respectively converted to BEV features by a multi-layer perceptron to obtain one-dimensional BEV features of each scale; the one-dimensional BEV features of each scale are respectively dimensionalized to obtain two-dimensional BEV features of each scale; the two-dimensional BEV features of each scale are fused to obtain fused BEV features; occupancy grid prediction is performed based on the fused The layer perceptron near-perspective transformation uses a fully connected layer, which is a linear operator. The neural network processing units also support linear operators and have fast execution speed, which improves the conversion efficiency of the bird's-eye view and is beneficial to improving the efficiency of occupied grid prediction. In addition, before the BEV feature conversion, the first image features of each scale are reduced in dimension, and after the BEV feature conversion, the one-dimensional BEV features of each scale are increased in dimension, and the obtained two-dimensional BEV features of each scale are fused. The fused BEV features are used for occupied grid prediction, which can effectively improve the accuracy of the prediction.

[0080] In one embodiment, before performing occupancy grid prediction, the terminal may use the teacher model to train the student model. The training process may refer to the following steps: the terminal extracts the target BEV feature from the surround view multi-camera image sample through the teacher model, and performs occupancy grid prediction based on the target BEV feature to obtain a second prediction result; extracts second image features of different scales from the surround view multi-camera image sample through the student model, performs dimensionality reduction processing on the second image features of each scale to obtain one-dimensional image features of each scale of the surround view multi-camera image sample, and performs BEV feature conversion on the one-dimensional image features of each scale based on the multi-layer perceptron to obtain one-dimensional image features of each scale of the surround view multi-camera image sample. The one-dimensional BEV features of the surround view multi-camera image samples at each scale are respectively upgraded to obtain the two-dimensional BEV features of the surround view multi-camera image samples at each scale, the two-dimensional BEV features of each scale are fused to obtain the fused BEV features of the surround view multi-camera image samples, and the occupancy grid is predicted based on the fused BEV features of the surround view multi-camera image samples to obtain a third prediction result; the BEV feature distillation loss is determined according to the target BEV features and the fused BEV features of the surround view multi-camera image samples, and the occupancy prediction distillation loss is determined according to the second prediction result and the third prediction result; the student model is optimized based on the BEV feature distillation loss and the occupancy prediction distillation loss to obtain the target student model.

[0081] The teacher model can be a target detection model, such as the YOLO series models, the Faster R-CNN model that combines the region proposal network and the convolutional neural network, the Single Shot MultiBox Detector, the FCOS model based on the full convolution design, and other target detection models. The student model can be a target detection model with fewer parameters and network layers than the teacher model.

[0082] The surround-view multi-camera image sample may be a surround-view multi-camera image used for model training of the student model. The surround-view multi-camera image may be an image captured by multiple cameras with different perspectives, that is, the multiple cameras mentioned above may capture images of various directions at the location of the camera, such as capturing images in the four directions of the front, back, left, and right of the location, or even more directions to ensure that the environment around the location is captured; or, the surround-view multi-camera image may also be an image captured by a camera in different perspectives during rotation along the target axis. In some practical application scenarios, the surround-view multi-camera image may include at least one of a surround-view multi-camera perspective driving image or a multi-camera perspective environment image.

[0083] In one embodiment, the terminal can extract the first image feature from the surround multi-camera image through the backbone network of the teacher model; perform perspective transformation on the first image feature through the perspective transformation network of the teacher model to obtain a preliminary BEV feature; and fine-tune the preliminary BEV feature through the BEV encoder of the teacher model to obtain a target BEV feature.

[0084] In one embodiment, the terminal can perform occupancy grid prediction on the target BEV feature through the occupancy prediction head of the teacher model to obtain a second intermediate result of occupancy grid prediction; perform height conversion on the first intermediate result to obtain a second prediction result with length, width and height; therefore, perform occupancy grid prediction on the fused BEV features of the surround multi-camera image samples through the occupancy prediction head of the student model to obtain a third intermediate result of occupancy grid prediction; perform height conversion on the second intermediate result to obtain a third prediction result with length, width and height.

[0085] The specific processing process in the training process may refer to the processing process in the above application process. For example, the specific process of extracting the second image feature may refer to the process of extracting the first image feature in the above application process.

[0086] In the above embodiment, the target BEV features are extracted from the surround view multi-camera image samples through the teacher model, and the occupancy grid prediction is performed based on the target BEV features to obtain the second prediction result; the second image features of different scales are extracted from the surround view multi-camera image samples through the student model, and the second image features of each scale are respectively subjected to dimensionality reduction processing to obtain the one-dimensional image features of each scale of the surround view multi-camera image samples, and the one-dimensional image features of each scale are respectively subjected to BEV feature conversion based on the multi-layer perceptron to obtain the one-dimensional BEV features of each scale of the surround view multi-camera image samples, and the one-dimensional image features of each scale of the surround view multi-camera image samples are respectively subjected to dimensionality reduction processing to obtain the one-dimensional BEV features of each scale of the surround view multi-camera image samples. The 2D BEV features are processed by dimensionality upgrade to obtain the 2D BEV features of each scale of the surround view multi-camera image samples, and the fused BEV features of the surround view multi-camera image samples are obtained by fusing the 2D BEV features of each scale. The occupancy grid is predicted based on the fused BEV features of the surround view multi-camera image samples to obtain the third prediction result; the BEV feature distillation loss is determined according to the target BEV features and the fused BEV features of the surround view multi-camera image samples, and the occupancy prediction distillation loss is determined according to the second prediction result and the third prediction result; the student model is optimized based on the BEV feature distillation loss and the occupancy prediction distillation loss to obtain the target student model. The BEV feature distillation and occupancy grid prediction distillation are introduced to ensure the expression ability of the model and the quality of the occupancy grid prediction results, so that the obtained target student model has better expression ability and improves the prediction accuracy when performing occupancy grid prediction; in addition, the fully connected layer is used to realize the BEV perspective transformation in the occupancy grid network, which improves the deployment timeliness requirements and friendliness of the model.

[0087] In order to more intuitively and clearly understand the solution of the present application, further description is given here in conjunction with FIG. 3 and FIG. 4, as follows:

[0088] For the training process, please refer to Figure 3, which is as follows:

[0089] 1. Input the surround multi-camera images used for training into the backbone network of the Teacher model to obtain image features.

[0090] The surround view multi-camera image is a surround view multi-camera image used for training, and the scale can be .

[0091] 2. The image features of the Teacher model are input into the lift-splat perspective transformation network to obtain preliminary BEV features.

[0092] 3. The preliminary BEV features are input into the BEV encoder to obtain the fine-tuned BEV features.

[0093] Among them, the scale of the fine-tuned BEV feature can be , among which , and are the number of feature channels, height and width of BEV features respectively.

[0094] 4. The fine-tuned BEV features are input into the occupancy prediction head to obtain the intermediate result of occupancy prediction.

[0095] The scale of the intermediate result can be , where Z represents the spatial height of the BEV feature space.

[0096] 5. The intermediate result of occupancy prediction is operated through Channel2Height to obtain the final prediction result with length, width and height.

[0097] 6. Input the surround multi-camera images used for training into the backbone of the Student model to obtain image feature 1 downsampled by 16 times and image feature 2 downsampled by 32 times.

[0098] 7. After the 16-fold downsampled image feature 1 undergoes morphological transformation (reshape) and dimensional permute, a 16-fold downsampled one-dimensional image feature 1 is obtained.

[0099] 8. The one-dimensional image feature 1 downsampled 16 times is passed through a multi-layer perceptron to obtain the one-dimensional BEV feature 1.

[0100] 9. The one-dimensional BEV feature 1 downsampled 16 times is reshaped to obtain the two-dimensional BEV feature 1.

[0101] The two-dimensional BEV feature 1 downsampled by 10 and 16 times is passed through convolution and residual network to obtain the BEV feature 1 after channel transformation.

[0102] 11. For the 32-fold downsampled image feature 2 of the Student model, refer to steps 7 to 10 to obtain the BEV feature 2 after channel transformation.

[0103] 12. The BEV feature 1 after channel transformation and the BEV feature 2 after channel transformation are concatenated in the channel dimension to obtain the fused BEV feature.

[0104] 13. The fused BEV features are upsampled and processed through a convolutional network to obtain the final BEV features (also called fine-tuned BEV features).

[0105] 14. The final BEV feature is input into the occupancy prediction head to obtain the intermediate result of occupancy prediction.

[0106] 15. The intermediate result of occupancy prediction is converted to height through channel conversion (Channel2Height operation) to obtain the final prediction result with length, width and height.

[0107] 16. Use the fine-tuned BEV features of the Teacher model and the final BEV features of the Student model to construct the BEV feature distillation loss and add it to the training.

[0108] 17. Use the final prediction results of the Teacher model and the Student model to construct the occupancy grid prediction distillation loss and add it to the training.

[0109] For the application process (i.e., the process of occupancy grid prediction), please refer to Figure 4, which is as follows:

[0110] 1. Input the surround multi-camera image to be detected into the backbone of the trained Student model to obtain image feature 1 downsampled by 16 times and image feature 2 downsampled by 32 times.

[0111] The surround multi-camera image is the surround multi-camera image to be detected, and the scale can be .

[0112] 2. After reshape and permute, the image feature 1 downsampled 16 times is obtained as the one-dimensional image feature 1 downsampled 16 times.

[0113] 3. The one-dimensional image feature 1 downsampled 16 times is passed through a multi-layer perceptron to obtain the one-dimensional BEV feature 1.

[0114] 4. The one-dimensional BEV feature 1 downsampled 16 times is reshaped to obtain the two-dimensional BEV feature 1.

[0115] 5. The 16-fold downsampled two-dimensional BEV feature 1 is subjected to convolution and residual network to obtain the channel-transformed BEV feature 1.

[0116] 6. For the 32-fold downsampled image feature 2 of the Student model, refer to steps 7 to 10 to obtain the BEV feature 2 after channel transformation.

[0117] 7. The BEV feature 1 after channel transformation and the BEV feature 2 after channel transformation in the Student model are concatenated in the channel dimension to obtain the fused BEV feature.

[0118] 8. The fused BEV features are upsampled and processed through a convolutional network to obtain the final BEV features (i.e., the fine-tuned BEV features).

[0119] 9. The final BEV features of the Student model are input into the occupancy prediction head to obtain the intermediate results of occupancy prediction.

[0120] 10. The intermediate result of occupancy prediction is operated through Channel2Height to obtain the final prediction result with length, width and height.

[0121] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0122] Based on the same inventive concept, the embodiment of the present application also provides an occupied grid prediction device for implementing the occupied grid prediction method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more occupied grid prediction device embodiments provided below can refer to the limitations of the occupied grid prediction method above, and will not be repeated here.

[0123] In one embodiment, as shown in FIG5 , an occupancy grid prediction device is provided, comprising: an extraction module 502, a dimensionality reduction module 504, a feature conversion module 506, a dimensionality increase module 508, a fusion module 510 and a prediction module 512, wherein:

[0124] An extraction module 502, configured to extract first image features of different scales from the surround multi-camera images;

[0125] A dimension reduction module 504 is used to perform dimension reduction processing on the first image features of each scale respectively to obtain a one-dimensional image feature of each scale;

[0126] A feature conversion module 506 is used to perform BEV feature conversion on the one-dimensional image features of each scale through a multi-layer perceptron to obtain one-dimensional BEV features of each scale;

[0127] A dimension-increasing module 508 is used to perform dimension-increasing processing on the one-dimensional BEV features of each scale to obtain two-dimensional BEV features of each scale;

[0128] A fusion module 510 is used to fuse the two-dimensional BEV features of each scale to obtain a fused BEV feature;

[0129] The prediction module 512 is used to perform occupancy grid prediction based on the fused BEV features to obtain a first prediction result.

[0130] In one embodiment, the image features of different scales include first scale image features and second scale image features;

[0131] The extraction module 502 is further used to extract the first scale image features and the second scale image features from the surround view multi-camera images through the backbone network in the target learning model;

[0132] The scale of the first-scale image feature is greater than the scale of the second-scale image feature.

[0133] In one embodiment, the dimension reduction module 504 is further used to perform morphological transformation and dimension rearrangement on the first-scale image features to obtain one-dimensional image features of the first scale; and perform morphological transformation and dimension rearrangement on the second-scale image features to obtain one-dimensional image features of the second scale.

[0134] In one embodiment, the one-dimensional image features at each scale include one-dimensional image features at a first scale and one-dimensional image features at a second scale;

[0135] The feature conversion module 506 is also used to perform BEV feature conversion on the one-dimensional image features of the first scale through the multi-layer perceptron of the first branch in the target learning model to obtain a one-dimensional first BEV feature; and perform BEV feature conversion on the one-dimensional image features of the second scale through the multi-layer perceptron of the second branch in the target learning model to obtain a one-dimensional second BEV feature.

[0136] In one embodiment, the two-dimensional BEV features at each scale include a two-dimensional BEV feature corresponding to a first-scale image feature and a two-dimensional BEV feature corresponding to a second-scale image feature, and the device further includes:

[0137] A channel transformation module is used to perform channel transformation on the two-dimensional BEV feature corresponding to the first scale image feature and / or the two-dimensional BEV feature corresponding to the second scale image feature to obtain a channel-transformed BEV feature corresponding to the first scale image feature and / or a channel-transformed BEV feature corresponding to the second scale image feature;

[0138] The fusion module 510 is also used to fuse the channel-transformed BEV features corresponding to the first-scale image features and the two-dimensional BEV features corresponding to the second-scale image features, or to fuse the two-dimensional BEV features corresponding to the first-scale image features and the channel-transformed BEV features corresponding to the second-scale image features, or to fuse the channel-transformed BEV features corresponding to the first-scale image features and the channel-transformed BEV features corresponding to the second-scale image features, to obtain fused BEV features.

[0139] In one embodiment, the prediction module 512 is also used to fine-tune the fused BEV features to obtain the fine-tuned BEV features; perform occupancy grid prediction on the fused BEV features through the occupancy prediction head of the target student model to obtain a first intermediate result of the occupancy grid prediction; perform height conversion on the first intermediate result to obtain a first prediction result with length, width and height.

[0140] In the above embodiment, first image features of different scales are extracted from the surround multi-camera image; dimensionality reduction processing is performed on the first image features of each scale to obtain one-dimensional image features of each scale; BEV feature conversion is performed on the one-dimensional image features of each scale through a multi-layer perceptron to obtain one-dimensional BEV features of each scale; the one-dimensional BEV features of each scale are respectively dimensionalized to obtain two-dimensional BEV features of each scale; the two-dimensional BEV features of each scale are fused to obtain fused BEV features; occupancy grid prediction is performed based on the fused BEV features to obtain a first prediction result, and the above multi-camera image features are respectively converted to BEV features by a multi-layer perceptron to obtain one-dimensional BEV features of each scale; the one-dimensional BEV features of each scale are respectively dimensionalized to obtain two-dimensional BEV features of each scale; the two-dimensional BEV features of each scale are fused to obtain fused BEV features; occupancy grid prediction is performed based on the fused The layer perceptron near-perspective transformation uses a fully connected layer, which is a linear operator. The neural network processing units also support linear operators and have fast execution speed, which improves the conversion efficiency of the bird's-eye view and is beneficial to improving the efficiency of occupied grid prediction. In addition, before the BEV feature conversion, the first image features of each scale are reduced in dimension, and after the BEV feature conversion, the one-dimensional BEV features of each scale are increased in dimension, and the obtained two-dimensional BEV features of each scale are fused. The fused BEV features are used for occupied grid prediction, which can effectively improve the accuracy of the prediction.

[0141] In one embodiment, the device is applied to a target student model, as shown in FIG6 , and the device further includes:

[0142] The training module 514 is used to extract the target BEV features from the surround view multi-camera image samples through the teacher model, and perform occupancy grid prediction based on the target BEV features to obtain a second prediction result; extract second image features of different scales from the surround view multi-camera image samples through the student model, perform dimensionality reduction processing on the second image features of each scale to obtain one-dimensional image features of each scale of the surround view multi-camera image samples, perform BEV feature conversion on the one-dimensional image features of each scale based on the multi-layer perceptron to obtain one-dimensional BEV features of each scale of the surround view multi-camera image samples, and convert the second image features of each scale of the surround view multi-camera image samples into BEV features based on the multi-layer perceptron to obtain one-dimensional BEV features of each scale of the surround view multi-camera image samples. The one-dimensional BEV features are upgraded to obtain two-dimensional BEV features of each scale of the surround view multi-camera image samples, the two-dimensional BEV features of each scale are fused to obtain the fused BEV features of the surround view multi-camera image samples, and the occupancy grid is predicted based on the fused BEV features of the surround view multi-camera image samples to obtain a third prediction result; the BEV feature distillation loss is determined according to the target BEV features and the fused BEV features of the surround view multi-camera image samples, and the occupancy prediction distillation loss is determined according to the second prediction results and the third prediction results; the student model is optimized based on the BEV feature distillation loss and the occupancy prediction distillation loss to obtain the target student model.

[0143] In one embodiment, the training module 514 is also used to extract a first image feature from the surround multi-camera image through the backbone network of the teacher model; perform perspective transformation on the first image feature through the perspective transformation network of the teacher model to obtain a preliminary BEV feature; and fine-tune the preliminary BEV feature through the BEV encoder of the teacher model to obtain a target BEV feature.

[0144] In one embodiment, the training module 514 is also used to perform occupancy grid prediction on the target BEV feature through the occupancy prediction head of the teacher model to obtain a second intermediate result of the occupancy grid prediction; perform height conversion on the first intermediate result to obtain a second prediction result with length, width and height; perform occupancy grid prediction on the fused BEV features of the surround multi-camera image samples through the occupancy prediction head of the student model to obtain a third intermediate result of the occupancy grid prediction; perform height conversion on the second intermediate result to obtain a third prediction result with length, width and height.

[0145] In the above embodiment, the target BEV features are extracted from the surround view multi-camera image samples through the teacher model, and the occupancy grid prediction is performed based on the target BEV features to obtain the second prediction result; the second image features of different scales are extracted from the surround view multi-camera image samples through the student model, and the second image features of each scale are respectively subjected to dimensionality reduction processing to obtain the one-dimensional image features of each scale of the surround view multi-camera image samples, and the one-dimensional image features of each scale are respectively subjected to BEV feature conversion based on the multi-layer perceptron to obtain the one-dimensional BEV features of each scale of the surround view multi-camera image samples, and the one-dimensional image features of each scale of the surround view multi-camera image samples are respectively subjected to dimensionality reduction processing to obtain the one-dimensional BEV features of each scale of the surround view multi-camera image samples. The 2D BEV features are processed by dimensionality upgrade to obtain the 2D BEV features of each scale of the surround view multi-camera image samples, and the fused BEV features of the surround view multi-camera image samples are obtained by fusing the 2D BEV features of each scale. The occupancy grid is predicted based on the fused BEV features of the surround view multi-camera image samples to obtain the third prediction result; the BEV feature distillation loss is determined according to the target BEV features and the fused BEV features of the surround view multi-camera image samples, and the occupancy prediction distillation loss is determined according to the second prediction result and the third prediction result; the student model is optimized based on the BEV feature distillation loss and the occupancy prediction distillation loss to obtain the target student model. The BEV feature distillation and occupancy grid prediction distillation are introduced to ensure the expression ability of the model and the quality of the occupancy grid prediction results, so that the obtained target student model has better expression ability and improves the prediction accuracy when performing occupancy grid prediction; in addition, the fully connected layer is used to realize the BEV perspective transformation in the occupancy grid network, which improves the deployment timeliness requirements and friendliness of the model.

[0146] Each module in the above-mentioned occupancy grid prediction device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0147] In one embodiment, a computer device is provided, which may be a server, and includes a processor, a memory, an input / output interface (I / O for short) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, an occupancy grid prediction method is implemented.

[0148] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in FIG7. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected via a system bus, and the communication interface, the display unit, and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an occupancy grid prediction method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0149] Those skilled in the art will understand that the structure shown in FIG. 7 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0150] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above-mentioned occupancy grid prediction method when executing the computer program.

[0151] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned occupancy grid prediction method are implemented.

[0152] In one embodiment, a computer program product is provided, comprising a computer program, which implements the steps of the above-mentioned occupancy grid prediction method when executed by a processor.

[0153] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0154] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0155] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0156] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for predicting an occupied grid, characterized in that: The method comprises: Extracting first image features of different scales from surround-view multi-camera images; Performing dimensionality reduction processing on the first image features of each scale respectively to obtain one-dimensional image features of each scale; Performing BEV feature conversion on the one-dimensional image features of each scale by a multi-layer perceptron to obtain one-dimensional BEV features of each scale; Performing dimensionality-upgrading processing on the one-dimensional BEV features of each scale respectively to obtain the two-dimensional BEV features of each scale; Fusing the two-dimensional BEV features of each scale to obtain a fused BEV feature; Fine-tune the fused BEV feature to obtain a fine-tuned BEV feature; perform occupancy grid prediction on the fused BEV feature through the occupancy prediction head of the target student model to obtain a first intermediate result of occupancy grid prediction; perform height conversion on the first intermediate result to obtain a first prediction result having length, width and height; The target student model is trained by the following method: Extracting target BEV features from surround multi-camera image samples through the teacher model, and performing occupancy grid prediction based on the target BEV features to obtain a second prediction result; Through the student model, second image features of different scales are extracted from the surround view multi-camera image samples, and the second image features of each scale are respectively subjected to dimensionality reduction processing to obtain one-dimensional image features of each scale of the surround view multi-camera image samples, and the one-dimensional image features of each scale are respectively subjected to BEV feature conversion based on the multi-layer perceptron to obtain one-dimensional BEV features of each scale of the surround view multi-camera image samples, and the one-dimensional BEV features of each scale of the surround view multi-camera image samples are respectively subjected to dimensionality increase processing to obtain two-dimensional BEV features of each scale of the surround view multi-camera image samples, and the two-dimensional BEV features of each scale are fused to obtain the fused BEV features of the surround view multi-camera image samples, and the occupancy grid is predicted based on the fused BEV features of the surround view multi-camera image samples to obtain a third prediction result; Determine a BEV feature distillation loss according to the target BEV feature and the fused BEV feature of the surround multi-camera image sample, and determine an occupancy prediction distillation loss according to the second prediction result and the third prediction result; The student model is optimized based on the BEV feature distillation loss and the occupancy prediction distillation loss to obtain the target student model.

2. The method according to claim 1, characterized in that The image features of different scales include first-scale image features and second-scale image features; The extracting of image features of different scales from the surround multi-camera images includes: Extracting the first-scale image features and the second-scale image features from the surround-view multi-camera images through a backbone network in the target learning model; The scale of the first-scale image feature is greater than the scale of the second-scale image feature.

3. The method according to claim 2, characterized in that The one-dimensional image features of each scale include a one-dimensional image feature of a first scale and a one-dimensional image feature of a second scale; The one-dimensional image features of each scale are converted into BEV features by a multi-layer perceptron to obtain the one-dimensional BEV features of each scale, including: Performing BEV feature conversion on the one-dimensional image features of the first scale by using the multi-layer perceptron of the first branch in the target learning model to obtain a one-dimensional first BEV feature; The one-dimensional image features of the second scale are converted into BEV features by the multilayer perceptron of the second branch in the target learning model to obtain a one-dimensional second BEV feature.

4. The method according to claim 2, characterized in that: The two-dimensional BEV features of each scale include a two-dimensional BEV feature corresponding to the first-scale image feature and a two-dimensional BEV feature corresponding to the second-scale image feature; the method further includes: Performing channel transformation on the two-dimensional BEV feature corresponding to the first scale image feature and / or the two-dimensional BEV feature corresponding to the second scale image feature to obtain a channel-transformed BEV feature corresponding to the first scale image feature and / or a channel-transformed BEV feature corresponding to the second scale image feature; The fusing of the two-dimensional BEV features of each scale to obtain the fused BEV features includes: The channel-transformed BEV feature corresponding to the first-scale image feature and the two-dimensional BEV feature corresponding to the second-scale image feature are fused, or the two-dimensional BEV feature corresponding to the first-scale image feature and the channel-transformed BEV feature corresponding to the second-scale image feature are fused, or the channel-transformed BEV feature corresponding to the first-scale image feature and the channel-transformed BEV feature corresponding to the second-scale image feature are fused to obtain a fused BEV feature.

5. The method according to claim 2, characterized in that: The performing dimensionality reduction processing on the first image features of each scale to obtain the one-dimensional image features of each scale comprises: Performing morphological transformation and dimension rearrangement on the first-scale image features to obtain one-dimensional image features of the first scale; The second-scale image features are morphologically transformed and dimensionally rearranged to obtain one-dimensional image features of the second scale.

6. The method according to claim 1, characterized in that The target BEV features extracted from the surround multi-camera image samples by the teacher model include: Extracting the first image features from the surround multi-camera image samples through the backbone network of the teacher model; Performing perspective transformation on the first image feature of the surround view multi-camera image sample through the perspective transformation network of the teacher model to obtain a preliminary BEV feature; The preliminary BEV features are fine-tuned by the BEV encoder of the teacher model to obtain the target BEV features.

7. An occupancy grid prediction device, characterized in that: The device comprises: An extraction module, used for extracting first image features of different scales from the surround multi-camera images; A dimension reduction module, used for performing dimension reduction processing on the first image features of each scale respectively to obtain one-dimensional image features of each scale; A feature conversion module, used for performing BEV feature conversion on the one-dimensional image features of each scale through a multi-layer perceptron to obtain one-dimensional BEV features of each scale; A dimension-increasing module is used to perform dimension-increasing processing on the one-dimensional BEV features of each scale to obtain the two-dimensional BEV features of each scale; A fusion module, used for fusing the two-dimensional BEV features of each scale to obtain a fused BEV feature; A prediction module is used to fine-tune the fused BEV features to obtain fine-tuned BEV features; perform occupancy grid prediction on the fused BEV features through the occupancy prediction head of the target student model to obtain a first intermediate result of occupancy grid prediction; perform height conversion on the first intermediate result to obtain a first prediction result with length, width and height; The training module is used to extract the target BEV features from the surround-view multi-camera image samples through the teacher model, and perform occupancy grid prediction based on the target BEV features to obtain a second prediction result; extract second image features of different scales from the surround-view multi-camera image samples through the student model, perform dimensionality reduction processing on the second image features of each scale to obtain one-dimensional image features of each scale of the surround-view multi-camera image samples, perform BEV feature conversion on the one-dimensional image features of each scale based on the multi-layer perceptron to obtain one-dimensional BEV features of each scale of the surround-view multi-camera image samples, and convert the one-dimensional BEV features of each scale of the surround-view multi-camera image samples into BEV features. The features are processed by dimensionality upgrading to obtain two-dimensional BEV features of each scale of the surround view multi-camera image samples, the two-dimensional BEV features of each scale are fused to obtain fused BEV features of the surround view multi-camera image samples, and occupancy grid prediction is performed based on the fused BEV features of the surround view multi-camera image samples to obtain a third prediction result; the BEV feature distillation loss is determined according to the target BEV features and the fused BEV features of the surround view multi-camera image samples, and the occupancy prediction distillation loss is determined according to the second prediction result and the third prediction result; the student model is optimized based on the BEV feature distillation loss and the occupancy prediction distillation loss to obtain the target student model.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images