Bird-eye view feature generation method and device, equipment and storage medium

Abstract

The invention discloses a bird's-eye view feature generation method and device, equipment and a storage medium, and relates to the technical field of automatic driving, and the bird's-eye view feature generation method comprises the steps: obtaining a target scale feature map according to a preset residual network and input image data; performing depth estimation according to internal and external parameters of a camera, a depth estimation network and the target scale feature map to obtain target pixel depth information; semantic features are extracted according to a first scale feature map in the target scale feature map and the internal and external parameters of the camera, and target semantic feature information is obtained; and performing feature generation according to the target pixel depth information and the target semantic feature information to obtain a target aerial view feature. By combining supervision and self-supervision methods, the precision and generalization ability of depth estimation are improved, the labeling cost is reduced, the robustness of the model is enhanced, the model has better adaptability to noise and abnormal data, and the accuracy of depth estimation is improved.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a method, device, equipment and storage medium for generating bird's-eye view features. Background Art

[0002] In the current field of autonomous driving, methods based on BEV (Bird's Eye View) perception have become the mainstream perception technology in the field of autonomous driving. In the BEV perception task, BEV feature map learning / extraction is an important upstream task. The quality of BEV feature learning will directly affect downstream tasks such as target detection and lane segmentation. The BEV feature learning framework based on depth estimation is one of the most widely used and efficient BEV feature learning architectures. One of the important steps of this task is to extract depth information from 2D images, which is divided into supervised deep learning and self-supervised deep learning. The supervised depth estimation method has accurate depth estimation but requires true depth values ​​and relies heavily on depth labels. The acquisition of accurate labels will consume a lot of computing power and manual resources. The self-supervised depth estimation method does not rely on depth labels, but its accuracy is defective and the computational overhead is large. Therefore, how to solve the low accuracy of depth estimation in the process of BEV perception view generation has become an urgent problem to be solved. Summary of the invention

[0003] The main purpose of this application is to provide a bird's-eye view feature generation method, device, equipment and storage medium, aiming to solve the technical problem of low depth estimation accuracy in the BEV perception view generation process.

[0004] To achieve the above purpose, the present application proposes a bird's-eye view feature generation method, which includes: Obtain a target scale feature map according to a preset residual network and input image data; Depth estimation is performed based on the internal and external parameters of the camera, the depth estimation network, and the target scale feature map to obtain target pixel depth information; Extracting semantic features according to the first scale feature map in the target scale feature map and the camera internal and external parameters to obtain target semantic feature information; Feature generation is performed according to the target pixel depth information and the target semantic feature information to obtain a target bird's-eye view feature.

[0005] In one embodiment, after the step of performing depth estimation according to the camera internal and external parameters, the depth estimation network and the target scale feature map to obtain the target pixel depth information, the step further includes: Obtaining a mean square error loss according to the image depth label and the target pixel depth information; Obtaining a reprojection photometric loss according to camera internal and external parameters and the target pixel depth information; Determine a total target depth loss based on at least one of the input image data, the mean square error loss, and the reprojection photometric loss; The depth estimation network is back-propagated and optimized according to the target total depth loss to obtain an optimized depth estimation network.

[0006] In one embodiment, the step of determining the total target depth loss based on at least one of the input image data, the mean square error loss, and the reprojection photometric loss comprises: Perform label recognition on the input image data to obtain the target label recognition result; When the target label recognition result is the first label recognition result, determining the total target depth loss according to a preset supervised ratio, a preset self-supervised ratio, the mean square error loss, and the reprojection photometric loss; When the target tag recognition result is the second tag recognition result, the total target depth loss is determined according to the reprojection photometric loss.

[0007] In one embodiment, the step of obtaining a target scale feature map according to a preset residual network and input image data includes: Downsampling the input image data a preset number of times according to a preset residual network to obtain an initial scale feature map; The initial scale feature map is subjected to feature fusion according to a feature pyramid network to obtain a target scale feature map.

[0008] In one embodiment, the step of performing depth estimation according to camera internal and external parameters, a depth estimation network, and the target scale feature map to obtain target pixel depth information includes: According to the internal and external parameters of the camera and the depth estimation network, information fusion is performed on the target scale feature map to obtain a fused scale feature map; Upsampling the fused scale feature map to obtain target scale depth information; The target scale depth information is refined according to a preset convolutional neural network to obtain target pixel depth information.

[0009] In one embodiment, the step of extracting semantic features according to the first scale feature map in the target scale feature map and the camera internal and external parameters to obtain target semantic feature information includes: Fusing the first scale feature map in the target scale feature map with the camera internal and external parameters to obtain a first fused feature map; Semantic features are extracted according to the first fusion feature map to obtain target semantic feature information.

[0010] In one embodiment, the step of generating features according to the target pixel depth information and the target semantic feature information to obtain the target bird's-eye view features includes: Mapping is performed according to the target pixel depth information and the target semantic feature information to obtain target mapping feature information; The target mapping feature information is aggregated according to a voxel pooling strategy to obtain a bird's-eye view feature of the target.

[0011] In addition, to achieve the above purpose, the present application also proposes a bird's-eye view feature generation device, the bird's-eye view feature generation device comprising: A processing module, used for obtaining a target scale feature map according to a preset residual network and input image data; An estimation module, used to perform depth estimation based on camera internal and external parameters, a depth estimation network and the target scale feature map to obtain target pixel depth information; An extraction module, configured to extract semantic features according to the first scale feature map in the target scale feature map and the camera internal and external parameters to obtain target semantic feature information; The generating module is used to generate features according to the target pixel depth information and the target semantic feature information to obtain the target bird's-eye view features.

[0012] In addition, to achieve the above-mentioned purpose, the present application also proposes a bird's-eye view feature generation device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the bird's-eye view feature generation method described above.

[0013] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the bird's-eye view feature generation method described above are implemented.

[0014] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the bird's-eye view feature generation method described above are implemented.

[0015] This application obtains a target scale feature map based on a preset residual network and input image data; performs depth estimation based on camera internal and external parameters, a depth estimation network, and the target scale feature map to obtain target pixel depth information; extracts semantic features based on the first scale feature map in the target scale feature map and the camera internal and external parameters to obtain target semantic feature information; performs feature generation based on the target pixel depth information and the target semantic feature information to obtain target bird's-eye view features. By combining supervised and self-supervised methods, the accuracy and generalization ability of depth estimation are improved, the annotation cost is reduced, and the robustness of the model is enhanced, making it more adaptable to noise and abnormal data, and improving the accuracy of depth estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] Figure 1 A schematic diagram of a flow chart provided for the first embodiment of the method for generating a bird's-eye view feature of the present application; Figure 2 A schematic diagram of a structural flow chart of a BEV perception generation method combining self-supervision and supervised depth estimation provided in Example 1 of the bird's-eye view feature generation method of the present application; Figure 3 A schematic diagram of the DepthNet depth estimation structure flow provided in Example 1 of the bird's-eye view feature generation method of this application; Figure 4 A schematic diagram of a flow chart provided for Embodiment 2 of the method for generating a bird's-eye view feature of the present application; Figure 5 A schematic diagram of the structure flow of the deep supervision module provided in Example 2 of the bird's-eye view feature generation method of this application; Figure 6 This is a schematic diagram of the module structure of the bird's-eye view feature generation device according to an embodiment of the present application; Figure 7 Schematic diagram of the device structure of the hardware operating environment involved in the bird's-eye view feature generation method in the embodiment of the present application.

[0019] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0020] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0021] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0022] The main solution of the embodiment of the present application is: obtain a target scale feature map according to a preset residual network and input image data; perform depth estimation according to camera internal and external parameters, a depth estimation network and the target scale feature map to obtain target pixel depth information; extract semantic features according to the first scale feature map in the target scale feature map and the camera internal and external parameters to obtain target semantic feature information; perform feature generation according to the target pixel depth information and the target semantic feature information to obtain a bird's-eye view feature of the target.

[0023] In the current field of autonomous driving, methods based on BEV (Bird's Eye View) perception have become the mainstream perception technology in the field of autonomous driving. In the BEV perception task, BEV feature map learning / extraction is an important upstream task. The quality of BEV feature learning will directly affect downstream tasks such as target detection and lane segmentation. The BEV feature learning framework based on depth estimation is one of the most widely used and efficient BEV feature learning architectures. One of the important steps of this task is to extract depth information from 2D images, which is divided into supervised deep learning and self-supervised deep learning. The supervised depth estimation method has accurate depth estimation but requires true depth values ​​and relies heavily on depth labels. The acquisition of accurate labels will consume a lot of computing power and manual resources. The self-supervised depth estimation method does not rely on depth labels, but its accuracy is defective and the computational overhead is large. Therefore, how to solve the low accuracy of depth estimation in the process of BEV perception view generation has become an urgent problem to be solved.

[0024] This application obtains a target scale feature map based on a preset residual network and input image data; performs depth estimation based on camera internal and external parameters, a depth estimation network, and the target scale feature map to obtain target pixel depth information; extracts semantic features based on the first scale feature map in the target scale feature map and the camera internal and external parameters to obtain target semantic feature information; performs feature generation based on the target pixel depth information and the target semantic feature information to obtain target bird's-eye view features. By combining supervised and self-supervised methods, the accuracy and generalization ability of depth estimation are improved, the annotation cost is reduced, and the robustness of the model is enhanced, making it more adaptable to noise and abnormal data, and improving the accuracy of depth estimation.

[0025] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a bird's-eye view feature generation device capable of realizing the above functions, etc. The following takes the bird's-eye view feature generation device as an example of the execution subject to illustrate this embodiment and the following embodiments.

[0026] Based on this, the present application embodiment provides a method for generating a bird's-eye view feature, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the bird's-eye view feature generation method of the present application.

[0027] In this embodiment, the bird's-eye view feature generation method includes steps S10 to S40: Step S10, obtaining a target scale feature map according to a preset residual network and input image data; It should be noted that if Figure 2 As shown, the BEV perception generation method combining self-supervision and supervised depth estimation in this embodiment includes a multi-scale feature extraction module, a depth estimation module, a semantic information extraction module, a depth supervision module and a BEV feature generation module, wherein the multi-scale feature extraction module uses the ResNet (Residual Network) residual network as the backbone to downsample the input image five times, the depth estimation module is the DepthNet depth estimation network, and DepthNet is a deep learning model for monocular depth estimation.

[0028] It can be understood that the preset residual network refers to a preset ResNet residual network, the input image data refers to image data, or image data and depth labels, the target scale feature map refers to the feature map extracted by the multi-scale feature extraction module, and the target scale feature map includes four scale feature maps of 1 / 2, 1 / 4, 1 / 8 and 1 / 16 of the original image size.

[0029] In the specific implementation, the input image data is downsampled five times according to the pre-set ResNet residual network to obtain feature maps of five scales. The feature map after each downsampling is 1 / 2 of the scale of the feature map before sampling. The feature maps of 1 / 16 and 1 / 32 are then fused, and finally four scale feature maps of 1 / 2, 1 / 4, 1 / 8 and 1 / 16 of the original image size are obtained, namely the target scale feature map.

[0030] In a feasible implementation, step S10 may include steps A11-A12: Step A11, downsampling the input image data a preset number of times according to a preset residual network to obtain an initial scale feature map; It should be noted that the initial scale feature map includes five scale feature maps of 1 / 2, 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original image size.

[0031] In the specific implementation, the input image data is downsampled five times through a pre-set ResNet residual network to obtain five scale feature maps of 1 / 2, 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original image size, namely the initial scale feature maps.

[0032] Step A12: performing feature fusion on the initial scale feature map according to a feature pyramid network to obtain a target scale feature map.

[0033] It can be understood that the feature pyramid network refers to the FPN feature pyramid network, where the feature pyramid network (FPN) is a deep learning architecture for multi-scale object detection.

[0034] In the specific implementation, the feature maps of 1 / 16 and 1 / 32 in the initial scale feature map are input into the FPN feature pyramid network for feature fusion to obtain a feature map of 1 / 16 of the original image size, which is then combined with the feature maps of 1 / 2, 1 / 4 and 1 / 8 of the original image size to obtain the target scale feature map.

[0035] It should be noted that in this embodiment, the input image is downsampled five times through the ResNet residual network to obtain feature maps of five scales. The feature map after each downsampling is 1 / 2 of the scale of the feature map before sampling. The 16-fold downsampling and 32-fold downsampling feature maps are sent to the FPN feature pyramid network for feature fusion. The scale after fusion is 1 / 16 of the original image size, enriching the semantic information of the features at this scale. The feature map at this scale and the four-scale feature maps of the previous three downsamplings are sent to the depth estimation and semantic information extraction network for subsequent operations.

[0036] Step S20, performing depth estimation according to the camera internal and external parameters, the depth estimation network and the target scale feature map to obtain target pixel depth information; It can be understood that the camera internal and external parameters are used to describe the geometric properties of the camera and its position and orientation in the world coordinate system, the depth estimation network refers to the DepthNet depth estimation network, and the target pixel depth information refers to the depth information of each pixel of the original image.

[0037] In the specific implementation, the DepthNet depth estimation network is used to combine the internal and external parameters of the camera to perform depth estimation on the target scale feature map, thereby obtaining the depth information of each pixel of the original image, that is, the target pixel depth information.

[0038] In a feasible implementation, step S20 may include steps A21 to A23: Step A21, fusing the target scale feature map according to the camera internal and external parameters and the depth estimation network to obtain a fused scale feature map; It can be understood that the fused scale feature map refers to the feature maps of each scale after feature fusion.

[0039] In the specific implementation, the DepthNet depth estimation network is used to fuse the feature maps of each scale in the target scale feature map and the internal and external parameters of the camera, and then obtain the feature maps of each scale after feature fusion. For example, the 1 / 16 feature map is combined with the internal and external parameters of the camera to obtain a 16-fold downsampled image feature that incorporates the internal and external parameter information of the camera.

[0040] Step A22, upsampling the fused scale feature map to obtain target scale depth information; It can be understood that the target scale depth information refers to the original image scale depth information.

[0041] In a specific implementation, the feature maps of each scale after feature fusion are upsampled, and finally the original scale depth information, that is, the target scale depth information, is obtained. For example, the 16-fold downsampled image features that incorporate the camera's internal and external parameter information are upsampled to obtain the original scale depth information. Figure 1 / 8 scale depth information, and then the 8 times downsampled image features that incorporate the camera's internal and external parameter information are combined with the original Figure 1 / 8 scale depth information is fused and upsampled to obtain the original Figure 1 / 4 scale depth information, feature fusion and upsampling are performed through the above method, and finally the original image scale depth information is obtained.

[0042] Step A23: Refine the target scale depth information according to a preset convolutional neural network to obtain target pixel depth information.

[0043] It can be understood that the preset convolutional neural network refers to the pre-set DCN (Deformable Convolutional Networks) and ASPP (Atrous Spatial Pyramid Pooling) convolutional neural networks.

[0044] In the specific implementation, the scale depth information of the original image is refined through the DCN network and the ASPP network to obtain the final depth information of each pixel of the original image, that is, the target pixel depth information.

[0045] It should be noted that in this embodiment, the DepthNet depth estimation network incorporates the internal and external parameters of the camera as input, expands to the dimension of the feature space through a multi-layer perceptron (MLP), and acts on the image and depth features as attention weights. This helps the network better understand and utilize the spatial position information of the camera. The depth distribution is estimated through convolutional layers and SE (Squeeze-and-Excitation) modules, and the estimated depth is refined using the DCN module and the ASPP depth refinement module to improve the accuracy of depth estimation.

[0046] It should be noted that if Figure 3 As shown in FIG. 1 , in this embodiment, the camera internal and external parameters are integrated into the multi-scale features of the image to obtain depth information. Specifically, the four scale image feature maps extracted previously are sequentially input into the network, and the 16-fold downsampled image is integrated with the camera internal and external parameter information. Through the attention mechanism of the SE network, the network is helped to perceive the spatial information of the camera; after extracting the depth information from the 16-fold downsampled image features integrated with the camera internal and external parameter information, an upsampling is performed to obtain the original Figure 1 / 8 scale depth information, the nearest neighbor interpolation is used in the upsampling process; after the 8 times downsampled image features are cat (Concatenation Fusion, Cat Fusion) fused with the obtained depth information, the depth information is extracted again, and the extracted depth information is upsampled again to obtain the original Figure 1 / 4 scale depth information, the nearest neighbor interpolation is used in the upsampling process; after cat fusion of the 4-fold downsampled image features and the obtained depth information, the depth information is extracted again, and the extracted depth information is upsampled again to obtain the original Figure 1 / 2 scale depth information, and the nearest neighbor interpolation is used in the upsampling process; after cat fusion of the 2x downsampled image features and the obtained depth information, the depth information is extracted again, and the extracted depth information is upsampled again to obtain the original image scale depth information, and the nearest neighbor interpolation is used in the upsampling process; finally, the obtained original image scale depth information is refined through the DCN network and the ASPP network to obtain the final depth information of each pixel of the original image.

[0047] Step S30, extracting semantic features according to the first scale feature map in the target scale feature map and the camera internal and external parameters to obtain target semantic feature information; It can be understood that the first-scale feature map refers to a feature map whose scale is 1 / 16 of the original image size, and the target semantic feature information refers to the category or meaning represented by each pixel or area in the image.

[0048] In the specific implementation, the internal and external parameters of the camera are fused, and semantic extraction is performed on the feature map with a scale of 1 / 16 of the original image size to obtain the category or meaning represented by each pixel or area in the image, that is, the target semantic feature information.

[0049] In a feasible implementation, step S30 may include steps A31-A32: Step A31, fusing the first scale feature map in the target scale feature map with the camera internal and external parameters to obtain a first fused feature map; It can be understood that the first fused feature map refers to a 16-fold downsampled feature map of the fused camera internal and external parameters.

[0050] In the specific implementation, in order to accurately extract semantic information, the 16-fold downsampled feature map (a feature map with a scale of 1 / 16 of the original image size) is fused with the camera internal and external parameters to obtain a 16-fold downsampled feature map that integrates the camera internal and external parameters, i.e., the first fused feature map.

[0051] Step A32: extracting semantic features according to the first fusion feature map to obtain target semantic feature information.

[0052] In the specific implementation, semantic extraction is performed on the 16-fold downsampled feature map of the fusion camera's internal and external parameters to obtain the category or meaning represented by each pixel or area in the image, that is, the target semantic feature information.

[0053] It should be noted that in this embodiment, the semantic information extraction module inputs the image 16 times downsampled features, and integrates the internal and external parameters of the camera as input, expands it to the dimension of the feature space through a multi-layer perceptron (MLP), and acts on the image features as attention weights to extract semantic information through a convolutional layer.

[0054] Step S40 , generating features according to the target pixel depth information and the target semantic feature information to obtain a target bird's-eye view feature.

[0055] It can be understood that the target bird's-eye view feature refers to converting point cloud data, image data or other sensor data in the three-dimensional world into a two-dimensional plane view feature viewed from top to bottom.

[0056] In the specific implementation, the BEV feature generation module in this embodiment accepts depth information and semantic information as input, maps the semantic information features to the 3DBEV space where the vehicle is located through the depth information and the camera internal and external parameter information, and then aggregates these features through voxel pooling operations. Converted into a unified BEV view, the target bird's-eye view features are obtained.

[0057] In a feasible implementation, step S40 may include steps A41-A42: Step A41, mapping is performed according to the target pixel depth information and the target semantic feature information to obtain target mapping feature information; It can be understood that the target mapping feature information refers to the image feature information after 3D mapping.

[0058] In the specific implementation, 3D mapping is performed according to the depth information of each pixel in the original image and the category or meaning represented by each pixel or area in the image, that is, the pixel coordinates in the image are converted into 3D coordinates in the camera coordinate system, and finally the target mapping feature information is obtained.

[0059] Step A42, aggregating the target mapping feature information according to a voxel pooling strategy to obtain a bird's-eye view feature of the target.

[0060] It can be understood that the voxel pooling strategy refers to a strategy for aggregating features through a voxel pooling operation.

[0061] In the specific implementation, the feature information of the image after 3D mapping is aggregated through voxel pooling operation and converted into a unified BEV view to obtain the target bird's-eye view features.

[0062] It should be noted that the 3D mapping operation in this embodiment first uses the intrinsic parameters of the camera and the depth value to convert the pixel coordinates in the image into 3D coordinates in the camera coordinate system, and then uses the extrinsic parameters of the camera to convert the 3D points in the camera coordinate system into points in the world coordinate system where the vehicle is located. The voxel pooling operation divides the 3D space where the vehicle is located into voxel columns one by one from the BEV perspective, and adds and fuses the pixel semantic features of the points in the same column.

[0063] This embodiment obtains a target scale feature map according to a preset residual network and input image data; performs depth estimation according to camera internal and external parameters, a depth estimation network, and the target scale feature map to obtain target pixel depth information; extracts semantic features according to the first scale feature map in the target scale feature map and the camera internal and external parameters to obtain target semantic feature information; performs feature generation according to the target pixel depth information and the target semantic feature information to obtain a bird's-eye view feature of the target. By combining supervised and self-supervised methods, the accuracy and generalization ability of depth estimation are improved, the annotation cost is reduced, and the robustness of the model is enhanced, so that it has better adaptability to noise and abnormal data, and the accuracy of depth estimation is improved.

[0064] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 4 After step S20, the bird's-eye view feature generation method further includes steps S21 to S24: Step S21, obtaining a mean square error loss according to the image depth label and the target pixel depth information; It can be understood that the image depth label refers to assigning a value to each pixel in the image that indicates the distance of the object in the scene corresponding to the pixel from the camera, that is, the depth label or the true depth value, and the mean square error loss refers to the MSE (Mean Squared Error) mean square error loss.

[0065] In the specific implementation, the supervised depth constraint receives the depth label of the image input and the depth information of each pixel of the original image, and minimizes the MSE mean square error loss between the depth estimate and the true value to obtain the mean of the sum of squares of the difference between the predicted value and the target value, that is, the mean square error loss.

[0066] Step S22, obtaining the reprojection photometric loss according to the camera internal and external parameters and the target pixel depth information; It can be understood that the reprojection photometric loss refers to the photometric difference between the reconstructed image and the original image after reprojecting the pixels of the current frame to the adjacent frame through the predicted depth map.

[0067] In a specific implementation, the camera's internal and external participating image data and the depth information input of each pixel of the original image are received, and the reprojection photometric loss is obtained by minimizing the photometric loss between the image after reprojection and the input image.

[0068] Step S23, determining a total target depth loss according to at least one of the input image data, the mean square error loss, and the reprojection photometric loss; In a specific implementation, the total target depth loss refers to the total loss function of the depth estimation network. The total target depth loss is determined based on at least one of the input image data, the mean square error loss, and the reprojection photometric loss. That is, it is determined whether the input image data has a depth label, and then the total target depth loss is determined by combining the mean square error loss and the reprojection photometric loss.

[0069] In a feasible implementation, step S23 may include steps A231 to A233: Step A231, performing label recognition on the input image data to obtain a target label recognition result; It can be understood that the target label recognition result refers to the recognition result of whether the input image has a depth label.

[0070] In a specific implementation, the input image data may include only image data, or image data and a depth label, and then the depth label is identified on the input image data, that is, it is determined whether the input image data has a depth label, and finally the target label identification result is determined.

[0071] Step A232, when the target label recognition result is the first label recognition result, determining the total target depth loss according to the preset supervised ratio, the preset self-supervised ratio, the mean square error loss, and the reprojection photometric loss; It can be understood that the first label recognition result refers to the recognition result of the presence of a deep label in the input image, the preset supervised ratio refers to the ratio of the preset supervised mean square error loss to the total loss, and the preset self-supervised ratio refers to the ratio of the preset self-supervised reprojection loss to the total loss.

[0072] In the specific implementation, when the target label recognition result is the recognition result of the presence of a depth label in the input image, it indicates that depth estimation needs to be performed in combination with supervision and self-supervision, and then the total target depth loss is determined based on the proportion of the pre-set supervised mean square error loss in the total loss, the pre-set self-supervised reprojection loss in the total loss, the mean square error loss, and the reprojection luminosity loss.

[0073] Step A233, when the target tag recognition result is the second tag recognition result, determine the total target depth loss according to the reprojection photometric loss.

[0074] It can be understood that the second tag recognition result refers to the recognition result that the input image does not have a depth tag.

[0075] In a specific implementation, when the target label recognition result is that the input image does not have a depth label, it indicates that the input data has no depth label, and then depth estimation is performed only through self-supervision, that is, the total target depth loss is determined based on the reprojection photometric loss.

[0076] Step S24, performing back propagation optimization on the depth estimation network according to the target total depth loss to obtain an optimized depth estimation network.

[0077] In the specific implementation, the depth estimation network is back-propagated and optimized according to the total loss function of the depth estimation network, that is, the depth estimation process of the DepthNet network is optimized by back-propagation, and finally the optimized depth estimation network is obtained.

[0078] It should be noted that if Figure 5As shown, in this embodiment, during the training process, when the input data has a true depth value, the proportion of supervised constraints on the true value is increased, and the proportion of self-supervised constraints is reduced; when the input data does not contain a depth label, only self-supervised depth constraints are performed to ensure the accuracy of depth estimation. Specifically: the supervised depth constraint receives the depth label of the image input and the original image scale depth information obtained, and optimizes the depth estimation of the DepthNet network by minimizing the MSE mean square error loss between the depth estimate and the true value; the mean square error loss is used to calculate the predicted value With target value The mean of the sum of squares of the differences between the two is calculated as follows: .

[0079] It should be noted that the self-supervised depth constraint receives the camera's internal and external reference image data and the original image scale depth information input, maps the depth information into a 3D depth map in the camera coordinate system, and puts the pixel information of the image data at each position. Then, the 3D image is reprojected back to a 2D image by multiplying it with the camera's internal and external reference matrix. By minimizing the photometric loss between the reprojected image and the input image, back propagation optimizes the depth estimation process of the DepthNet network. The reprojection photometric loss formula is as follows:

[0080] in is the input image, is the reconstructed image. SSIM is the structural similarity measure, is the L1 distance metric, It is the ratio parameter of the two. This patent sets it to 0.75 and can be adjusted according to different scenarios.

[0081] It should be understood that the combination of self-supervision and supervised depth estimation described in this embodiment requires determining whether there is a depth label when inputting an image. When the input data has a depth label, the mean square error loss between the true value and the predicted value in the supervision is adjusted to account for alpha of the total loss to 0.8, and the self-supervised reprojection loss accounts for (1-alpha) to 0.2. 0.8 and 0.2 are empirical values ​​and can be reset according to the input data scenario; when the input data has no depth label, the supervised loss ratio is adjusted to 0 (there is no true value input at this time, and the true value is set to 0); at this time, all network losses are reprojection losses in the self-supervised case, constraining the depth estimation.

[0082] This embodiment obtains the mean square error loss according to the image depth label and the target pixel depth information; obtains the reprojection photometric loss according to the camera internal and external parameters and the target pixel depth information; determines the total target depth loss according to the input image data, the mean square error loss and at least one of the reprojection photometric loss; performs back propagation optimization on the depth estimation network according to the total target depth loss to obtain an optimized depth estimation network. During the training process, when the input data has a true depth value, the proportion of supervised constraints on the true value is increased and the proportion of self-supervised constraints is reduced; when the input data does not contain a depth label, only self-supervised depth constraints are performed to ensure the accuracy of depth estimation.

[0083] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the bird's-eye view feature generation method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0084] This application also provides a bird's-eye view feature generation device, please refer to Figure 6 , the bird's-eye view feature generating device comprises: A processing module 10 is used to obtain a target scale feature map according to a preset residual network and input image data; An estimation module 20, configured to perform depth estimation based on camera internal and external parameters, a depth estimation network and the target scale feature map to obtain target pixel depth information; An extraction module 30 is used to extract semantic features according to the first scale feature map in the target scale feature map and the camera internal and external parameters to obtain target semantic feature information; The generating module 40 is used to generate features according to the target pixel depth information and the target semantic feature information to obtain a target bird's-eye view feature.

[0085] Optionally, the estimation module 20 is further configured to: Obtaining a mean square error loss according to the image depth label and the target pixel depth information; Obtaining a reprojection photometric loss according to camera internal and external parameters and the target pixel depth information; Determine a total target depth loss based on at least one of the input image data, the mean square error loss, and the reprojection photometric loss; The depth estimation network is back-propagated and optimized according to the target total depth loss to obtain an optimized depth estimation network.

[0086] Optionally, the estimation module 20 is further configured to: Perform label recognition on the input image data to obtain the target label recognition result; When the target label recognition result is the first label recognition result, determining the total target depth loss according to a preset supervised ratio, a preset self-supervised ratio, the mean square error loss, and the reprojection photometric loss; When the target tag recognition result is the second tag recognition result, the total target depth loss is determined according to the reprojection photometric loss.

[0087] Optionally, the processing module 10 is further used for: Downsampling the input image data a preset number of times according to a preset residual network to obtain an initial scale feature map; The initial scale feature map is subjected to feature fusion according to a feature pyramid network to obtain a target scale feature map.

[0088] Optionally, the estimation module 20 is further configured to: According to the internal and external parameters of the camera and the depth estimation network, information fusion is performed on the target scale feature map to obtain a fused scale feature map; Upsampling the fused scale feature map to obtain target scale depth information; The target scale depth information is refined according to a preset convolutional neural network to obtain target pixel depth information.

[0089] Optionally, the extraction module 30 is further used for: Fusing the first scale feature map in the target scale feature map with the camera internal and external parameters to obtain a first fused feature map; Semantic features are extracted according to the first fusion feature map to obtain target semantic feature information.

[0090] Optionally, the generating module 40 is further configured to: Mapping is performed according to the target pixel depth information and the target semantic feature information to obtain target mapping feature information; The target mapping feature information is aggregated according to a voxel pooling strategy to obtain a bird's-eye view feature of the target.

[0091] The bird's-eye view feature generation device provided by the present application adopts the bird's-eye view feature generation method in the above embodiment, which can solve the technical problem of low depth estimation accuracy in the BEV perception view generation process. Compared with the prior art, the beneficial effects of the bird's-eye view feature generation device provided by the present application are the same as the beneficial effects of the bird's-eye view feature generation method provided by the above embodiment, and the other technical features in the bird's-eye view feature generation device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0092] The present application provides a bird's-eye view feature generation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the bird's-eye view feature generation method in the above-mentioned embodiment one.

[0093] Reference below Figure 7 , which shows a schematic diagram of the structure of a bird's-eye view feature generation device suitable for implementing the embodiment of the present application. The bird's-eye view feature generation device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The bird's-eye view feature generation device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0094] like Figure 7As shown, the bird's-eye view feature generation device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the bird's-eye view feature generation device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the bird's-eye view feature generation device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a bird's-eye view feature generation device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0095] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0096] The bird's-eye view feature generation device provided by the present application adopts the bird's-eye view feature generation method in the above embodiment, which can solve the technical problem of low depth estimation accuracy in the BEV perception view generation process. Compared with the prior art, the beneficial effects of the bird's-eye view feature generation device provided by the present application are the same as the beneficial effects of the bird's-eye view feature generation method provided by the above embodiment, and the other technical features in the bird's-eye view feature generation device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0097] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0098] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0099] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the bird's-eye view feature generation method in the above-mentioned embodiment.

[0100] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0101] The computer-readable storage medium may be included in the bird's-eye view feature generation device; or may exist independently without being assembled into the bird's-eye view feature generation device.

[0102] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by a bird's-eye view feature generation device, the bird's-eye view feature generation device: obtains a target scale feature map according to a preset residual network and input image data; performs depth estimation according to camera internal and external parameters, a depth estimation network, and the target scale feature map to obtain target pixel depth information; extracts semantic features according to a first scale feature map in the target scale feature map and the camera internal and external parameters to obtain target semantic feature information; performs feature generation according to the target pixel depth information and the target semantic feature information to obtain a target bird's-eye view feature.

[0103] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0104] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0105] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0106] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned bird's-eye view feature generation method, and can solve the technical problem of low depth estimation accuracy in the BEV perception view generation process. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the bird's-eye view feature generation method provided in the above-mentioned embodiment, and will not be repeated here.

[0107] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned bird's-eye view feature generation method when executed by a processor.

[0108] The computer program product provided by the present application can solve the technical problem of low depth estimation accuracy in the process of generating BEV perception views. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the bird's-eye view feature generation method provided by the above embodiment, which will not be repeated here.

[0109] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for generating bird's-eye view features, characterized in that: The bird's-eye view feature generation method comprises: Obtain a target scale feature map according to a preset residual network and input image data; Depth estimation is performed based on the internal and external parameters of the camera, the depth estimation network, and the target scale feature map to obtain target pixel depth information; Extracting semantic features according to the first scale feature map in the target scale feature map and the camera internal and external parameters to obtain target semantic feature information; Feature generation is performed according to the target pixel depth information and the target semantic feature information to obtain a target bird's-eye view feature.

2. The method according to claim 1, characterized in that After the step of performing depth estimation according to the camera internal and external parameters, the depth estimation network and the target scale feature map to obtain the target pixel depth information, the method further includes: Obtaining a mean square error loss according to the image depth label and the target pixel depth information; Obtaining a reprojection photometric loss according to camera internal and external parameters and the target pixel depth information; Determine a total target depth loss based on at least one of the input image data, the mean square error loss, and the reprojection photometric loss; The depth estimation network is back-propagated and optimized according to the target total depth loss to obtain an optimized depth estimation network.

3. The method according to claim 2, characterized in that The step of determining the total target depth loss based on at least one of the input image data, the mean square error loss, and the reprojection photometric loss comprises: Perform label recognition on the input image data to obtain the target label recognition result; When the target label recognition result is the first label recognition result, determining the total target depth loss according to a preset supervised ratio, a preset self-supervised ratio, the mean square error loss, and the reprojection photometric loss; When the target tag recognition result is the second tag recognition result, the total target depth loss is determined according to the reprojection photometric loss.

4. The method according to claim 1, characterized in that The step of obtaining a target scale feature map according to a preset residual network and input image data comprises: Down-sample the input image data a preset number of times according to a preset residual network to obtain an initial scale feature map; The initial scale feature map is subjected to feature fusion according to a feature pyramid network to obtain a target scale feature map.

5. The method according to claim 1, characterized in that The step of performing depth estimation according to the camera internal and external parameters, the depth estimation network and the target scale feature map to obtain target pixel depth information includes: According to the internal and external parameters of the camera and the depth estimation network, information fusion is performed on the target scale feature map to obtain a fused scale feature map; Upsampling the fused scale feature map to obtain target scale depth information; The target scale depth information is refined according to a preset convolutional neural network to obtain target pixel depth information.

6. The method according to claim 1, characterized in that The step of extracting semantic features according to the first scale feature map in the target scale feature map and the camera internal and external parameters to obtain target semantic feature information includes: Fusing the first scale feature map in the target scale feature map with the camera internal and external parameters to obtain a first fused feature map; Semantic features are extracted according to the first fusion feature map to obtain target semantic feature information.

7. The method according to claim 1, characterized in that The step of generating features according to the target pixel depth information and the target semantic feature information to obtain the target bird's-eye view features comprises: Mapping is performed according to the target pixel depth information and the target semantic feature information to obtain target mapping feature information; The target mapping feature information is aggregated according to a voxel pooling strategy to obtain a bird's-eye view feature of the target.

8. A bird's-eye view feature generation device, characterized in that: The device comprises: A processing module, used for obtaining a target scale feature map according to a preset residual network and input image data; An estimation module, used to perform depth estimation based on camera internal and external parameters, a depth estimation network and the target scale feature map to obtain target pixel depth information; An extraction module, configured to extract semantic features according to the first scale feature map in the target scale feature map and the camera internal and external parameters to obtain target semantic feature information; The generating module is used to generate features according to the target pixel depth information and the target semantic feature information to obtain the target bird's-eye view features.

9. A bird's-eye view feature generation device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the bird's-eye view feature generation method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the bird's-eye view feature generation method according to any one of claims 1 to 7 are implemented.

Images