Bird's-eye view feature map processing method, device, electronic device and storage medium
Through the depth completion network, the depth map of the camera's view angle is predicted and projected, and combined with the lidar characteristics, efficient bird's-eye view angle feature map processing is achieved, solving the problem of low computing efficiency in the existing technology and improving the depth of autonomous driving Perceived accuracy and efficiency.
Patent Information
- Application Number
- CN202211729788.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-30
- Publication Date
- 2024-12-20
- Estimated Expiration
- 2042-12-30
AI Technical Summary
The prior art introduces unnecessary 2D-3D connections in the field of autonomous driving, resulting in low computing efficiency in the conversion process from camera perspective to bird's-eye view feature map.
By obtaining the original image and depth map of the camera's perspective, using the depth completion network for depth prediction, obtaining the target depth map, and projecting the image features into three-dimensional space, pooling the aerial view perspective, combining the aerial view perspective feature map of lidar Perform feature fusion to generate a bird's-eye view feature map.
Improves the accuracy of depth reverse projection, reduces unnecessary 2D-3D connections, saves memory and video memory, and improves computing efficiency.
Smart Images
Figure CN116051489B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method for processing a bird's-eye view feature map, a device for processing a bird's-eye view feature map, an electronic device, and a computer-readable storage medium. Background Art
[0002] Recently, due to the importance of Bird's Eye View (BEV) features in the field of 3D perception technology in the field of autonomous driving, many works in the field of deep learning focus on the transformation from camera view to BEV feature maps. The following methods are used:
[0003] 1) The feature map is back-projected into three-dimensional space by implicitly predicting the depth distribution, and then the BEV pooling operation is used to obtain the feature map from a bird's-eye view perspective;
[0004] 2) Instead of explicitly predicting the depth distribution, the 3D voxels are projected onto a 2D bounding box on the image and the features within the 2D box are accumulated as the features of the 3D voxels;
[0005] 3) The image perspective feature map is transformed into the bird's-eye view perspective feature through sparse matrix multiplication.
[0006] However, the above methods do not rely on the method of displaying the depth map prediction, which introduces many unnecessary 2D-3D connections, increases unnecessary calculations, and leads to poor operating efficiency. Summary of the invention
[0007] In view of the above problems, embodiments of the present invention are proposed to provide a method for processing a bird's-eye view perspective feature map, a bird's-eye view perspective feature map processing device, an electronic device and a computer-readable storage medium that overcome the above problems or at least partially solve the above problems.
[0008] In order to solve the above problem, an embodiment of the present invention discloses a method for processing a bird's-eye view feature map, the method comprising:
[0009] Obtaining an original image and an original depth map of the camera perspective, and extracting image features of the original image;
[0010] Performing depth prediction on the original depth map to obtain N target depth maps, and projecting the image features into a three-dimensional space according to the N target depth maps to obtain a three-dimensional projection point of each pixel in the original image; N is a positive integer;
[0011] Performing bird's-eye view perspective pooling on the three-dimensional projection points to obtain a first bird's-eye view perspective feature map;
[0012] Obtain a second bird's-eye view feature map corresponding to the laser radar;
[0013] The first bird's-eye view perspective feature map and the second bird's-eye view perspective are feature-fused to obtain a target bird's-eye view perspective feature map.
[0014] In one or more embodiments, extracting the image features of the original image includes:
[0015] The image feature of each pixel in the original image is extracted.
[0016] In one or more embodiments, performing depth prediction on the original depth map to obtain N target depth maps, and projecting the image features into a three-dimensional space according to the N target depth maps to obtain a three-dimensional projection point of each pixel in the original image, includes:
[0017] Performing N depth predictions on each pixel in the original depth map through the trained depth completion network to obtain N target depth maps; wherein each pixel in each target depth map corresponds to one depth prediction value;
[0018] The image feature of each pixel in the original image is projected into a three-dimensional space using the N depth prediction values to obtain N three-dimensional projection points of each pixel in the original image.
[0019] In one or more embodiments, performing bird's-eye view pooling on the three-dimensional projection points to obtain a first bird's-eye view feature map includes:
[0020] Performing bird's-eye view perspective pooling on the three-dimensional projection point of each pixel to obtain a feature vector of each bird's-eye view perspective grid;
[0021] The feature vectors of each bird's-eye view perspective grid are merged to obtain a first bird's-eye view perspective feature map.
[0022] In one or more embodiments, obtaining a second bird's-eye view feature map corresponding to the laser radar includes:
[0023] Acquire radar data collected by the laser radar;
[0024] A bird's-eye view viewing angle feature is extracted from the radar data to obtain a second bird's-eye view viewing angle feature map.
[0025] In one or more embodiments, further comprising:
[0026] Entity prediction is performed on the original image based on the target bird's-eye view perspective feature map to obtain prediction results for each entity in the original image.
[0027] Correspondingly, an embodiment of the present invention further discloses a device for processing a bird's-eye view feature map, the device comprising:
[0028] A first acquisition module is used to acquire an original image and an original depth map of a camera perspective;
[0029] An extraction module, used for extracting image features of the original image;
[0030] A processing module, configured to perform depth prediction on the original depth map to obtain N target depth maps, and project the image features into a three-dimensional space according to the N target depth maps to obtain a three-dimensional projection point of each pixel in the original image; N is a positive integer;
[0031] A pooling module, used for performing bird's-eye view pooling on the three-dimensional projection points to obtain a first bird's-eye view feature map;
[0032] A second acquisition module is used to acquire a second bird's-eye view feature map corresponding to the laser radar;
[0033] A fusion module is used to perform feature fusion on the first bird's-eye view perspective feature map and the second bird's-eye view perspective to obtain a target bird's-eye view perspective feature map.
[0034] In one or more embodiments, the extraction module is specifically used to:
[0035] The image feature of each pixel in the original image is extracted.
[0036] In one or more embodiments, the processing module is specifically used to:
[0037] Performing N depth predictions on each pixel in the original depth map through the trained depth completion network to obtain N target depth maps; wherein each pixel in each target depth map corresponds to one depth prediction value;
[0038] The image feature of each pixel in the original image is projected into a three-dimensional space using the N depth prediction values to obtain N three-dimensional projection points of each pixel in the original image.
[0039] In one or more embodiments, the pooling module is specifically used to:
[0040] Performing bird's-eye view perspective pooling on the three-dimensional projection point of each pixel to obtain a feature vector of each bird's-eye view perspective grid;
[0041] The feature vectors of each bird's-eye view perspective grid are merged to obtain a first bird's-eye view perspective feature map.
[0042] In one or more embodiments, the second acquisition module is specifically configured to:
[0043] Acquire radar data collected by the laser radar;
[0044] A bird's-eye view viewing angle feature is extracted from the radar data to obtain a second bird's-eye view viewing angle feature map.
[0045] In one or more embodiments, further comprising:
[0046] The prediction module is used to perform entity prediction on the original image based on the target bird's-eye view perspective feature map to obtain prediction results for each entity in the original image.
[0047] Correspondingly, an embodiment of the present invention discloses an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the various steps of the above-mentioned bird's-eye view perspective feature map processing method embodiment are implemented.
[0048] Correspondingly, an embodiment of the present invention discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various steps of the above-mentioned bird's-eye view perspective feature map processing method embodiment are implemented.
[0049] The embodiments of the present invention include the following advantages:
[0050] The original image and original depth map of the camera perspective are obtained, and the image features of the original image are extracted. Then, the original depth map is depth predicted to obtain N target depth maps, and the image features are projected into three-dimensional space according to the N target depth maps to obtain the three-dimensional projection points of each pixel in the original image; N is a positive integer; the three-dimensional projection points are then bird's-eye view pooling to obtain a first bird's-eye view feature map, and the second bird's-eye view feature map corresponding to the laser radar is obtained, and the first bird's-eye view feature map and the second bird's-eye view feature map are feature-fused to obtain the target bird's-eye view feature map. In this way, a depth completion method is used to perform depth prediction on each pixel, thereby providing a more accurate dense depth prediction, and the predicted depth value will be used for the reverse projection of the image features, thereby realizing the reverse projection of the image features to the three-dimensional space, and then the three-dimensional projection points are projected onto the bird's-eye view plane through the bird's-eye view pooling method, and the final bird's-eye view feature map is obtained by combining the bird's-eye view feature map of the laser radar. Since a more accurate depth prediction result is adopted, the accuracy of the depth reverse projection is improved.
[0051] To further improve the robustness of the method, multiple depth maps can be integrated during the back-projection process to reduce the impact of noise in a single depth map.
[0052] At the same time, unlike the method of using discrete depth distribution prediction, the embodiment of the present invention can directly project the depth feature to the predicted three-dimensional space position, reducing unnecessary 2D-3D connections, saving memory and video memory usage, and improving the computational efficiency of perspective transformation, thereby improving operating efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] Figure 1 This is a flowchart of the steps of Embodiment 1 of a method for processing a bird's-eye view feature map of the present invention;
[0054] Figure 2 It is a schematic diagram of the reverse projection of the present invention;
[0055] Figure 3 It is a logic schematic diagram of a first embodiment of a method for processing a bird's-eye view feature map of the present invention;
[0056] Figure 4 This is a flowchart of the steps of Embodiment 2 of a method for processing a bird's-eye view feature map of the present invention;
[0057] Figure 5 It is a structural block diagram of an embodiment of a bird's-eye view perspective feature map processing device of the present invention. DETAILED DESCRIPTION
[0058] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0059] One of the core concepts of the embodiment of the present invention is to obtain the original image and the original depth map of the camera perspective, extract the image features of the original image, and then perform depth prediction on the original depth map to obtain N target depth maps, and project the image features into three-dimensional space according to the N target depth maps to obtain the three-dimensional projection points of each pixel in the original image; N is a positive integer; and then perform bird's-eye view pooling on the three-dimensional projection points to obtain a first bird's-eye view feature map, and obtain a second bird's-eye view feature map corresponding to the laser radar, and perform feature fusion on the first bird's-eye view feature map and the second bird's-eye view feature map to obtain a target bird's-eye view feature map. In this way, by using the depth completion method to perform depth prediction on each pixel, a more accurate dense depth prediction is provided, and the predicted depth value will be used for the reverse projection of the image features, thereby realizing the reverse projection of the image features into the three-dimensional space, and then the three-dimensional projection points are projected onto the bird's-eye view plane through the bird's-eye view pooling method, and the final bird's-eye view feature map is obtained by combining the bird's-eye view feature map of the laser radar. Since a more accurate depth prediction result is adopted, the accuracy of depth back projection is improved.
[0060] To further improve the robustness of the method, multiple depth maps can be integrated during the back-projection process to reduce the impact of noise in a single depth map.
[0061] At the same time, unlike the method of using discrete depth distribution prediction, the embodiment of the present invention can directly project the depth feature to the predicted three-dimensional space position, reducing unnecessary 2D-3D connections, saving memory and video memory usage, and improving the computational efficiency of perspective transformation, thereby improving operating efficiency.
[0062] Reference Figure 1 , shows a flowchart of the steps of a method for processing a bird's-eye view feature map in accordance with an embodiment of the present invention, which may specifically include the following steps:
[0063] Step 101: obtain an original image and an original depth map of a camera perspective, and extract image features of the original image.
[0064] Among them, obtaining the original image of the camera perspective may be obtaining the original image captured by the camera. The camera may be installed on the vehicle to capture images of the vehicle's surrounding environment, for example, capturing images in front of the vehicle.
[0065] Furthermore, the original depth map may be an original depth map acquired by a depth camera. The depth camera may also be installed on the vehicle to acquire a depth image of the vehicle's surrounding environment, for example, to acquire a depth image in front of the vehicle.
[0066] It should be noted that the camera for collecting the original image and the camera for collecting the original depth map can be one, that is, it has the function of collecting both the image and the depth map; or it can be two, that is, one is used to collect the image and the other is used to collect the depth map; it can also be other methods. In practical applications, the methods that can collect the original image and the original depth map are applicable to the embodiments of the present invention, and the embodiments of the present invention are not limited to this.
[0067] After obtaining the original image and the original depth map, image features can be extracted from the original image. The step of extracting the image features of the original image includes:
[0068] The image feature of each pixel in the original image is extracted.
[0069] Specifically, when extracting image features, the image features of each pixel in the original image can be extracted. The more pixels there are, the denser the density of subsequent depth prediction will be, and the higher the accuracy of depth prediction will be.
[0070] Step 102, performing depth prediction on the original depth map to obtain N target depth maps, and projecting the image features into a three-dimensional space according to the N target depth maps to obtain a three-dimensional projection point of each pixel in the original image; N is a positive integer.
[0071] After obtaining the original depth map, depth predictions of N depths can be performed on it to obtain N target depth maps, and then the image features of each pixel in the original image are projected into three-dimensional space according to the N target depth maps to obtain the three-dimensional projection point of each pixel in the original image; where N is a positive integer.
[0072] In an embodiment of the present invention, performing depth prediction on the original depth map to obtain N target depth maps, and projecting the image features into a three-dimensional space according to the N target depth maps to obtain a three-dimensional projection point of each pixel in the original image, includes:
[0073] Performing N depth predictions on each pixel in the original depth map through the trained depth completion network to obtain N target depth maps; wherein each pixel in each target depth map corresponds to one depth prediction value;
[0074] The image feature of each pixel in the original image is projected into a three-dimensional space using the N depth prediction values to obtain N three-dimensional projection points of each pixel in the original image.
[0075] Specifically, the trained depth completion network can be used to perform N depth predictions on each pixel in the original depth map. Since each pixel has N depth predictions, each depth prediction can obtain a target depth map, thereby obtaining N target depth maps. In other words, each pixel in each target depth map corresponds to 1 depth prediction value. When N target depth maps are added together, each pixel corresponds to N depth prediction values, and the N depth prediction values corresponding to each pixel are on a straight line. In this way, by integrating multiple depth maps in the projection process, the impact of noise in a single depth map can be reduced, thereby improving the robustness of the method.
[0076] Among them, the trained depth completion network can be a CSPN (Convolutional Spatial Propagation Network) depth completion network with ResNet-50 as the backbone, and the CSPN network can output a higher quality depth map at a lower computational cost. Of course, in addition to the CSPN network, other depth completion networks can also be used. In practical applications, they can be set according to actual needs, and the embodiment of the present invention does not limit this.
[0077] Then, the image features of each pixel in the original image are projected into three-dimensional space using N depth prediction values, thereby obtaining N three-dimensional projection points for each pixel in the original image.
[0078] For example, Figure 2 As shown, assuming that there are 4 target depth maps, that is, N=4, the image feature of a pixel in the original image is projected into the three-dimensional space according to the 4 depth prediction values, thereby obtaining 4 projection points, and the 4 projection points exist on a straight line (shown by the dotted line).
[0079] Step 103, performing bird's-eye view perspective pooling on the three-dimensional projection points to obtain a first bird's-eye view perspective feature map.
[0080] After obtaining N three-dimensional projection points of each pixel, bird's-eye view perspective pooling (BEV Pooling) can be performed on each three-dimensional projection point to obtain a bird's-eye view perspective feature map corresponding to the original image (referred to as the "first bird's-eye view perspective feature map").
[0081] In the embodiment of the present invention, the step of performing bird's-eye view pooling on the three-dimensional projection points to obtain a first bird's-eye view feature map includes:
[0082] Performing bird's-eye view perspective pooling on the three-dimensional projection point of each pixel to obtain a feature vector of each bird's-eye view perspective grid;
[0083] The feature vectors of each bird's-eye view perspective grid are merged to obtain a first bird's-eye view perspective feature map.
[0084] Specifically, for any pixel, the N corresponding three-dimensional projection points are pooled to obtain the feature vector of each three-dimensional projection point corresponding to the bird's-eye view grid. Figure 2 As shown, the gray grid is the bird's-eye view grid. After pooling any three-dimensional projection point, the feature vector of the corresponding grid is obtained. Similarly, the feature vector of each bird's-eye view grid can be obtained by pooling each pixel. Then, the feature vectors of each bird's-eye view grid are merged to obtain the first bird's-eye view feature map.
[0085] It should be noted that, in practical applications, a bird's-eye view grid may correspond to multiple three-dimensional projection points, and the feature vector of the bird's-eye view grid is the sum of the feature vectors corresponding to all three-dimensional projection points.
[0086] Step 104, obtaining a second bird's-eye view feature map corresponding to the laser radar.
[0087] When the first bird's-eye view feature map is obtained, a bird's-eye view feature map corresponding to a laser radar (LiDAR) may be obtained (referred to as a "second bird's-eye view feature map").
[0088] In the embodiment of the present invention, the step of obtaining a second bird's-eye view feature map corresponding to the laser radar includes:
[0089] Acquire radar data collected by the laser radar;
[0090] A bird's-eye view viewing angle feature is extracted from the radar data to obtain a second bird's-eye view viewing angle feature map.
[0091] Specifically, the laser radar data collected by the laser radar can be obtained, and then the bird's-eye view feature extraction is performed on the laser radar data to obtain a second bird's-eye view feature map.
[0092] Step 105 , performing feature fusion on the first bird's-eye view perspective feature map and the second bird's-eye view perspective to obtain a target bird's-eye view perspective feature map.
[0093] After obtaining the first bird's-eye view perspective feature map and the second bird's-eye view perspective feature map, the features of the two are fused to obtain the final bird's-eye view perspective feature map (referred to as "target bird's-eye view perspective feature map").
[0094] For ease of understanding, Figure 3 A logical schematic diagram of an embodiment of the present invention is shown. Specifically, image feature extraction is performed on the collected original image to obtain image features of each pixel, and depth prediction is performed on the collected original depth map to obtain N target depth maps, and then the image features are projected and pooled based on the N target depth maps to obtain a first bird's-eye view feature map. Bird's-eye view feature extraction is performed on the lidar data to obtain a second bird's-eye view feature map. The first bird's-eye view feature map and the second bird's-eye view feature map are then feature fused to obtain a target bird's-eye view feature map.
[0095] In an embodiment of the present invention, the original image and the original depth map of the camera perspective are obtained, and the image features of the original image are extracted, and then the original depth map is depth predicted to obtain N target depth maps, and the image features are projected into three-dimensional space according to the N target depth maps to obtain the three-dimensional projection points of each pixel in the original image; N is a positive integer; the three-dimensional projection points are then bird's-eye view pooling to obtain a first bird's-eye view feature map, and the second bird's-eye view feature map corresponding to the laser radar is obtained, and the first bird's-eye view feature map and the second bird's-eye view feature map are feature fused to obtain the target bird's-eye view feature map. In this way, by using the depth completion method to predict the depth of each pixel, a more accurate dense depth prediction is provided, and the predicted depth value will be used for the reverse projection of the image feature, thereby realizing the reverse projection of the image feature to the three-dimensional space, and then the three-dimensional projection point is projected onto the bird's-eye view plane by the bird's-eye view pooling method, and the final bird's-eye view feature map is obtained by combining the bird's-eye view feature map of the laser radar. Due to the use of a more accurate depth prediction result, the accuracy of the depth reverse projection is improved.
[0096] To further improve the robustness of the method, multiple depth maps can be integrated during the back-projection process to reduce the impact of noise in a single depth map.
[0097] At the same time, unlike the method of using discrete depth distribution prediction, the embodiment of the present invention can directly project the depth feature to the predicted three-dimensional space position, reducing unnecessary 2D-3D connections, saving memory and video memory usage, and improving the computational efficiency of perspective transformation, thereby improving operating efficiency.
[0098] Reference Figure 4 , shows a flowchart of the steps of Embodiment 2 of a method for processing a bird's-eye view feature map of the present invention, which may specifically include the following steps:
[0099] Step 401: Acquire an original image and an original depth map of a camera viewing angle, and extract image features of the original image.
[0100] Step 402, performing depth prediction on the original depth map to obtain N target depth maps, and projecting the image features into a three-dimensional space according to the N target depth maps to obtain a three-dimensional projection point of each pixel in the original image; N is a positive integer.
[0101] Step 403: Perform bird's-eye view perspective pooling on the three-dimensional projection points to obtain a first bird's-eye view perspective feature map.
[0102] Step 404, obtaining a second bird's-eye view feature map corresponding to the laser radar.
[0103] Step 405: Fusing the first bird's-eye view perspective feature map and the second bird's-eye view perspective feature map to obtain a target bird's-eye view perspective feature map.
[0104] Among them, steps 401 to 405 are substantially the same as steps 101 to 105, and will not be described here for avoiding repetition.
[0105] Step 406 , performing entity prediction on the original image based on the target bird's-eye view feature map to obtain prediction results of each entity in the original image.
[0106] Specifically, after obtaining the target bird's-eye view feature map, the feature map can be used to predict the entities in the original image, for example, to predict people, vehicles, trees, etc., and obtain the prediction results of each entity.
[0107] In related entity prediction technologies, image features are obtained by finding the projections of the K nearest neighbor LiDAR points of a 3D voxel. However, this method relies on sparse LiDAR point clouds, resulting in poor prediction quality of entities, especially for distant entities. In the embodiments of the present invention, the dense point cloud provided by depth completion is fully and effectively used to fuse image features, thereby increasing the density of the original point cloud and improving the prediction quality of distant entities.
[0108] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.
[0109] Reference Figure 5 , shows a structural block diagram of an embodiment of a bird's-eye view feature map processing device of the present invention, which may specifically include the following modules:
[0110] A first acquisition module 501 is used to acquire an original image and an original depth map of a camera perspective;
[0111] An extraction module 502, used to extract image features of the original image;
[0112] The processing module 503 is used to perform depth prediction on the original depth map to obtain N target depth maps, and project the image features into a three-dimensional space according to the N target depth maps to obtain a three-dimensional projection point of each pixel in the original image; N is a positive integer;
[0113] A pooling module 504 is used to perform bird's-eye view pooling on the three-dimensional projection points to obtain a first bird's-eye view feature map;
[0114] The second acquisition module 505 is used to acquire a second bird's-eye view feature map corresponding to the laser radar;
[0115] The fusion module 506 is used to perform feature fusion on the first bird's-eye view perspective feature map and the second bird's-eye view perspective to obtain a target bird's-eye view perspective feature map.
[0116] In the embodiment of the present invention, the extraction module is specifically used for:
[0117] The image feature of each pixel in the original image is extracted.
[0118] In the embodiment of the present invention, the processing module is specifically used to:
[0119] Performing N depth predictions on each pixel in the original depth map through the trained depth completion network to obtain N target depth maps; wherein each pixel in each target depth map corresponds to one depth prediction value;
[0120] The image feature of each pixel in the original image is projected into a three-dimensional space using the N depth prediction values to obtain N three-dimensional projection points of each pixel in the original image.
[0121] In the embodiment of the present invention, the pooling module is specifically used for:
[0122] Performing bird's-eye view perspective pooling on the three-dimensional projection point of each pixel to obtain a feature vector of each bird's-eye view perspective grid;
[0123] The feature vectors of each bird's-eye view perspective grid are merged to obtain a first bird's-eye view perspective feature map.
[0124] In the embodiment of the present invention, the second acquisition module is specifically used to:
[0125] Acquire radar data collected by the laser radar;
[0126] A bird's-eye view viewing angle feature is extracted from the radar data to obtain a second bird's-eye view viewing angle feature map.
[0127] In an embodiment of the present invention, it also includes:
[0128] The prediction module is used to perform entity prediction on the original image based on the target bird's-eye view perspective feature map to obtain prediction results for each entity in the original image.
[0129] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0130] An embodiment of the present invention further provides an electronic device, including:
[0131] It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned bird's-eye view feature map processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0132] An embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned bird's-eye view perspective feature map processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0133] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0134] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0135] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0136] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0137] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0138] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.
[0139] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.
[0140] The above is a detailed introduction to a method for processing a bird's-eye view perspective feature map and a device for processing a bird's-eye view perspective feature map provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A method for processing a bird's-eye view feature map, characterized in that: The method comprises: Obtaining an original image and an original depth map of the camera perspective, and extracting image features of the original image; Performing depth prediction on the original depth map to obtain N target depth maps, and projecting the image features into a three-dimensional space according to the N target depth maps to obtain a three-dimensional projection point of each pixel in the original image; N is a positive integer; Performing bird's-eye view perspective pooling on the three-dimensional projection points to obtain a first bird's-eye view perspective feature map; Obtain a second bird's-eye view feature map corresponding to the laser radar; Performing feature fusion on the first bird's-eye view perspective feature map and the second bird's-eye view perspective to obtain a target bird's-eye view perspective feature map; The performing depth prediction on the original depth map to obtain N target depth maps, and projecting the image features into a three-dimensional space according to the N target depth maps to obtain a three-dimensional projection point of each pixel in the original image, includes: Performing N depth predictions on each pixel in the original depth map through the trained depth completion network to obtain N target depth maps; wherein each pixel in each target depth map corresponds to one depth prediction value; The image feature of each pixel in the original image is projected into a three-dimensional space using the N depth prediction values to obtain N three-dimensional projection points of each pixel in the original image.
2. The method for processing a bird's-eye view feature map according to claim 1, characterized in that: The extracting the image features of the original image comprises: The image feature of each pixel in the original image is extracted.
3. The method for processing a bird's-eye view feature map according to claim 1, characterized in that: The step of performing bird's-eye view pooling on the three-dimensional projection points to obtain a first bird's-eye view feature map includes: Performing bird's-eye view perspective pooling on the three-dimensional projection point of each pixel to obtain a feature vector of each bird's-eye view perspective grid; The feature vectors of each bird's-eye view perspective grid are merged to obtain a first bird's-eye view perspective feature map.
4. The method for processing a bird's-eye view feature map according to claim 1, characterized in that: The obtaining of a second bird's-eye view feature map corresponding to the laser radar includes: Acquire radar data collected by the laser radar; A bird's-eye view viewing angle feature is extracted from the radar data to obtain a second bird's-eye view viewing angle feature map.
5. The method for processing a bird's-eye view feature map according to claim 1, characterized in that: Also includes: Entity prediction is performed on the original image based on the target bird's-eye view perspective feature map to obtain prediction results for each entity in the original image.
6. A device for processing a bird's-eye view feature map, characterized in that: The device comprises: A first acquisition module is used to acquire an original image and an original depth map of a camera perspective; An extraction module, used for extracting image features of the original image; A processing module, configured to perform depth prediction on the original depth map to obtain N target depth maps, and project the image features into a three-dimensional space according to the N target depth maps to obtain a three-dimensional projection point of each pixel in the original image; N is a positive integer; A pooling module, used for performing bird's-eye view pooling on the three-dimensional projection points to obtain a first bird's-eye view feature map; A second acquisition module is used to acquire a second bird's-eye view feature map corresponding to the laser radar; A fusion module, used for fusing the first bird's-eye view perspective feature map and the second bird's-eye view perspective feature map to obtain a target bird's-eye view perspective feature map; The processing module is specifically used for: Performing N depth predictions on each pixel in the original depth map through the trained depth completion network to obtain N target depth maps; wherein each pixel in each target depth map corresponds to one depth prediction value; The image feature of each pixel in the original image is projected into a three-dimensional space using the N depth prediction values to obtain N three-dimensional projection points of each pixel in the original image.
7. The device for processing a bird's-eye view feature map according to claim 6, characterized in that: The extraction module is specifically used for: The image feature of each pixel in the original image is extracted.
8. The device for processing a bird's-eye view feature map according to claim 6, characterized in that: The pooling module is specifically used for: Performing bird's-eye view perspective pooling on the three-dimensional projection point of each pixel to obtain a feature vector of each bird's-eye view perspective grid; The feature vectors of each bird's-eye view perspective grid are merged to obtain a first bird's-eye view perspective feature map.
9. The device for processing a bird's-eye view feature map according to claim 6, characterized in that: The second acquisition module is specifically used for: Acquire radar data collected by the laser radar; A bird's-eye view viewing angle feature is extracted from the radar data to obtain a second bird's-eye view viewing angle feature map.
10. The bird's-eye view feature map processing device according to claim 6, characterized in that: Also includes: The prediction module is used to perform entity prediction on the original image based on the target bird's-eye view perspective feature map to obtain prediction results for each entity in the original image.
11. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the steps of the method for processing a bird's-eye view feature map as described in any one of claims 1 to 5 are implemented.
12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for processing a bird's-eye view feature map as claimed in any one of claims 1 to 5 are implemented.