Three-dimensional scene processing method and device, equipment, storage medium and program product

Abstract

The invention relates to a three-dimensional scene processing method and device, equipment, a storage medium and a program product. The method comprises the following steps: acquiring point cloud and image information corresponding to a three-dimensional scene, extracting a first spatial feature according to the image information, extracting a second spatial feature according to the point cloud, fusing the first spatial feature and the second spatial feature according to a feature dimension, and coding a fusion result to obtain a fusion result; and decoding the coding result through a preset completion model to obtain a three-dimensional scene completion result. According to the scheme, the three-dimensional scene is perceived through the multi-view image, the point cloud features are introduced, the image perception result and the point cloud perception result are fused, the consistency problem in the process of integrating the images of different view angles can be improved by means of accurate positioning of the point cloud to the position, the accuracy and consistency of three-dimensional occupied semantic scene completion are improved, and the accuracy and the consistency of the three-dimensional occupied semantic scene completion are improved. And a more reliable technical support is provided for automatic driving and other related fields.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional vision technology, and in particular to a three-dimensional scene processing method, device, equipment, storage medium and program product. Background Art

[0002] In the field of 3D vision technology such as autonomous driving and robot positioning and navigation, environmental perception is a key factor in achieving autonomous navigation and decision-making of target objects. As part of environmental perception, 3D occupancy semantic scene completion aims to generate accurate 3D scene representations through data observed by tools such as lidar and cameras, thereby helping vehicles or intelligent robots understand the spatial distribution and location of objects in the surrounding environment.

[0003] At present, 3D scene perception is mainly achieved by taking multi-view images with a camera, converting the 2D image data into 3D space and giving it semantic information through geometric projection. However, this method is inconsistent in capturing information from different viewpoints, and it is difficult to effectively integrate the information of multiple images when processing images from different viewpoints, and it is impossible to avoid cumulative errors, resulting in the lack of accuracy and consistency of the 3D scenes perceived in this way. Summary of the invention

[0004] One of the purposes of the present invention is to provide a three-dimensional scene processing method, device, equipment, storage medium and program product to solve the problem that the three-dimensional scene perception result is not accurate enough.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] A three-dimensional scene processing method, comprising:

[0007] Obtain point cloud and image information corresponding to the three-dimensional scene;

[0008] Extracting a first spatial feature corresponding to the three-dimensional scene according to the image information, and extracting a second spatial feature corresponding to the three-dimensional scene according to the point cloud;

[0009] fusing the first spatial feature and the second spatial feature according to the feature dimension, and encoding the fusion result to obtain an encoding result;

[0010] The encoding result is decoded by using a preset completion model to obtain a completion result of the three-dimensional scene.

[0011] Furthermore, the image information includes images of the three-dimensional scene corresponding to multiple perspectives, and extracting the first spatial feature corresponding to the three-dimensional scene according to the image information includes:

[0012] Acquire depth distribution features and semantic features of the image;

[0013] Perform forward projection based on the depth distribution features and semantic features of images from multiple perspectives to obtain a forward three-dimensional mesh;

[0014] Performing backward projection based on the forward three-dimensional grid, the depth distribution features and the semantic features of the images of the multiple perspectives to obtain backward two-dimensional features;

[0015] The backward two-dimensional features are processed according to the height dimension of the forward three-dimensional grid to obtain first spatial features.

[0016] Further, obtaining the depth distribution characteristics of the image includes:

[0017] Obtaining camera parameters corresponding to the image, wherein the camera parameters include intrinsic parameters and extrinsic parameters;

[0018] For an image of the same viewing angle, the intrinsic parameters and extrinsic parameters of the image are coupled, and high-dimensional features are extracted according to the coupled parameters;

[0019] According to the high-dimensional features of the image extracted from the same viewing angle, the image is depth-encoded to obtain the depth distribution features of the image.

[0020] Further, extracting a second spatial feature corresponding to the three-dimensional scene according to the point cloud includes:

[0021] Establishing a cylindrical voxel grid based on the point cloud, and mapping features of the cylindrical voxel grid to a two-dimensional grid;

[0022] establishing a three-dimensional occupancy grid based on the point cloud;

[0023] The features in the two-dimensional grid are processed according to the height dimension of the three-dimensional occupied grid to obtain second spatial features.

[0024] Further, fusing the first spatial feature and the second spatial feature according to the feature dimension includes:

[0025] Mapping the first spatial feature and the second spatial feature to a target three-dimensional space of the same scale, wherein the target three-dimensional space includes a plurality of voxel grids;

[0026] The features of the first spatial features and the second spatial features corresponding to the same voxel grid in the target three-dimensional space are spliced ​​to obtain spliced ​​features of the voxel grid, wherein the spliced ​​features represent the geometric structure and semantic information of the target three-dimensional space.

[0027] Further, the fusion result is encoded to obtain an encoding result, including:

[0028] Processing the voxel grid of the target three-dimensional space through a three-dimensional residual network to obtain three-dimensional space features of different scales;

[0029] The three-dimensional spatial features of different scales are fused to obtain cross-scale features, and the cross-scale features are encoded.

[0030] Further, decoding the encoding result through a preset completion model includes:

[0031] Identifying cross-scale features represented by the encoding result through the completion model, upsampling low-scale features to the scale of the target three-dimensional space for integration, so as to obtain integrated features of the voxel grid;

[0032] The integrated features are normalized, and the occupancy semantics completion result of the voxel grid is determined according to the normalization result.

[0033] Furthermore, the three-dimensional spatial features of different scales are fused to obtain cross-scale features, including:

[0034] The three-dimensional spatial features of different scales are input into a feature pyramid network, and low-scale features are fused with high-scale features through the feature pyramid network to obtain cross-scale features.

[0035] A three-dimensional scene processing device, comprising:

[0036] An acquisition module is used to obtain point cloud and image information corresponding to the three-dimensional scene;

[0037] A feature extraction module, configured to extract a first spatial feature corresponding to the three-dimensional scene according to the image information, and to extract a second spatial feature corresponding to the three-dimensional scene according to the point cloud;

[0038] A fusion encoding module, used for fusing the first spatial feature and the second spatial feature according to a feature dimension, and encoding the fusion result to obtain an encoding result;

[0039] The decoding and completion module is used to decode the encoding result through a preset completion model to obtain a completion result of the three-dimensional scene.

[0040] An electronic device comprises: a processor, and a memory communicatively connected to the processor;

[0041] The memory stores computer-executable instructions;

[0042] The processor executes the computer-executable instructions stored in the memory to implement the three-dimensional scene processing method as described in any one of the above items.

[0043] A computer-readable storage medium comprises: the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to implement the three-dimensional scene processing method as described in any one of the above when executed by a processor.

[0044] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the three-dimensional scene processing method as described in any one of the above items is implemented.

[0045] The beneficial effects of the present invention are as follows: by perceiving three-dimensional scenes through multi-perspective images, introducing point cloud features and fusing the image perception results with the point cloud perception results, the point cloud can be used to accurately locate the position to improve the consistency problem in the process of integrating images from different perspectives, and the accuracy and consistency of three-dimensional occupancy semantic scene completion can be improved, providing more reliable technical support for autonomous driving and other related fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A schematic flow chart of a three-dimensional scene processing method provided by an exemplary embodiment of the present invention;

[0047] Figure 2 A schematic diagram of a process of extracting features from an image provided by an exemplary embodiment of the present invention;

[0048] Figure 3 A schematic diagram of a process of extracting features based on a point cloud provided by an exemplary embodiment of the present invention;

[0049] Figure 4 A schematic diagram of a flow chart of fusing features according to feature dimensions provided by an exemplary embodiment of the present invention;

[0050] Figure 5 A schematic diagram of a feature encoding and decoding process provided for an exemplary embodiment of the present invention;

[0051] Figure 6 A specific application diagram of a three-dimensional occupation semantic scene completion method provided by an exemplary embodiment of the present invention;

[0052] Figure 7 A schematic diagram of the structure of a three-dimensional scene processing device provided by an exemplary embodiment of the present invention;

[0053] Figure 8 A schematic structural diagram of an electronic device provided by an exemplary embodiment of the present invention.

[0054] Through the above drawings, clear embodiments of the present invention have been shown, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present invention in any way, but to illustrate the concept of the present invention for those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0055] The following will describe the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention, not for limiting the scope of protection of the present invention.

[0056] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0057] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0058] The terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, product or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, product or device. In the absence of further restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or device including the elements. For example, if the words first, second, etc. are used to indicate names, they do not indicate any particular order.

[0059] Figure 1 FIG. 1 is a flow chart of a three-dimensional scene processing method provided by an exemplary embodiment of the present invention. Figure 1 As shown, the method may include:

[0060] Step S101, obtaining point cloud and image information corresponding to a three-dimensional scene.

[0061] The point cloud may be acquired by a laser radar on the vehicle. The image information may include images taken by the vehicle through image acquisition devices such as cameras or cameras with different viewing angles.

[0062] Step S102: extracting a first spatial feature corresponding to the three-dimensional scene according to the image information, and extracting a second spatial feature corresponding to the three-dimensional scene according to the point cloud.

[0063] For example, for image information, semantic encoding and depth encoding can be performed on images from different perspectives to obtain semantic information and depth information of each image. The images from different perspectives are projected and transformed in combination with the camera intrinsic parameters and camera extrinsic parameters when the image was taken, and the semantic information of the images from different perspectives is integrated into an image-based three-dimensional space (which may be referred to as the image three-dimensional space), thereby obtaining the first spatial feature corresponding to the image. When performing the projection transformation, the camera intrinsic parameters can also be used to combine with the laser radar registration parameters, so that the relative position relationship between the spatial features obtained based on the image and the image features obtained based on the point cloud can be determined based on the laser radar registration parameters.

[0064] In some possible implementations, the image may be preprocessed before encoding. The preprocessing may include operations such as removing distortion and resizing to a uniform dimension.

[0065] For example, for point clouds, ROI (region of interest) processing can be performed on single-frame point clouds, and feature encoding can be performed on the point clouds in the extracted ROI region through a neural network to obtain three-dimensional spatial features based on the point clouds. The neural network for feature encoding the point clouds can include a Pillar feature extraction network and an Occupancy feature extraction network.

[0066] In some possible implementations, when extracting three-dimensional spatial features from point clouds and images, the point clouds and images may be time synchronized. For example, the shooting time of the multi-view image used to extract the first spatial feature may be the same as the acquisition time of the single-frame point cloud used to extract the second spatial feature.

[0067] Step S103: fuse the first spatial feature and the second spatial feature according to the feature dimension, and encode the fusion result to obtain an encoding result.

[0068] The above processing of the image and the point cloud respectively can obtain a three-dimensional space based on the image and a three-dimensional space based on the point cloud. The first spatial feature can represent the geometric structure and semantic information of the three-dimensional space of the image, and the second spatial feature can represent the geometric structure and semantic information of the three-dimensional space of the point cloud. This step can first unify the scales of the two three-dimensional spaces so that the first spatial feature and the second spatial feature can correspond to a unified area in the actual three-dimensional scene, and then splice and fuse the geometric structure and semantic information represented by the first spatial feature and the second spatial feature at the same position in the actual three-dimensional scene to obtain a fusion result so that the fusion result can be encoded.

[0069] For example, the point cloud three-dimensional space can be used as a unified benchmark to divide the three-dimensional space into several cubic grids (i.e., voxels) of the same size but different coordinates. The second spatial feature can characterize the geometric structure (whether it is occupied) and semantic features (category label) of each voxel. Then the first spatial feature can also be mapped to the point cloud three-dimensional space, and the first spatial feature can be used to represent the geometric structure and semantic features of each voxel. Then the first spatial feature and the second spatial feature corresponding to the geometric structure and semantic features of the same voxel are spliced ​​together to complete the fusion.

[0070] In some possible implementations, after fusing the first spatial feature and the second spatial feature, some three-dimensional feature encoders may be used to encode the spliced ​​features represented by the voxels. For example, the fusion result may be downsampled by the encoder, and a single voxel after downsampling may correspond to multiple voxels before downsampling. The encoder may encode the geometric structure and semantic information represented by the downsampled voxels, and may process features at different sampling scales to obtain more features.

[0071] Step S104: decoding the encoding result by using a preset completion model to obtain a completion result of the three-dimensional scene.

[0072] Specifically, a spatial model corresponding to a three-dimensional scene can be established, and the spatial model can be divided into a number of voxels. The completion model can include multiple hidden layers and a fully connected layer. The hidden layer can process the encoding results of the encoder and align the processed features with the voxels of the spatial model. The fully connected layer can perform normalization according to the features corresponding to each voxel in the spatial model, and output the eigenvalues ​​of the geometric structure and the eigenvalues ​​of the semantic information of each voxel. The geometric structure and semantic information represented by each voxel can be finally determined based on the eigenvalues. For example, after normalization, the eigenvalue of the geometric structure of a voxel is [0.1, 0.9], where 0.1 indicates that the voxel is unoccupied and 0.9 indicates that the voxel is occupied. The eigenvalue of the semantic information of the voxel is [0.05, 0.7, 0.25], where 0.05 indicates that the voxel is a person, 0.7 indicates that the voxel is a car, and 0.25 indicates that the voxel is another obstacle. Finally, it can be determined that the geometric structure of the voxel is "occupied" and the semantic information is "car". In this way, the geometric structure and semantic information of all voxels in the entire spatial model corresponding to the three-dimensional scene can be obtained, thereby completing the occupancy semantics of the three-dimensional scene.

[0073] In some possible implementations, the point cloud three-dimensional space or the image three-dimensional space may be directly used as a spatial model corresponding to the three-dimensional scene.

[0074] In the above embodiment, by obtaining the point cloud and image information corresponding to the three-dimensional scene, the first spatial feature can be extracted according to the image information, and the second spatial feature can be extracted according to the point cloud. The first spatial feature and the second spatial feature can be fused according to the feature dimension and the fusion result can be encoded, and then the encoding result can be decoded by a preset completion model to obtain the three-dimensional scene completion result. This solution perceives the three-dimensional scene through multi-view images, introduces point cloud features and fuses the image perception results with the point cloud perception results. It can improve the consistency problem in the process of integrating images from different viewpoints with the help of the precise positioning of the point cloud, improve the accuracy and consistency of the three-dimensional occupancy semantic scene completion, and provide more reliable technical support for autonomous driving and other related fields.

[0075] In one embodiment, the image information includes images of a three-dimensional scene corresponding to multiple viewing angles, such as Figure 2 As shown, extracting the first spatial feature corresponding to the three-dimensional scene according to the image information may include:

[0076] Step S201, obtaining depth distribution features and semantic features of an image.

[0077] Specifically, the residual network algorithm can be used to semantically encode multi-view images to obtain the global semantic information of the multi-view images, and the depth information can be assigned to the semantically encoded images based on the camera parameters corresponding to the images to obtain depth distribution features.

[0078] In some possible implementations, obtaining depth distribution features of an image includes: obtaining camera parameters corresponding to the image, the camera parameters including intrinsic parameters and extrinsic parameters; for images of the same perspective, coupling the intrinsic parameters and extrinsic parameters of the image, and extracting high-dimensional features based on the coupled parameters; and performing depth encoding on the image based on the high-dimensional features extracted from the image of the same perspective to obtain depth distribution features of the image.

[0079] For example, the camera intrinsic parameters and camera extrinsic parameters (or lidar registration parameters) of the corresponding viewing angle can be coupled, and high-dimensional features can be extracted through MLP (Multi Layer Perceptron), and the image can be deeply distributed encoded in combination with the image semantic features to obtain the depth distribution features.

[0080] Step S202 , performing forward projection based on the depth distribution features and semantic features of the images from multiple perspectives to obtain a forward three-dimensional grid.

[0081] Specifically, a two-dimensional image of multiple perspectives can be mapped into a three-dimensional space through a forward projection algorithm, and pixels in the image can be converted into three-dimensional points in the camera coordinate system according to the intrinsic parameters and depth distribution characteristics of the image. A three-dimensional grid corresponding to the voxel space can be established based on the three-dimensional points, and the semantic features of the image can be mapped to the three-dimensional grid. After obtaining the three-dimensional grid, the three-dimensional points can also be projected onto a bird's-eye view plane through a forward projection algorithm. For example, the three-dimensional points in the camera coordinate system can be converted into a world coordinate system and projected onto a two-dimensional plane.

[0082] The forward projection algorithm may be a LSS (Lift, Splat, Shoot) algorithm.

[0083] Step S203 , performing backward projection based on the forward three-dimensional grid, the depth distribution features and the semantic features of the images of multiple perspectives to obtain backward two-dimensional features.

[0084] Specifically, the above results can be processed by a back-projection algorithm. The bird's-eye view plane obtained by forward projection is used as the query initial value. A BEV (Bird's Eye View) feature map is generated according to the bird's-eye view plane and semantic features. Reverse mapping is performed according to the points in the BEV feature map to combine the backward BEV features with the image information.

[0085] Step S204, processing the backward two-dimensional features according to the height dimension of the forward three-dimensional grid to obtain the first spatial features.

[0086] Specifically, the BEV features obtained by back-projection may be linearly superimposed on the height dimension of the forward three-dimensional grid, and the superimposed forward three-dimensional grid represents the first spatial feature based on the image.

[0087] In the above embodiment, by processing image-based features through forward projection and backward projection, the perception capability of three-dimensional semantics based on multi-view views can be enhanced, and the accuracy of three-dimensional scene perception results can be further improved.

[0088] In one embodiment, Figure 3 As shown, extracting the second spatial feature corresponding to the three-dimensional scene according to the point cloud includes:

[0089] Step S301 : establishing a cylindrical voxel grid according to the point cloud, and mapping the features of the cylindrical voxel grid to a two-dimensional grid.

[0090] Specifically, we can divide the point cloud into a grid in three-dimensional space, compress the points in the height direction to the plane, obtain the index vector of the Pillar voxel grid, and then perform Pillar (column) feature encoding on the point cloud. We can further extract high-dimensional features from the point features in each Pillar through one-dimensional convolution, and then map the extracted features to the two-dimensional BEV grid according to the index of the Pillar voxel grid.

[0091] Step S302: establishing a three-dimensional occupancy grid according to the point cloud.

[0092] Specifically, the point cloud can be divided into grids in three-dimensional space, and the number of points in the grid can be counted. The grids with a number of points greater than or equal to 1 are marked as occupied, and the grids without points are marked as unoccupied, and the index vector of the occupancy voxel grid is obtained. Then, the point cloud can be encoded with occupancy features, and the voxel grid can be mapped to the three-dimensional occupancy grid according to the index of the occupancy voxel grid.

[0093] Step S303: Process the features in the two-dimensional grid according to the height dimension of the three-dimensional occupied grid to obtain a second spatial feature.

[0094] Specifically, the features of the BEV grid can be linearly superimposed on the height dimension of the three-dimensional occupancy grid map, and the superimposed three-dimensional occupancy grid represents the second spatial feature based on the point cloud.

[0095] In one embodiment, Figure 4 As shown, the first spatial feature and the second spatial feature are fused according to the feature dimension, including:

[0096] Step S401 : Mapping the first spatial feature and the second spatial feature to a target three-dimensional space of the same scale.

[0097] The target three-dimensional space includes a plurality of voxel grids.

[0098] In some possible implementations, the target three-dimensional space may be established based on an actual three-dimensional scene, and the voxel grid may be aligned with a world coordinate system.

[0099] In some possible implementations, the target three-dimensional space may directly use an image-based three-dimensional space and map the second spatial features to the three-dimensional space. In this case, the voxel grid of the target three-dimensional space may be a forward three-dimensional grid; or, the target three-dimensional space may use a point cloud-based three-dimensional space and map the first spatial features to the three-dimensional space. In this case, the voxel grid of the target three-dimensional space may be a three-dimensional occupied grid.

[0100] Step S402 : splicing the features of the first spatial features and the second spatial features corresponding to the same voxel grid in the target three-dimensional space to obtain splicing features of the voxel grid.

[0101] Among them, the splicing features represent the geometric structure and semantic information of the target three-dimensional space.

[0102] Specifically, after mapping the first spatial feature and the second spatial feature to the same three-dimensional space, the geometric structure and semantic information represented by the first spatial feature and the geometric structure and semantic information represented by the second spatial feature can be spliced ​​into the voxels of the target three-dimensional space. For example, for a certain voxel in the target three-dimensional space, the geometric structure of the voxel is determined to be "occupied" and the semantics is "vehicle" according to the first spatial feature, and the geometric structure of the voxel is determined to be "occupied" and the semantics is "obstacle" according to the second spatial feature. After splicing, the geometric structure of the voxel can be "occupied; occupied" and the semantics is "vehicle; obstacle".

[0103] In one embodiment, Figure 5 As shown, the fusion result is encoded to obtain an encoding result, including:

[0104] Step S501, processing the voxel grid of the target three-dimensional space through a three-dimensional residual network to obtain three-dimensional space features of different scales.

[0105] Among them, the three-dimensional residual network can be a ResNet3D neural network.

[0106] Step S502: fuse three-dimensional spatial features of different scales to obtain cross-scale features, and encode the cross-scale features.

[0107] Specifically, three-dimensional spatial features of different scales can be input into the feature pyramid network, and the low-scale features can be fused with the high-scale features through the feature pyramid network to obtain cross-scale features. Among them, the feature pyramid network can select the FPN3D (Feature Pyramid Network 3D) neural network.

[0108] In some possible implementations, ResNet3D and FPN3D can be connected in series to construct an encoder, and the voxel features in the target three-dimensional space are encoded by the encoder to obtain multi-scale three-dimensional voxel geometric structures and semantic features.

[0109] In one embodiment, Figure 5 As shown, the encoding result is decoded through the preset completion model, including:

[0110] Step S503 , identifying cross-scale features represented by the encoding results through the completion model, upsampling the low-scale features to the scale of the target three-dimensional space for integration, so as to obtain integrated features of the voxel grid.

[0111] Step S504, normalizing the integrated features, and determining the occupancy semantics completion result of the voxel grid according to the normalization result.

[0112] Specifically, the completion model can refine the multi-scale features encoded by the encoder, upsample the low-scale features to the voxel scale of the target three-dimensional space for linear feature superposition, and then obtain the final three-dimensional occupancy semantic scene completion features through three-dimensional convolution. For the features superimposed on the voxel scale of the target three-dimensional space, the vector value corresponding to the geometric structure of each voxel and the vector value corresponding to the semantic feature can be determined through softmax (normalization algorithm), and the index corresponding to the maximum value of the geometric structure vector and the maximum value of the semantic feature vector can be taken to generate the label of the voxel.

[0113] Figure 6 A specific application diagram of a three-dimensional occupancy semantic scene completion method provided by an exemplary embodiment of the present invention. Figure 6 As shown, the application example includes an image preprocessing module, an image semantic encoder, an MLP module, a depth distribution encoder, a forward projection module, a backward projection module, a point cloud preprocessing module, a voxel generation module, a Pillar feature processing module, a three-dimensional occupancy grid mapping module, a three-dimensional feature encoder module and an occupancy semantic completion head module. These modules and application processes are described in detail below.

[0114] The image preprocessing module takes multi-view images as input and performs operations such as removing distortion and resizing each view. The output dimension of the image preprocessing module is [N cams , Ccams , H, W], N cams is the number of cameras, N cams ≥1, C cams is the number of image channels, 1 is a grayscale image, 3 is an RGB image, H and W are the height and width of the image after unification.

[0115] The image semantic encoder uses the ResNet-50 algorithm to extract features from all view images and obtain the global semantic information of multi-view images. The output dimension of the image semantic encoder module is [N cams , C cams_enc , scale×H, scale×W], C cams_enc is the number of feature channels extracted for each perspective image, and scale is the final scaling factor of the ResNet-50 receptive field.

[0116] The input of the MLP module is the camera’s internal parameters [N cams , 3×3] and the registration parameters of the lidar relative to each camera [N cams , 4×4], cascade coupling is performed along the last dimension, and high-dimensional features are extracted through a multi-layer perceptron (one-dimensional convolution). The MLP module can use the lidar / camera parameters to improve the quality of the depth distribution, and its output dimension is [N cams , C m ].

[0117] The depth distribution encoder can reweight image features through Squeeze-Excitation (SE), and the depth distribution D of the i-th image is i , D i =SE(F i |MLP i ), F i is the output of the i-th image of the image semantic encoder, MLP i is the output of the i-th image of the MLP module. The output dimension of the deep distribution encoder is [N cams , C depth_enc , scale×H, scale×W], C depth_enc is the number of depth distribution feature channels.

[0118] The forward projection module can convert the depth distribution information of the image into three-dimensional space based on the LSS (Lift, Splat, Shoot) algorithm. The output dimension of this module is [C forward_lift , Y, X, Z], where Y, X, and Z are the width, length, and height of the three-dimensional voxel grid, respectively.

[0119] The back-projection module is based on the BEV query algorithm of the spatial attention mechanism. It uses the output of the forward projection module and projects it into the BEV space as the initial value of the BEV query, so that it can be better optimized. The output dimension of the back-projection module is [C backward_bev , Y, X].

[0120] The point cloud preprocessing module takes the initial single-frame point cloud synchronized with the multi-view camera as input. Each point in the initial single-frame point cloud has three-dimensional coordinates (x, y, z) and reflection intensity information. The point cloud preprocessing module traverses all points in the initial single-frame point cloud one by one, retains the laser point cloud in the ROI area of ​​interest, and removes the point cloud outside the ROI area.

[0121] The voxel generation module takes the filtered point cloud as input for voxelization. The voxelization process can be divided into two steps. The first step is to divide the filtered point cloud into grids on the x, y planes, and press the point cloud in the z direction into the grid to obtain the index vector of the Pillar voxel grid; the second step is to divide the filtered point cloud into grids on the x, y, z planes, and count the number of points in the grid, mark the grids with a number of points greater than or equal to 1 as occupied, and mark the grids without points as unoccupied, and obtain the index vector of the Occupancy voxel grid.

[0122] The Pillar feature processing module uses PointNet to extract the features belonging to each Pillar voxel grid, and then maps it to the BEV view according to the index of the voxel grid. The output dimension of the Pillar feature processing module is [C lidar_bev , Y, X], C lidar_bev The feature dimension after feature extraction for the Pillar voxel grid.

[0123] The 3D occupancy grid mapping module maps the voxel grid to the 3D grid according to the index of the occupancy voxel grid. The output dimension of the 3D occupancy grid mapping module is [1, Y, X, Z].

[0124] The 3D feature encoder module can be a concatenation of ResNet3D and FPN3D, which can generate multi-scale 3D voxel geometric structures and semantic features. The output dimension of the 3D feature encoder module is [C enc , scale×Y, scale×X, scale×Z].

[0125] Before encoding through the three-dimensional feature encoder module, the outputs of the forward projection module and the backward projection module, as well as the outputs of the three-dimensional occupancy grid mapping module and the Pillar feature processing module can be feature expanded and then cascaded and spliced ​​into the three-dimensional feature encoder module. The feature expansion method can be to linearly superimpose the features of the backward projection module on the height dimension of the forward projection module, and to linearly superimpose the features of the Pillar feature processing module on the Z dimension of the three-dimensional occupancy grid mapping. The dimensions after feature expansion are [C bf , Y, X, Z] and [C lidar , Y, X, Z]. The feature dimension after cascade concatenation is [C fused , Y, X, Z], C fused = C bf + C lidar .

[0126] The occupancy semantics completion head module can refine the multi-scale semantic features output by the 3D feature encoder module. The low-scale features will be upsampled to the [Y, X, Z] scale for linear feature superposition, and then the final 3D occupancy semantic scene completion features will be obtained through 3D convolution, and the feature dimensions are [C+1, Y, X, Z]. After performing softmax calculation on the feature dimensions, the index number of the maximum value in the geometric structure and semantic feature vector of each 3D voxel is taken as the category label of the voxel. After all 3D voxels have undergone this operation, the 3D occupancy semantic completion result of the entire scene can be obtained.

[0127] Figure 7 The following is a schematic diagram of the structure of a three-dimensional scene processing device provided by an exemplary embodiment of the present invention. Figure 7 As shown, the three-dimensional scene processing device 700 may include:

[0128] The acquisition module 701 is used to acquire point cloud and image information corresponding to the three-dimensional scene.

[0129] The feature extraction module 702 is used to extract a first spatial feature corresponding to the three-dimensional scene according to the image information, and to extract a second spatial feature corresponding to the three-dimensional scene according to the point cloud.

[0130] The fusion encoding module 703 is used to fuse the first spatial feature and the second spatial feature according to the feature dimension, and encode the fusion result to obtain an encoding result.

[0131] The decoding and completion module 704 is used to decode the encoding result through a preset completion model to obtain a completion result of the three-dimensional scene.

[0132] In one embodiment, the feature extraction module 702 may include:

[0133] The first acquisition unit can be used to acquire the depth distribution features and semantic features of the image.

[0134] The forward projection unit can be used to perform forward projection based on the depth distribution features and semantic features of images from multiple perspectives to obtain a forward three-dimensional grid.

[0135] The backward projection unit can be used to perform backward projection based on the forward three-dimensional grid, the depth distribution features and semantic features of the images of the multiple perspectives to obtain backward two-dimensional features.

[0136] The first processing unit can be used to process the backward two-dimensional features according to the height dimension of the forward three-dimensional grid to obtain the first spatial features.

[0137] In one embodiment, the first acquisition unit is also used to: acquire camera parameters corresponding to the image, the camera parameters including intrinsic parameters and extrinsic parameters; for images of the same perspective, couple the intrinsic parameters and extrinsic parameters of the image, and extract high-dimensional features based on the coupled parameters; and perform depth encoding on the image based on the high-dimensional features extracted from the image of the same perspective to obtain depth distribution features of the image.

[0138] In one embodiment, the feature extraction module 702 may further include:

[0139] The first mapping unit may be configured to establish a cylindrical voxel grid according to the point cloud, and map features of the cylindrical voxel grid to a two-dimensional grid.

[0140] The grid building unit can be used to build a three-dimensional occupancy grid according to the point cloud.

[0141] The second processing unit may be configured to process the features in the two-dimensional grid according to the height dimension of the three-dimensional occupied grid to obtain a second spatial feature.

[0142] In one embodiment, the fusion coding module 703 may include:

[0143] The second mapping unit may be configured to map the first spatial feature and the second spatial feature to a target three-dimensional space of the same scale, where the target three-dimensional space includes a plurality of voxel grids.

[0144] The feature stitching unit can be used to stitch the features of the first spatial feature and the second spatial feature corresponding to the same voxel grid in the target three-dimensional space to obtain the stitching features of the voxel grid, and the stitching features represent the geometric structure and semantic information of the target three-dimensional space.

[0145] In one embodiment, the fusion encoding module 703 is also used to: process the voxel grid of the target three-dimensional space through a three-dimensional residual network to obtain three-dimensional space features of different scales; fuse the three-dimensional space features of different scales to obtain cross-scale features, and encode the cross-scale features.

[0146] In one embodiment, the decoding completion module 704 may include:

[0147] The identification integration unit can be used to identify the cross-scale features represented by the encoding result through the completion model, and upsample the low-scale features to the scale of the target three-dimensional space for integration to obtain the integrated features of the voxel grid.

[0148] The normalization processing unit can be used to perform normalization processing on the integrated features and determine the occupancy semantic completion result of the voxel grid according to the normalization processing result.

[0149] In one embodiment, the fusion encoding module 703 is further used to: input the three-dimensional spatial features of different scales into a feature pyramid network, and fuse low-scale features with high-scale features through the feature pyramid network to obtain cross-scale features.

[0150] The three-dimensional scene processing device provided in this embodiment is used to execute the technical solution in any of the aforementioned method embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0151] It should be understood that the above device embodiments are only illustrative, and the device of the present invention can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0152] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present invention may be integrated into one unit / module, or each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0153] Figure 8 FIG. 1 is a schematic diagram of a structure of an electronic device provided by an exemplary embodiment of the present invention. Figure 8 As shown, the electronic device 80 includes:

[0154] Processor 81, memory 82, and communication interface 83;

[0155] The memory 82 is used to store executable instructions of the processor 81; the executable instructions may be computer-executable instructions;

[0156] The processor 81 is configured to execute the technical solution in any of the aforementioned method embodiments by executing the executable instructions.

[0157] Optionally, the memory 82 can be independent or integrated with the processor 81.

[0158] Optionally, when the memory 82 is a device independent of the processor 81, the electronic device 80 may further include:

[0159] The bus 84 , the memory 82 and the communication interface 83 are connected to the processor 81 via the bus 84 and communicate with each other. The communication interface 83 is used to communicate with other devices.

[0160] Optionally, the communication interface 83 may be implemented by a transceiver. The communication interface is used to implement communication between the database access device and other devices (such as a client, a read-write library, and a read-only library). The memory may include a random access memory (RAM) and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage.

[0161] The bus 84 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0162] The above-mentioned processor can be a general-purpose processor, including a central processing unit CPU, a network processor (NP), etc.; it can also be a digital signal processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0163] The electronic device is used to execute the technical solution in any of the aforementioned method embodiments, and its implementation principle and technical effect are similar and will not be repeated here.

[0164] An embodiment of the present invention further provides a readable storage medium, which may be a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the technical solution provided by any of the aforementioned method embodiments is implemented.

[0165] An embodiment of the present invention further provides a computer program product, including a computer program, which is used to implement the technical solution provided by any of the aforementioned method embodiments when executed by a processor.

[0166] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0167] In the above embodiments, the description of each embodiment has its own emphasis. For the part not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0168] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The description and examples are to be considered exemplary only, and the true scope and spirit of the present invention is indicated by the following claims.

[0169] The above embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Any equivalent substitution or change made by a person skilled in the art based on the present invention is within the protection scope of the present invention.

Claims

1. A three-dimensional scene processing method, characterized in that: include: Obtain point cloud and image information corresponding to the three-dimensional scene; Extracting a first spatial feature corresponding to the three-dimensional scene according to the image information, and extracting a second spatial feature corresponding to the three-dimensional scene according to the point cloud; fusing the first spatial feature and the second spatial feature according to the feature dimension, and encoding the fusion result to obtain an encoding result; The encoding result is decoded by using a preset completion model to obtain a completion result of the three-dimensional scene.

2. The three-dimensional scene processing method according to claim 1, characterized in that: The image information includes images of the three-dimensional scene corresponding to multiple viewing angles, and extracting the first spatial feature corresponding to the three-dimensional scene according to the image information includes: Acquire depth distribution features and semantic features of the image; Perform forward projection based on the depth distribution features and semantic features of images from multiple perspectives to obtain a forward three-dimensional mesh; Performing backward projection based on the forward three-dimensional grid, the depth distribution features and the semantic features of the images of the multiple perspectives to obtain backward two-dimensional features; The backward two-dimensional features are processed according to the height dimension of the forward three-dimensional grid to obtain first spatial features.

3. The three-dimensional scene processing method according to claim 2, characterized in that: The acquiring the depth distribution feature of the image comprises: Obtaining camera parameters corresponding to the image, wherein the camera parameters include intrinsic parameters and extrinsic parameters; For an image of the same viewing angle, the intrinsic parameters and extrinsic parameters of the image are coupled, and high-dimensional features are extracted according to the coupled parameters; According to the high-dimensional features of the image extracted from the same viewing angle, the image is depth-encoded to obtain the depth distribution features of the image.

4. The three-dimensional scene processing method according to any one of claims 1 to 3, characterized in that: The extracting a second spatial feature corresponding to the three-dimensional scene according to the point cloud includes: Establishing a cylindrical voxel grid based on the point cloud, and mapping features of the cylindrical voxel grid to a two-dimensional grid; establishing a three-dimensional occupancy grid based on the point cloud; The features in the two-dimensional grid are processed according to the height dimension of the three-dimensional occupied grid to obtain second spatial features.

5. The three-dimensional scene processing method according to any one of claims 1 to 3, characterized in that: The fusing the first spatial feature and the second spatial feature according to the feature dimension includes: Mapping the first spatial feature and the second spatial feature to a target three-dimensional space of the same scale, wherein the target three-dimensional space includes a plurality of voxel grids; The features of the first spatial features and the second spatial features corresponding to the same voxel grid in the target three-dimensional space are spliced ​​to obtain spliced ​​features of the voxel grid, wherein the spliced ​​features represent the geometric structure and semantic information of the target three-dimensional space.

6. The three-dimensional scene processing method according to claim 5, characterized in that: The step of encoding the fusion result to obtain an encoding result includes: Processing the voxel grid of the target three-dimensional space through a three-dimensional residual network to obtain three-dimensional space features of different scales; The three-dimensional spatial features of different scales are fused to obtain cross-scale features, and the cross-scale features are encoded.

7. The three-dimensional scene processing method according to claim 6, characterized in that: The decoding of the encoding result by using a preset completion model includes: Identifying cross-scale features represented by the encoding result through the completion model, upsampling low-scale features to the scale of the target three-dimensional space for integration, so as to obtain integrated features of the voxel grid; The integrated features are normalized, and the occupancy semantics completion result of the voxel grid is determined according to the normalization result.

8. The three-dimensional scene processing method according to claim 6, characterized in that: The fusing of the three-dimensional spatial features of different scales to obtain cross-scale features includes: The three-dimensional spatial features of different scales are input into a feature pyramid network, and low-scale features are fused with high-scale features through the feature pyramid network to obtain cross-scale features.

9. A three-dimensional scene processing device, characterized in that: include: An acquisition module is used to obtain point cloud and image information corresponding to the three-dimensional scene; A feature extraction module, configured to extract a first spatial feature corresponding to the three-dimensional scene according to the image information, and to extract a second spatial feature corresponding to the three-dimensional scene according to the point cloud; A fusion encoding module, used for fusing the first spatial feature and the second spatial feature according to a feature dimension, and encoding the fusion result to obtain an encoding result; The decoding and completion module is used to decode the encoding result through a preset completion model to obtain a completion result of the three-dimensional scene.

10. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 8 when executed by a processor.

12. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 8 when being executed by a processor.

Images