Semantic scene completion method based on image and point cloud fusion in automatic driving scene

Abstract

A kind of semantic scene completion method based on image and point cloud fusion under automatic driving scene, comprising the following steps: image and point cloud data are used as input, through the semantic scene completion network built, image and point cloud information are fused, and the network is trained by minimizing target loss function;Image and point cloud data from acquisition equipment are input into the trained network, to obtain the semantic segmentation result of each object in the scene and the scene completion result.This method can improve the classification accuracy and positioning accuracy of semantic scene completion task under automatic driving scene, and also has the advantages of low cost, high robustness, convenient operation and the like.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and autonomous driving, and in particular to a semantic scene completion method based on image and point cloud fusion in autonomous driving scenarios. Background Technology

[0002] Automated driving is a significant application area in computer vision. However, during autonomous driving, complex road conditions and severe occlusion between vehicles make it crucial to complete the scene for vehicles and pedestrians that are difficult to observe directly. Therefore, semantic scene completion technology is needed to address this issue. With the development of autonomous driving technology, semantic scene completion has become one of the most critical technologies in the process. Utilizing advanced computer vision technology and fusing point cloud and image information modalities to complete semantic scene is a major problem that needs to be solved in the field of autonomous driving.

[0003] Semantic scene completion technology is a technique that uses partially observed information to complete the overall information of a scene. The goal of this technology is to achieve a complete geometric and semantic understanding of the environment. This technique couples point cloud semantic segmentation with voxel scene completion, achieving scene semanticization while supplementing the complete geometric shapes of objects. Using the semantic information of objects, drivers can better understand the types of surrounding objects; using the complete geometric shape information of objects, drivers can understand the spatial geometric relationship between the vehicle and objects, thus improving safe driving. Over the past decade, autonomous driving has attracted global interest as a promising technology. Especially after the emergence and rapid development of deep learning, the revolution in computer-assisted autonomous driving technology has officially begun and has generated significant benefits in practical applications by many automotive companies.

[0004] In real-world driving scenarios, drivers can often quickly and accurately judge the position and shape of nearby occluded objects based on experience. However, for distant occluded objects, especially in high-speed driving environments (such as highways) and in adverse weather conditions (such as rain or fog), drivers often find it difficult to make quick and accurate decisions. This can lead to serious personal safety issues. Image-based methods can accurately segment distant objects that are difficult for the human eye to discern using high-precision cameras, but they struggle to obtain location information and are easily affected by ambient lighting conditions. Point cloud-based methods can accurately measure the distance to surrounding objects and obtain precise depth information using high-line-count radar, but they are ineffective for occluded objects. Based on this background information, it can be concluded that semantic scene completion remains a highly valuable and challenging task, with the main difficulties as follows:

[0005] 1) Due to the influence of severe weather and lighting, image-based semantic scene completion methods are unable to obtain sufficient visual information. Using images to complete scenes is essentially an ill-posed estimation problem, which cannot achieve high-performance results.

[0006] 2) Due to occlusion, point cloud-based methods have difficulty accurately estimating distant objects. In addition, point cloud-based methods lack RGB color information and cannot accurately identify object types.

[0007] 3) In autonomous driving scenarios, adverse weather conditions and unexpected situations on the road cannot be ignored, and high-precision, high-performance methods are needed to ensure the safety of personnel.

[0008] It should be noted that the information disclosed in the background section above is only for understanding the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0009] The main objective of this invention is to overcome the shortcomings of the aforementioned background technology and provide a semantic scene completion method based on image and point cloud fusion in autonomous driving scenarios, thereby improving the classification accuracy and localization accuracy of semantic scene completion tasks.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] A semantic scene completion method based on image and point cloud fusion for autonomous driving scenarios includes the following steps:

[0012] Image and point cloud data are used as inputs. A semantic scene completion network is built to fuse image and point cloud information. The network is trained by minimizing the target loss function.

[0013] Images and point cloud data from the acquisition device are input into the trained network to obtain semantic segmentation results for each object in the scene and scene completion results.

[0014] Furthermore:

[0015] The images and point cloud data are preprocessed before input to reduce the impact of noise.

[0016] The image and point cloud data are sampled from the furthest point before input, and the point cloud data is downsampled; preferably, the sampling from the furthest point includes the following steps:

[0017] 1) Read the point cloud set P = {p1,2,…, n};

[0018] 2) Randomly select a point ps ;

[0019] 3) In set Pp s Find the match with p s The farthest point p i ;

[0020] 4) In set Pp s -p i Find the match with p s The farthest point p of i l ;

[0021] This process continues until a predetermined set of k points is selected.

[0022] The semantic scene completion network is a deep network model based on image point cloud fusion, comprising an image semantic feature extraction module, an image point cloud mapping module, a point cloud deep completion module, and a semantic scene completion module. Specifically, the image semantic feature extraction module processes the input image using a residual learning network and a fully connected network to obtain semantic segmentation features. The image point cloud mapping module projects each point cloud in three-dimensional space onto the two-dimensional space of the image by performing a projection transformation, establishing a connection and coupling between the point cloud data in three-dimensional space and the image data in two-dimensional space. The point cloud deep completion module generates a dense point cloud and fuses semantic information with it. The semantic scene completion module generates a dense voxel-based semantic scene completion result based on the point cloud containing semantic information.

[0023] The image semantic feature extraction module incorporates local residual learning and global skip connections into the network.

[0024] The image point cloud mapping module uses the camera's intrinsic and extrinsic parameter matrices to project the point cloud in three-dimensional space onto the two-dimensional space of the image. First, the extrinsic parameter matrix is ​​used to map the point cloud to its camera coordinate system, and then the intrinsic parameter matrix is ​​used to transform the point cloud from three-dimensional space to two-dimensional space.

[0025] The point cloud depth completion module uses the image point cloud mapping module to match the original image information and sparse original point cloud information, and then performs depth estimation on each pixel through the dense pixels on the image to generate a dense point cloud in the image space, and fuses semantic information with the point cloud.

[0026] The depth estimation process employs a two-branch architecture: one branch is dominated by image information, and the other by depth information. A dense depth map is output from the image-information-dominated branch. This dense depth map is then used as input to the depth-information-dominated branch to optimize and calibrate the predicted depth map. Based on the category features of each point in the image obtained by the image semantic feature extraction module, the image features are concatenated with the depth information. Then, through the inverse operation of the image point cloud mapping module, the resulting dense depth map is projected back into the three-dimensional space of the point cloud, yielding point cloud data with semantic information.

[0027] The semantic scene completion module is based on a voxel network and uses a three-dimensional convolutional neural network with an encoder-decoder structure with skip connections to extract global information or supplement information for scene completion. During the prediction process, it outputs multi-scale prediction results and optimizes them from coarse to fine to obtain the final semantic segmentation result and voxelized scene.

[0028] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the semantic scene completion method.

[0029] The beneficial effects of this invention compared to the prior art include:

[0030] This invention proposes a semantic scene completion method based on image and point cloud fusion for autonomous driving scenarios, significantly improving the classification accuracy and localization accuracy of deep learning technology in semantic scene completion tasks in autonomous driving. This invention can serve as a key technology for autonomous driving assistance systems. The proposed image and point cloud fusion-based semantic scene completion method provides the driver with information about the vehicle's surroundings, including object categories, geometric shapes, and distances, based on the input image and point cloud information. This facilitates more accurate judgment and driving by the driver, reducing the probability of accidents caused by adverse weather conditions and human visual impairment, and improving driving safety. The method also boasts advantages such as low cost, high robustness, and ease of operation. Attached Figure Description

[0031] Figure 1 This is a simplified flowchart of a semantic scene completion method based on image and point cloud fusion in an autonomous driving scenario according to an embodiment of the present invention.

[0032] Figure 2 This is a schematic diagram showing the four main modules of a semantic scene completion method according to an embodiment of the present invention.

[0033] Figure 3This is a schematic diagram illustrating the working mode of the four main modules of the semantic scene completion method based on image and point cloud fusion in an autonomous driving scenario according to an embodiment of the present invention. Detailed Implementation

[0034] The present patent application will now be described in further detail with reference to specific embodiments and the accompanying drawings. It should be emphasized that the following description is merely exemplary and is not intended to limit the scope or application of the present patent application.

[0035] Referring to the above figures, non-limiting and non-exclusive embodiments will be described, wherein the same reference numerals denote the same parts unless otherwise specifically stated.

[0036] Those skilled in the art will recognize that numerous variations are possible with respect to the above description, and therefore the embodiments are merely illustrative of one or more specific implementations.

[0037] See Figure 1 This invention provides a semantic scene completion method based on point cloud image fusion, comprising the following steps:

[0038] Step 1: Read and preprocess the training dataset. This mainly involves reading data from collected datasets or public datasets for autonomous driving and downsampling the point cloud.

[0039] The second step involves taking the preprocessed image point cloud data as input, using a semantic scene completion network to fuse the image and point cloud information, and training the network parameters by minimizing the target loss function.

[0040] Step 3: Input the images and point cloud data from the acquisition device into the trained network model to obtain the semantic segmentation results of each object in the scene and the scene completion results. This step can output the voxelized scene around the vehicle.

[0041] The input data types include both images and point clouds. In some embodiments, the training dataset is a publicly available large-scale dataset or a self-collected dataset.

[0042] In a preferred embodiment, the image and point cloud data are sampled from the furthest point before input, and the point cloud data is downsampled.

[0043] In a preferred embodiment, the semantic scene completion network comprises four modules: an image feature extraction module, an image point cloud mapping module, a point cloud depth completion module, and a semantic scene completion module. The semantic scene completion network is a deep network model based on image point cloud fusion, including the image semantic feature extraction module, the image point cloud mapping module, the point cloud depth completion module, and the semantic scene completion module. Specifically, the image semantic feature extraction module processes the input image using a residual learning network and a fully connected network to obtain semantic segmentation features. The image point cloud mapping module projects each point cloud in three-dimensional space onto the two-dimensional space of the image by performing a projection transformation, establishing a connection and coupling between the point cloud data in three-dimensional space and the image data in two-dimensional space. The point cloud depth completion module generates a dense point cloud and fuses semantic information with the point cloud. The semantic scene completion module generates a dense voxel-based semantic scene completion result based on the point cloud containing semantic information.

[0044] The following describes specific embodiments of the present invention.

[0045] In some specific embodiments, the semantic scene completion method based on image and point cloud fusion in autonomous driving scenarios of the present invention includes the following steps:

[0046] The input point cloud and images undergo preprocessing. In some embodiments, the KITTI dataset, jointly created by the Karlsruhe Institute of Technology in Germany and Toyota Research Institute of America, is used. It is currently the most influential computer vision algorithm evaluation dataset for autonomous driving scenarios internationally. KITTI contains real-world image data collected in urban, rural, and highway scenarios, with up to 15 vehicles and 30 pedestrians per image, and varying degrees of occlusion and truncation. Downsampling of the point cloud is performed because the collected point cloud data volume reaches millions, making direct processing difficult. Appropriately reducing the data has a negligible impact on performance but significantly reduces the model size, making it better suited for real-world applications.

[0047] Using deep learning technology, a deep network model based on image point cloud fusion is constructed. The network model includes four parts: an image semantic feature extraction module, an image point cloud mapping module, a point cloud depth completion module, and a semantic scene completion module.

[0048] In the image semantic feature extraction module, the input image is processed by a residual learning network and a fully connected network to obtain the semantic segmentation features of the image. In addition, in order to preserve the original information as much as possible, local residual learning and global skip connections are added to the network to increase the information flow in the network.

[0049] In the image point cloud mapping module, in order to fuse image and point cloud information in the future, a medium for communication between the two different modalities of point cloud and image is established. By using the camera's intrinsic and extrinsic parameter matrix, each point cloud in the three-dimensional space is projected and transformed onto the two-dimensional space of the image, thus establishing a connection and coupling between the point cloud data in the 3D space and the image data in the 2D space.

[0050] In the point cloud depth completion module, the original image information and sparse original point cloud information are input. The image point cloud mapping module is used to match the two information. Then, the depth of each pixel is estimated through the dense pixels on the image to generate a dense point cloud in the image space. The semantic information is then fused with the point cloud.

[0051] In the semantic scene completion module, the previously obtained point cloud with semantic information is input, and a dense voxelized semantic scene completion result is output. Because it incorporates additional semantic information from the image, the segmentation result is more accurate, and the addition of more point cloud data improves the accuracy of the completion.

[0052] Semantic scene completion is a computer vision task that combines 3D shape completion and 3D semantic segmentation. It helps robots perceive the 3D world and interact with their environment. In this embodiment of the invention, semantic scene completion is performed on nineteen common objects in autonomous driving scenarios, including cars, bicycles, motorcycles, trucks, other vehicles, pedestrians, bicycle drivers, motorcycle drivers, roads, parking lots, sidewalks, other ground surfaces, buildings, fences, vegetation, tree trunks, terrain, utility poles, and traffic signs. In this embodiment of the invention, the input point cloud and image are preprocessed to reduce the impact of noise and better represent their image features. Secondly, deep learning technology is used to build a deep network model based on image-point cloud fusion. This network model includes four parts: image semantic feature extraction, image point cloud mapping, point cloud depth completion, and semantic scene completion. In the image semantic feature extraction process, the input image is processed by a residual learning network and a fully connected network to obtain the semantic segmentation features of the image. Next, in the image point cloud mapping part, the point cloud data in 3D space and the image data in 2D space are connected and coupled through intrinsic and extrinsic parameter matrices. Then, in the point cloud depth completion part, a dense point cloud is generated and semantic information is fused with the point cloud. Finally, in the semantic scene completion part, the previously obtained point cloud with semantic information is input, and a dense voxelized semantic scene completion result is output. In this embodiment of the invention, a configuration of one monocular camera and one 48-line LiDAR is used to fully utilize the semantic information of the image and the depth information of the point cloud. This embodiment of the invention has the advantages of low cost, high robustness, and convenient operation.

[0053] Data reading and preprocessing:

[0054] The point cloud is downsampled by reading data from collected datasets or publicly available datasets for autonomous driving. This is achieved by downsampling the point cloud using the farthest point sampling method. This is because the collected point cloud data volume reaches millions, making direct processing difficult. Appropriately reducing the data has a negligible impact on performance but significantly reduces the model size, making it better suited for real-world applications.

[0055] The steps for sampling k points using the farthest point sampling method are as follows:

[0056] 1) Read the point cloud set P = {p1, p2, ..., p n};

[0057] 2) Randomly select a point p s ;

[0058] 3) In set Pp s Find the match with p s The farthest point p i ;

[0059] 4) In set Pp s -p i Find the match with p s ,p i The farthest point p l ;

[0060] This process continues until k points are selected from the set.

[0061] Semantic scene completion network:

[0062] like Figure 2 As shown, the overall structure of the semantic scene completion network can be divided into four modules: image semantic feature extraction module, image point cloud mapping module, point cloud depth completion module, and semantic scene completion module. Figure 3 The diagram shown illustrates how these four modules work:

[0063] A. Image semantic feature extraction module

[0064] The feature extraction part uses a pre-trained semantic segmentation network. Utilizing semantic labels and image data of nineteen common categories in autonomous driving scenarios provided by the KITTI dataset, a 101-layer residual network is trained, and F is defined as... E (·) is a function of this processing procedure, so the extracted two-dimensional semantic segmentation features are:

[0065] x E =F E (x)

[0066]

[0067] Where x is the input raw image data, x E This represents the semantic segmentation results for each image obtained through the image semantic feature extraction module.

[0068] B. Image point cloud mapping module

[0069] To facilitate the subsequent fusion of image and point cloud information, a medium for communication between the two different modalities of point cloud and image is established. Using the camera's intrinsic and extrinsic parameter matrices, each point cloud in 3D space is projected onto the 2D space of the image through a projection transformation. First, the extrinsic parameter matrix is ​​used to align the point cloud to the camera coordinate system; then, the intrinsic parameter matrix is ​​used to transform the point cloud from 3D space to 2D space. The specific implementation formula is as follows:

[0070]

[0071] C. Point Cloud Depth Completion Module

[0072] The system takes raw image information and sparse raw point cloud information as input. Using an image-point cloud mapping module, the two information are mapped, and then depth estimation is performed on each pixel using the dense pixels in the image. In the depth estimation process, this invention adopts a two-branch architecture: one branch is dominated by image information, and the other by depth information. First, a dense depth map is output from the image-independent branch. Then, this dense depth map is used as input to optimize and calibrate the predicted depth map through the depth-independent branch. Since the category features of each point in the image have already been obtained in the image semantic feature extraction module, the image features are first concatenated with the depth information.

[0073]

[0074] Then, through the inverse operation of the image point cloud mapping module, the obtained dense depth map is projected back into the three-dimensional space of the point cloud to obtain point cloud data with semantic information.

[0075] D. Semantic Scene Completion Module

[0076] The semantically information-rich point cloud obtained earlier is input into the semantic scene completion module, which outputs a dense voxel-based semantically complete scene. This module is based on a voxel network and uses a 3D convolutional neural network with an encoder-decoder structure with skip connections, which is beneficial for extracting global information and can also supplement the scene completion information. During the prediction process, multi-scale prediction results are output and optimized from coarse to fine to obtain the final semantic segmentation result and voxel-based scene.

[0077] Semantic scene completion implementation:

[0078] By inputting the acquired images and point cloud data into a pre-trained semantic scene completion network based on point cloud image fusion, high-quality semantic segmentation results and voxelized scenes can be obtained. The embodiments of this invention have been tested and achieved an accuracy of 71.7% and mIoU = 24.1 on the semanticKITTI dataset.

[0079] The above-described specific embodiments fully illustrate the semantic scene completion method based on image and point cloud fusion in the autonomous driving scenario of this application.

[0080] The applications and advantages of the embodiments of the present invention are mainly reflected in the following aspects:

[0081] 1) Using deep learning methods, a new deep neural network model is constructed to complete the semantic scene completion of common objects in nineteen autonomous driving scenarios, including cars, bicycles, motorcycles, trucks, other vehicles, pedestrians, bicycle drivers, motorcycle drivers, roads, parking lots, sidewalks, other ground surfaces, buildings, fences, vegetation, tree trunks, terrain, utility poles, and traffic signs.

[0082] 2) Given the input image and point cloud data, the model can classify objects, complete their shapes, and acquire and calculate their distances. This facilitates safe driving for vehicle drivers.

[0083] 3) Obtain information such as semantic segmentation results and scene completion results around the vehicle through semantic scene completion technology that can be used in autonomous driving assistance systems.

[0084] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0085] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0086] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0087] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0088] The background section of this invention may include background information about the problems or environment in which the invention is being developed, and is not necessarily a description of prior art. Therefore, the content included in the background section does not constitute an admission of prior art by the applicant.

[0089] The above description provides a further detailed explanation of the present invention in conjunction with specific / preferred embodiments, and it should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various substitutions or modifications can be made to these described embodiments without departing from the concept of the present invention, and all such substitutions or modifications should be considered within the scope of protection of the present invention. In the description of this specification, the reference to terms such as "an embodiment," "some embodiments," "preferred embodiment," "example," "specific example," or "some examples," etc., indicates that the specific features, structures, materials, or characteristics described in connection with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples. Although the embodiments of the present invention and their advantages have been described in detail, it should be understood that various changes, substitutions, and modifications can be made herein without departing from the scope of protection of the patent application.

Claims

1. A semantic scene completion method based on image and point cloud fusion in autonomous driving scenarios, characterized in that, Includes the following steps: Image and point cloud data are used as input, and a semantic scene completion network is constructed to fuse image and point cloud information. The network is trained by minimizing the target loss function. The semantic scene completion network is a deep network model based on image-point cloud fusion, including an image semantic feature extraction module, an image point cloud mapping module, a point cloud deep completion module, and a semantic scene completion module. The image semantic feature extraction module processes the input image using a residual learning network and a fully connected network to obtain semantic segmentation features. The image point cloud mapping module projects each point cloud in three-dimensional space... The process involves projecting a point cloud from three-dimensional space onto the two-dimensional space of an image, establishing a connection and coupling between the point cloud data in three-dimensional space and the image data in two-dimensional space. The point cloud depth completion module generates a dense point cloud and fuses semantic information with the point cloud. The semantic scene completion module generates a dense voxel-based scene completion result with semantic information based on the point cloud containing semantic information. Specifically, the point cloud depth completion module uses the image point cloud mapping module to map the original image information and sparse original point cloud information to generate a dense point cloud in image space and fuses semantic information with the point cloud. Images and point cloud data from the acquisition device are input into the trained network to obtain semantic segmentation results for each object in the scene and scene completion results.

2. The semantic scene completion method as described in claim 1, characterized in that, The images and point cloud data are preprocessed before input to reduce the impact of noise.

3. The semantic scene completion method as described in claim 1, characterized in that, The image and point cloud data are sampled from the furthest point before input, and the point cloud data is downsampled; the furthest point sampling includes the following steps: 1) Read the point cloud set ; 2) Randomly select a point ; 3) In the set Find with The farthest point ; 4) In the set Find with The farthest point ; This process continues until a predetermined set of k points is selected.

4. The semantic scene completion method as described in any one of claims 1 to 3, characterized in that, The image semantic feature extraction module incorporates local residual learning and global skip connections into the network.

5. The semantic scene completion method as described in any one of claims 1 to 3, characterized in that, The image point cloud mapping module uses the camera's intrinsic and extrinsic parameter matrices to project the point cloud in three-dimensional space onto the two-dimensional space of the image. First, the extrinsic parameter matrix is ​​used to map the point cloud to its camera coordinate system, and then the intrinsic parameter matrix is ​​used to transform the point cloud from three-dimensional space to two-dimensional space.

6. The semantic scene completion method as described in any one of claims 1 to 3, characterized in that, The point cloud depth completion module estimates the depth of each pixel in the image by using the dense pixels in the image, thereby generating a dense point cloud in the image space.

7. The semantic scene completion method as described in claim 6, characterized in that, The depth estimation process employs a two-branch architecture: one branch is dominated by image information, and the other by depth information. A dense depth map is output from the image-information-dominated branch. This dense depth map is then used as input to the depth-information-dominated branch to optimize and calibrate the predicted depth map. Based on the category features of each point in the image obtained by the image semantic feature extraction module, the image features are concatenated with the depth information. Then, through the inverse operation of the image point cloud mapping module, the resulting dense depth map is projected back into the three-dimensional space of the point cloud, yielding point cloud data with semantic information.

8. The semantic scene completion method as described in any one of claims 1 to 3, characterized in that, The semantic scene completion module is based on a voxel network and uses a three-dimensional convolutional neural network with an encoder-decoder structure with skip connections to extract global information or supplement information for scene completion. During the prediction process, it outputs multi-scale prediction results and optimizes them from coarse to fine to obtain the final semantic segmentation result and voxelized scene.

9. A computer-readable storage medium storing a computer program, characterized in that, The computer program, when executed by a processor, implements the semantic scene completion method as described in any one of claims 1 to 8.

Images