Three-dimensional reconstruction method and device

Abstract

The present application provides a 3D reconstruction method and apparatus, relating to the field of computer vision. The method comprises converting a planar region in an initial depth map into a corresponding planar point cloud based on the target semantic label of the image. The plane parameters of the plane are determined based on the plane point cloud. Furthermore, the corresponding 3D points in space corresponding to the pixels of the planar region in the initial depth map are obtained based on the plane parameters to construct a point cloud corresponding to the planar region in the image. The solution of the present application can improve the accuracy and density of 3D reconstruction of planar regions.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular to a three-dimensional reconstruction method and device. Background Art

[0002] Computer vision is an integral part of various intelligent / autonomous systems in various application areas, such as manufacturing, inspection, document analysis, medical diagnosis, and the military. It involves the use of cameras and computers to acquire the data and information we need about the objects being photographed. Figuratively speaking, it's like equipping computers with eyes (cameras) and brains (algorithms) to replace the human eye in identifying, tracking, and measuring objects, thus enabling computers to perceive their environment. Because perception can be viewed as extracting information from sensory signals, computer vision can also be considered the study of how artificial systems can "perceive" from images or multidimensional data. In short, computer vision uses various imaging systems to replace the visual organs to acquire input information, and then computers replace the brain to process and interpret this input information. The ultimate goal of computer vision research is to enable computers to observe and understand the world through vision like humans do, and to have the ability to autonomously adapt to their environments.

[0003] With the development of augmented reality (AR) technology, new AR applications are emerging in an endless stream. Among them, interactions related to real-world scenes are inseparable from the three-dimensional (3D) dense reconstruction of the scene. The current mainstream 3D dense reconstruction solutions mainly include laser-based 3D dense reconstruction and image-based 3D dense reconstruction. Laser-based 3D dense reconstruction uses a laser device to scan the environment to obtain a highly accurate depth map, and then fuses the depth map to obtain the final dense point cloud. The point cloud obtained by this solution is very accurate. However, the laser device is large in size and the use process is cumbersome. The laser-based 3D dense reconstruction solution has disadvantages such as high cost and low efficiency. In the image-based 3D dense reconstruction solution, an acquisition device, such as a mobile phone, is used to capture an image sequence, and then a dense point cloud is obtained through algorithm processing. This solution has the advantages of high acquisition efficiency, low cost, high accuracy, and support for tight coupling of 3D mapping and positioning, and has a very broad application prospect. Currently, there are many open source solutions that can achieve image-based 3D dense reconstruction, such as OpenMVS, colmap, PMVS, MVSNet, etc.

[0004] However, in existing image-based 3D dense reconstruction schemes, the estimated depth values ​​in weakly textured areas such as planar areas are inaccurate and will be filtered out during the filtering stage, resulting in holes in the point cloud obtained after fusion, making it impossible to reconstruct an accurate point cloud.

[0005] Therefore, how to improve the three-dimensional reconstruction effect of planar areas has become an urgent problem to be solved. Summary of the Invention

[0006] The present application provides a three-dimensional reconstruction method and device, which can improve the three-dimensional reconstruction effect of a planar area.

[0007] In a first aspect, a three-dimensional reconstruction method is provided, including: obtaining an initial depth map corresponding to a plurality of frame images; obtaining a target semantic label of the initial depth map corresponding to the multi-frame images, the target semantic label including a first semantic label, the first semantic label being used to indicate that pixels with the first semantic label in the initial depth map corresponding to the multi-frame images belong to a first plane; converting pixels with the first semantic label in the initial depth map corresponding to the multi-frame images into three-dimensional points to obtain a point cloud of the first plane; converting the initial depth map corresponding to the multi-frame images into a point cloud corresponding to the multi-frame images, wherein the multi-frame images include a first image, the three-dimensional point corresponding to the first pixel in the first image is the intersection of a first straight line and a first plane, the first pixel has a first semantic label, and the first straight line is determined according to the posture of the first image and the position of the first pixel in the first image; and fusing the point clouds corresponding to the multi-frame images to obtain a fused target point cloud.

[0008] In the solution of the embodiment of the present application, the planar area in the initial depth map is converted into a corresponding planar point cloud according to the target semantic label of the image, the plane parameters of the plane can be determined according to the point cloud of the plane, and then the three-dimensional points corresponding to the pixels in the planar area in space are obtained according to the plane parameters. In this way, more accurate three-dimensional points corresponding to the planar area can be obtained, avoiding the appearance of holes in the point cloud of the planar area with weak texture, which is conducive to improving the accuracy and density of three-dimensional reconstruction.

[0009] Moreover, two-dimensional image data is easier to collect. The multiple frames of images in the embodiment of the present application are two-dimensional images, which helps to reduce data collection costs.

[0010] A two-dimensional image refers to a planar image that does not include depth information, for example, an RGB image.

[0011] The initial depth map corresponding to the multi-frame images is obtained by processing the multi-frame images. The pixels in the multi-frame images correspond one-to-one to the pixels in the initial depth map.

[0012] Exemplarily, the semantic label of the image may be a plane semantic label, and the plane semantic label of a pixel is used to indicate the plane to which the pixel belongs.

[0013] For example, if the value of a pixel's plane semantic label is 0, then the pixel is not located on a plane. If the value of a pixel's plane semantic label is non-zero, then the pixel is located on a plane. Furthermore, if the values ​​of two pixels' plane semantic labels are the same, then the two pixels are located on the same plane.

[0014] The pixels with the first semantic label may be located in one frame of image or in multiple frames of image.

[0015] There is a corresponding relationship between the point cloud of the plane and the target semantic label. The point cloud of the first plane can also be understood as the point cloud corresponding to the first semantic label.

[0016] In combination with the first aspect, in certain implementations of the first aspect, converting an initial depth map corresponding to multiple frame images into a point cloud corresponding to the multiple frame images includes: updating the depth value of the initial depth map corresponding to the multiple frame images to obtain a target depth map corresponding to the multiple frame images, wherein the depth value of the first pixel in the target depth map corresponding to the first image is obtained by projecting the three-dimensional point corresponding to the first pixel onto the target depth map corresponding to the first image based on the posture of the first image; and converting the target depth map corresponding to the multiple frame images into a point cloud corresponding to the multiple frame images.

[0017] According to the solution of the embodiment of the present application, the planar area in the initial depth map is converted into the corresponding point cloud according to the semantic label, and then the actual depth value of the pixel in the planar area is obtained according to the plane parameters of the point cloud to update the depth value of the planar area in the initial depth map, thereby obtaining more accurate depth information in the planar area with weak texture, which is conducive to improving the accuracy and density of three-dimensional reconstruction.

[0018] In combination with the first aspect, in some implementations of the first aspect, the target semantic label is obtained by performing image semantic segmentation processing on multiple frames of images.

[0019] In combination with the first aspect, in certain implementations of the first aspect, multiple frames of images belong to an image sequence, and the target semantic label is obtained by merging initial semantic labels in the image sequence, and the initial semantic labels of the images in the image sequence are obtained by performing image semantic segmentation processing on the images, wherein the first semantic label is determined based on the semantic label obtained by merging the first initial semantic label in the first image and the second semantic label in the second image in the image sequence, and the semantic label obtained after the merger is obtained by merging the first initial semantic label and the second initial semantic label when the first area corresponding to the first initial semantic label in the first image and the second area corresponding to the second initial semantic label in the second image at least partially overlap.

[0020] According to the solution of the embodiment of the present application, the initial semantic labels in the image sequence are merged to obtain a global label, and the global label of the multi-frame image is used as the target semantic label, so as to extract three-dimensional points belonging to the same plane from the multi-frame image, that is, the pixels belonging to the same plane are converted into the same point cloud, which increases the number of pixels corresponding to a semantic label, solves the problem that there may be fewer effective pixels or inaccurate pixel values ​​in the area corresponding to some plane semantic labels that may exist in a single-frame image, improves the accuracy of the point cloud obtained after conversion, and then improves the accuracy of the plane parameters, and further improves the accuracy of three-dimensional reconstruction.

[0021] The initial semantic labels of the first image include a first initial semantic label, and the initial semantic labels of the second image include a second initial semantic label.

[0022] In combination with the first aspect, in certain implementations of the first aspect, multiple frames of images belong to an image sequence, and the target semantic label is obtained by merging initial semantic labels in the image sequence, and the initial semantic labels of the images in the image sequence are obtained by performing image semantic segmentation processing on the images, wherein the first semantic label is determined based on the semantic label obtained by merging the first initial semantic label in the first image and the second semantic label in the second image adjacent to the first image in the image sequence, and the semantic label obtained after the merger is obtained by merging the first initial semantic label and the second initial semantic label when the first area corresponding to the first initial semantic label in the first image and the second area corresponding to the second initial semantic label in the second image at least partially overlap.

[0023] In combination with the first aspect, in certain implementations of the first aspect, the semantic label obtained after merging is obtained by merging the first initial semantic label and the second initial semantic label when the first area corresponding to the first initial semantic label in the first image and the second area corresponding to the second initial semantic label in the second image at least partially overlap, including: the semantic label obtained after merging is obtained by merging the first initial semantic label and the second initial semantic label when the ratio of the number of pixels in the overlapping area in the first area to the number of pixels in the first area is greater than or equal to a first threshold, and / or the ratio of the number of pixels in the overlapping area in the second area to the number of pixels in the second area is greater than or equal to the first threshold.

[0024] According to the solution of the embodiment of the present application, the initial semantic labels are merged when the number of pixels in the overlapping area meets the conditions, which improves the accuracy of the label merging result and thus improves the accuracy of three-dimensional reconstruction.

[0025] In combination with the first aspect, in certain implementations of the first aspect, the plane parameters of the point cloud of the first plane include: a normal vector of the first plane and a target three-dimensional point in the first plane.

[0026] Exemplarily, the normal vector of the plane corresponding to the point cloud can be determined by a PCA algorithm.

[0027] Exemplarily, the target three-dimensional point in the first plane may be any three-dimensional point in the point cloud of the first plane.

[0028] In combination with the first aspect, in certain implementations of the first aspect, the target three-dimensional point in the first plane is obtained by calculating an average value of the three-dimensional points in the point cloud of the first plane.

[0029] According to the solution of the embodiment of the present application, the target three-dimensional point in the first plane is obtained by calculating the average value of the three-dimensional points in the point cloud of the first plane, thereby improving the accuracy of the plane parameters and thereby improving the accuracy of the three-dimensional reconstruction.

[0030] In combination with the first aspect, in certain implementations of the first aspect, pixels with a first semantic label in an initial depth map corresponding to multiple frames of images are converted into three-dimensional points to obtain a point cloud of a first plane, including: converting pixels with a first semantic label in an initial depth map corresponding to multiple frames of images into three-dimensional points, and filtering the three-dimensional points obtained after the conversion to obtain a point cloud of the first plane.

[0031] For example, statistical filtering and radius filtering are performed on the point cloud obtained after conversion, which can remove peripheral flying points and improve the accuracy of the point cloud.

[0032] In combination with the first aspect, in some implementations of the first aspect, the pose of the first image includes the optical center position of a device that captures the first image.

[0033] In combination with the first aspect, in certain implementations of the first aspect, the first straight line is determined based on the optical center position of the device that captures the first image and the position of the first three-dimensional point, and the first three-dimensional point is obtained by converting the first pixel into space based on the position of the first pixel in the first image and the initial depth value of the first pixel.

[0034] In a second aspect, a three-dimensional reconstruction device is provided, which includes a module or unit for executing the method in the above-mentioned first aspect and any one of the implementations of the first aspect.

[0035] It should be understood that the expansion, limitation, explanation and description of the relevant content in the above-mentioned first aspect also apply to the same content in the second aspect.

[0036] In a third aspect, a three-dimensional reconstruction device is provided, which includes: a memory for storing programs; a processor for executing the programs stored in the memory, and when the program stored in the memory is executed, the processor is used to execute the method in the first aspect and any one of the implementation methods of the first aspect.

[0037] According to a fourth aspect, a computer-readable medium is provided, which stores a program code for execution by a device, wherein the program code includes a method for executing the first aspect and any one of the implementations of the first aspect.

[0038] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the method in the first aspect and any one of the implementations of the first aspect.

[0039] In a sixth aspect, a chip is provided, comprising a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface and executes the method in the first aspect and any one of the implementations of the first aspect.

[0040] Optionally, as an implementation method, the chip may also include a memory, in which instructions are stored, and the processor is used to execute the instructions stored on the memory. When the instructions are executed, the processor is used to execute the method in the first aspect and any one of the implementation methods of the first aspect.

[0041] In a seventh aspect, an electronic device is provided, which includes the three-dimensional reconstruction device in the above-mentioned second aspect and any one of the implementation methods of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 Schematic rendering of virtual objects;

[0043] Figure 2 This is a schematic rendering of AR navigation;

[0044] Figure 3 This is a schematic effect diagram of the interaction between virtual and real occlusion;

[0045] Figure 4 A schematic block diagram of a three-dimensional reconstruction device provided in an embodiment of the present application;

[0046] Figure 5 A schematic flow chart of another three-dimensional reconstruction device provided in an embodiment of the present application;

[0047] Figure 6 A schematic flow chart of a three-dimensional reconstruction method provided in an embodiment of the present application;

[0048] Figure 7 A schematic diagram of the depth completion process provided in an embodiment of the present application;

[0049] Figure 8 A schematic flowchart of constructing a sparse point cloud according to an embodiment of the present application;

[0050] Figure 9 A comparison of point clouds corresponding to images obtained by different solutions provided in the embodiments of this application;

[0051] Figure 10 A comparison of point clouds corresponding to image sequences obtained by the existing solution provided in the embodiments of the present application and the solution of the present application;

[0052] Figure 11 A comparison chart of the virtual and real occlusion effects of point clouds obtained by different solutions provided in the embodiments of this application;

[0053] Figure 12 A comparison of point clouds corresponding to image sequences obtained by the two solutions of the present application provided in the embodiments of the present application;

[0054] Figure 13 A schematic block diagram of a three-dimensional reconstruction device provided in an embodiment of the present application;

[0055] Figure 14 A schematic block diagram of another three-dimensional reconstruction device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] The technical solution in this application will be described below with reference to the accompanying drawings.

[0057] 3D reconstruction is the process of restoring a 3D model of an object in a scene by using images taken at different angles. 3D reconstruction can be achieved using color images or depth images. Depth images are collected by depth image sensors (also called depth sensors), such as time of flight (TOF) sensors, light detection and ranging (LiDAR) devices, or structured light sensors. The pixel value of each pixel in the depth image is used to represent the distance (or depth) from the depth sensor to the corresponding point in the scene. For example, a depth sensor can transmit a signal to the scene and measure the distance based on the time difference between the transmitted signal and the reflected signal. The depth sensor can provide direct, fast, and accurate depth information.

[0058] Color images are captured by color image sensors (also called color sensors), such as conventional red, green, and blue (RGB) cameras, digital single-lens reflex (DSLR) cameras, webcams, mobile phones, tablets, or electronic devices such as near-eye displays. The pixel value of each pixel in a color image is used to represent the color of a specific point in the scene. For example, color images can be RGB images, black and white images, or grayscale images. Although color images themselves do not contain depth information, the depth information of a specific point in the scene can be determined through triangulation techniques using multiple color images acquired at different angles.

[0059] Specifically, 3D models can be represented by 3D point clouds. A point cloud is a collection of points on an object's surface, containing the three-dimensional coordinates of those points. Point clouds can be categorized as either sparse or dense. Sparse point clouds have fewer points and larger spacing between them. Dense point clouds have more points and smaller spacing between them.

[0060] The 3D reconstruction method in the embodiment of the present application can also be understood as a method for constructing a 3D dense point cloud, and a 3D dense point cloud of an object or scene is obtained through 3D reconstruction.

[0061] The method of the present application can be applied to three-dimensional reconstruction scenarios such as virtual object rendering, AR navigation, and virtual-reality occlusion interaction and rendering. The following describes the three scenarios of virtual object rendering, AR navigation, and virtual-reality occlusion interaction and rendering respectively.

[0062] (1) Virtual object rendering

[0063] Figure 1 A schematic diagram of virtual object rendering is shown. In AR applications, it is often necessary to render the reconstructed objects or scenes so that users can view the complete objects or scenes from a global perspective, such as Figure 1 shown.

[0064] The method in the embodiment of the present application can improve the 3D reconstruction effect in weak texture areas, that is, improve the accuracy of 3D dense point clouds in weak texture areas.

[0065] Exemplarily, the solution of the embodiment of the present application can be applied to various computing devices, such as near-eye display devices or terminal devices. For example, the solution of the embodiment of the present application can be built into the terminal, that is, the terminal executes the method of the embodiment of the present application to achieve virtual object rendering. Alternatively, the solution of the embodiment of the present application can be deployed in an application (APP) on the terminal, that is, the method of the embodiment of the present application is executed by calling the APP to achieve virtual object rendering.

[0066] (2) AR Navigation

[0067] Figure 2 A schematic diagram of AR navigation is shown, as Figure 2 As shown in Figure 1, AR navigation uses AR to achieve real-world navigation. The foundation of AR navigation is 3D reconstruction and visual positioning service (VPS), which uses 3D reconstruction to build maps and VPS to achieve high-precision positioning.

[0068] The solution of the embodiment of the present application can build a more accurate map (i.e., a 3D dense point cloud) for AR navigation, thereby improving the positioning effect of the VPS, thereby greatly improving the user experience.

[0069] Exemplarily, the solution of the embodiment of the present application can be applied to various computing devices, such as near-eye display devices or terminal devices. For example, the solution of the embodiment of the present application can be built into the terminal, that is, the terminal executes the method of the embodiment of the present application to provide AR navigation services for users and improve user experience. Alternatively, the solution of the embodiment of the present application can be deployed in an APP on the terminal, that is, the method of the embodiment of the present application is executed by calling the APP to provide AR navigation services for users and improve user experience.

[0070] (3) Virtual and real occlusion interaction and rendering

[0071] Figure 3 A schematic diagram of virtual and real occlusion interaction and rendering is shown. AR applications focus on the interaction between virtual and real, for example, Figure 3The interaction between the virtual dinosaur, virtual basketball and real scene shown in the figure. In order to obtain a more realistic interactive experience, the effect of 3D reconstruction needs to be guaranteed. For example, in order to achieve the occlusion effect of the virtual dinosaur, it is necessary to ensure the reconstruction effect of the area in front of the virtual dinosaur, that is, the 3D dense point cloud is dense and accurate enough. If the area in front of the virtual dinosaur is a weak texture area, the reconstruction effect of the weak texture area is poor, that is, there are holes in the 3D dense point cloud, and the occlusion effect of the virtual dinosaur cannot be achieved. For another example, in order to achieve the collision effect of the virtual basketball, it is necessary to ensure the reconstruction effect of the area around the virtual basketball, the 3D dense point cloud is dense and accurate enough. If the area around the virtual basketball is a weak texture area, the reconstruction effect of the weak texture area is poor, that is, there are holes in the 3D dense point cloud, and the collision effect of the virtual basketball cannot be achieved.

[0072] The method provided in the embodiment of the present application can improve the 3D reconstruction effect in weak texture areas, that is, improve the effect of 3D dense point cloud in weak texture areas. By rendering the generated 3D dense point cloud, it can ensure that users can also obtain a good virtual-reality interaction experience in areas with weak textures.

[0073] Exemplarily, the solution of the embodiment of the present application can be applied to various computing devices, such as near-eye display devices or terminal devices. For example, the solution of the embodiment of the present application can be built into the terminal, that is, the terminal executes the method of the embodiment of the present application to achieve the interaction between virtual and reality, providing users with a better virtual-reality interaction experience. Alternatively, the solution of the embodiment of the present application can be deployed in an APP on the terminal, that is, the method of the embodiment of the present application is executed by calling the APP to achieve the interaction between virtual and reality, providing users with a better virtual-reality interaction experience.

[0074] Laser-based 3D reconstruction methods can obtain 3D point clouds with high precision, but there are problems such as high reconstruction cost and low efficiency. Image-based 3D reconstruction methods have the advantages of high acquisition efficiency, low cost, and high precision, but the 3D point cloud obtained by this method is prone to form holes in weak texture areas, and the reconstruction effect is poor. Among them, weak texture areas refer to areas where there are no significant texture features. Specifically, the pixels in the weak texture areas are relatively similar in color and brightness, that is, the pixels are indistinguishable. For example, weak texture areas can include planar areas, such as walls, floors, or ceilings.

[0075] Existing image-based 3D reconstruction methods are essentially based on the consistent matching of feature points or feature blocks. However, in weakly textured areas, this matching is unreliable. During the subsequent verification of the accuracy of the depth values, the depth values ​​of the points in the weakly textured areas are generally deleted, resulting in holes in the final 3D dense point cloud. For example, in the openMVS scheme, the depth map of each frame is first estimated based on the similarity matching between image blocks. Then, information such as neighboring images is used for verification, and the depth values ​​of unreliable pixels in the depth map are filtered out. Finally, the filtered depth maps are fused to obtain a 3D dense point cloud. The core of this method is to utilize the similarity between image blocks. However, in weakly textured areas, an image block in one image may have a high similarity with multiple image blocks in another image. This can lead to inaccurate estimated depth values, which will be filtered out during the filtering stage, forming holes in the 3D dense point cloud.

[0076] The embodiment of the present application proposes a three-dimensional reconstruction method, which can obtain a more accurate 3D dense point cloud in a weak texture area, thereby improving the three-dimensional reconstruction effect in the weak texture area.

[0077] In order to better describe the method of the embodiment of the present application, the three-dimensional reconstruction device of the embodiment of the present application is described below with reference to the accompanying drawings.

[0078] Figure 4 A three-dimensional reconstruction device according to an embodiment of the present application is shown. Figure 4 As shown, the apparatus 400 includes an image pose and sparse point cloud estimation module 410 , a shape semantic segmentation module 420 , and a dense point cloud reconstruction module 430 .

[0079] The image pose and sparse point cloud estimation module 410 is used to obtain the pose and sparse point cloud of the input image sequence.

[0080] The image sequence is a two-dimensional image sequence, and a two-dimensional image refers to an image that does not include a depth value. In the embodiment of the present application, a two-dimensional image can also be understood as the aforementioned color image, for example, a two-dimensional image can be an RGB image, and the image sequence can be an RGB image sequence.

[0081] An image sequence consists of multiple frames captured at different angles. The pose of an image sequence refers to the pose of the frames in the sequence, that is, the pose of the camera that captured the frames. A sparse point cloud refers to the sparse point cloud corresponding to the image sequence. A sparse point cloud can also be called a sparse map point.

[0082] For example, the pose and sparse point cloud of the image sequence can be obtained by a structure from motion (SFM) method. In this case, the image pose and sparse point cloud estimation module 410 can also be referred to as an SFM module 410.

[0083] The image pose and sparse point cloud estimation module 410 inputs the output image sequence pose and sparse point cloud into the dense point cloud reconstruction module 430. It should be noted that the image pose and sparse point cloud estimation module 410 is an optional module, and the dense point cloud reconstruction module 430 can also obtain the image sequence pose and sparse point cloud from other devices.

[0084] The shape semantic segmentation module 420 is used to perform semantic segmentation processing on the images in the image sequence to obtain initial semantic labels of the images.

[0085] Exemplarily, the semantic segmentation processing in the embodiment of the present application can be a plane semantic segmentation processing. Specifically, a plane label value is assigned to each pixel in the image to obtain a plane semantic label, so that the plane area in the image can be extracted. The plane semantic label of a pixel is used to indicate the plane to which the pixel belongs. The plane to which the pixel belongs refers to the plane to which the three-dimensional point corresponding to the pixel belongs. For example, if the value of the plane semantic label of a pixel is 0, the pixel is not located on the plane, and if the value of the plane semantic label of the pixel is a non-zero value, the pixel is located on the plane. Furthermore, if the values ​​of the plane semantic labels of two pixels are the same, the two pixels are located on the same plane.

[0086] The shape semantic segmentation module 420 inputs the output image's initial semantic label as the image's target semantic label into the dense point cloud reconstruction module 430. It should be noted that the shape semantic segmentation module 420 is an optional module, and the dense point cloud reconstruction module 430 can also obtain the image's target semantic label from other devices.

[0087] The dense point cloud reconstruction module 430 is used to obtain the three-dimensional point cloud corresponding to the image sequence based on the pose and sparse point cloud of the image sequence and the target semantic label of the image. The three-dimensional point cloud is a dense point cloud, and the number of points in the three-dimensional point cloud is far greater than the number of points in the sparse point cloud obtained by module 410.

[0088] Figure 5 FIG2 shows a schematic diagram of another three-dimensional reconstruction device in an embodiment of the present application. Figure 5 (a) and (b) show two 3D reconstruction devices, which can be regarded as Figure 4 Two specific implementations of the device in .

[0089] The apparatus 400 includes an image pose and sparse point cloud estimation module 410, a shape semantic segmentation module 420, and a dense point cloud reconstruction module 430. Figure 5 As shown in (a), the dense point cloud reconstruction module 430 includes a depth map estimation filtering module 431, a 3D plane fitting module 432, a depth map completion module 433 and a depth map fusion module 434.

[0090] The depth map estimation filtering module 431 is used to estimate the depth map corresponding to the image sequence based on the image poses and the pose and sparse point cloud of the image sequence provided by the sparse point cloud estimation module 410. Furthermore, the depth map estimation filtering module 431 can filter the estimated depth map to obtain a filtered depth map, which can be used as the initial depth map corresponding to the image sequence. There is a one-to-one correspondence between the images in the image sequence and the initial depth map.

[0091] The depth map estimation filter module 431 inputs the initial depth map corresponding to the output image sequence into the 3D depth map completion module 433. It should be noted that the depth map estimation filter module 431 is an optional module, that is, the depth map completion module 433 can obtain the initial depth map corresponding to the image sequence from other devices.

[0092] The 3D plane fitting module 432 is used to obtain a 3D plane according to the initial depth map and the target semantic label. The 3D plane can be determined by the plane parameters of the 3D plane, and the 3D plane fitting module 432 can also be understood as being used to obtain the plane parameters of the 3D plane.

[0093] Specifically, the 3D plane fitting module 432 is configured to convert the depth values ​​corresponding to the plane semantic labels into a point cloud. That is, the pixels with the plane semantic labels are converted into a point cloud. The point cloud is the point cloud of the 3D plane indicated by the plane semantic label, and the 3D plane parameters are estimated based on the point cloud. For example, the 3D plane parameters can be estimated using principal component analysis (PCA).

[0094] The depth map completion module 433 is used to update the depth values ​​in the initial depth map according to the 3D plane parameters obtained by the 3D plane fitting module 430 . The updated depth map is the target depth map.

[0095] The depth map fusion module 434 is used to convert the pixels in the target depth map obtained by the depth map completion module 450 into 3D points. The set of 3D points is the reconstructed three-dimensional point cloud, which is a dense point cloud.

[0096] That is, the method of the embodiment of the present application adds a 3D plane fitting module and a depth map completion module to the process of multi-view stereo (MVS) dense reconstruction.

[0097] like Figure 5 As shown in (b), the device 400 may further include a label merging module 440.

[0098] The label merging module 440 is used to merge the initial semantic labels of multiple frames in the image sequence to obtain the merged semantic label as the target semantic label of the multiple frames, that is, the global label.

[0099] That is, the initial semantic labels of the same plane in the indication space in different images in the image sequence are merged into the same label.

[0100] The input of the label merging module 440 is the initial semantic labels of multiple frames of images in the image sequence, wherein the initial semantic labels of multiple frames of images in the image sequence can be input into the label merging module 440 by the shape semantic segmentation module 420, or can be obtained by the plane label merging module 440 from other devices.

[0101] The label merging module 440 outputs the target semantic labels of the multiple frames to the 3D plane fitting module 432. That is, when the apparatus 400 includes the label merging module 440, the 3D plane fitting module 432 can obtain plane parameters according to the initial depth map and the target semantic labels.

[0102] The following combination Figures 6 to 12 The three-dimensional reconstruction method of the embodiment of the present application is described.

[0103] Figure 6 A three-dimensional reconstruction method 600 provided in an embodiment of the present application is shown. The method of the embodiment of the present application can be applied to various computing devices, which can be electronic devices or servers. The electronic devices may include, but are not limited to, personal computers, cameras, server computers, handheld or laptop devices, mobile devices, computers, mobile robots, near-eye display devices, or drones. The mobile devices may include mobile phones, tablet computers, personal digital assistants, or media players. The near-eye display devices may include AR glasses or AR helmets.

[0104] In one possible implementation, an electronic device executes the 3D reconstruction method in the embodiments of the present application to achieve 3D reconstruction of an object or scene. In another possible implementation, the electronic device may send image data to another device, such as a cloud server or terminal device, which then executes the 3D reconstruction method in the embodiments of the present application. Furthermore, the other device may send the processed results to the electronic device.

[0105] For example, the solution of the embodiment of the present application can be built into a computing device, for example, built into a terminal, that is, the method of the embodiment of the present application is executed by the terminal. Alternatively, the solution of the embodiment of the present application can be deployed in an application (application, APP) on a computing device, for example, deployed in an APP on a terminal, that is, the method of the embodiment of the present application is executed by calling the APP.

[0106] For example, the method 600 may be performed by Figure 4 or Figure 5 Executed by the device 400 in .

[0107] The method 600 includes steps S610 to S650 , which are described below.

[0108] S610: Obtain an initial depth map corresponding to multiple frames of images.

[0109] The multiple frames of images are two-dimensional images, which refer to planar images that do not include depth information, namely the aforementioned color images, such as RGB images.

[0110] Exemplarily, the multiple frames of images are images in an image sequence. The image sequence is a two-dimensional image sequence, for example, the image sequence may be an RGB image sequence.

[0111] The image sequence includes multiple frames of images captured for the same scene or the same object. For example, the image sequence may include multiple frames of images captured from different viewpoints.

[0112] Step S610 may include receiving an initial depth map corresponding to the multiple frames of image from another device; or, step S610 may also include processing the multiple frames of image to obtain an initial depth map corresponding to the multiple frames of image, for example, obtaining the initial depth map using an MVS method. A detailed description is provided below and is not repeated here. This embodiment of the present application does not limit the specific method of "obtaining" in step S610.

[0113] The initial depth map corresponding to the multi-frame image is obtained by processing the multi-frame image. There is a one-to-one correspondence between the pixels in the multi-frame image and the pixels in the initial depth map. The pixel value of a pixel in a frame of the multi-frame image can be used to represent the color information of a point in space, and the pixel value of the corresponding pixel in the initial depth map corresponding to the frame image can be used to represent the distance information between the point in space and the device that captured the image. The pixel value of a pixel in the depth map can also be called the depth value of the pixel.

[0114] S620: Obtain target semantic labels of the initial depth maps corresponding to the multiple frames of images.

[0115] The target semantic label includes a first semantic label, where the first semantic label is used to indicate that the pixel with the first semantic label belongs to a first plane.

[0116] As mentioned above, the pixels in the multi-frame image correspond one-to-one to the pixels in the initial depth map. The target semantic label of the initial depth map corresponding to the multi-frame image is the target semantic label of the multi-frame image. If a pixel in the multi-frame image has a certain semantic label, the corresponding pixel in the initial depth map corresponding to the multi-frame image also has the semantic label. In other words, the target semantic label of the initial depth map in the embodiment of the present application can be determined based on the target semantic label of the multi-frame image. For the sake of ease of description, only the target semantic label of the multi-frame image will be used for explanation in the following text.

[0117] Step S620 may include: receiving the target semantic labels of the multiple frames of images from another device; or step S620 may also include: processing the multiple frames of images to obtain the target semantic labels of the multiple frames of images, for example, by performing semantic segmentation on the images to obtain the target semantic labels of the images. This embodiment of the application does not limit the specific method of "obtaining" in step S620.

[0118] The semantic label of an image refers to the semantic label of the pixels in the image. The pixel semantic label is used to indicate the category to which the pixel belongs. A frame of an image can contain one or more semantic labels. Pixels with the same semantic label belong to the same category.

[0119] For example, the semantic label of an image can be a plane semantic label. The plane semantic label of a pixel is used to indicate the plane to which the pixel belongs. The plane to which the pixel belongs can be understood as the plane to which the three-dimensional point corresponding to the pixel belongs in space, such as a wall, floor, or ceiling. Plane semantic labels can be used to distinguish planar regions in an image.

[0120] The value of a pixel's plane semantic label can be set as needed. For example, if the value of a pixel's plane semantic label is 0, then the pixel is not located on a plane. If the value of a pixel's plane semantic label is non-zero, then the pixel is located on a plane. Furthermore, if two pixels have the same plane semantic label value, then the two pixels are located on the same plane.

[0121] In the embodiment of the present application, method 600 is described only by taking the semantic label of the image as a plane semantic label as an example, which does not constitute a limitation on the embodiment of the present application, as long as the semantic label of the image can indicate the plane area in the image.

[0122] The following describes the target semantic labels of multiple frames of images using two specific implementation methods (method 1 and method 2) as examples.

[0123] Method 1

[0124] The target semantic labels of the multi-frame images are obtained by performing image semantic segmentation processing on the multi-frame images.

[0125] Optionally, the image semantic segmentation processing may be plane semantic segmentation processing.

[0126] Exemplarily, step S620 may be performed by the shape semantic segmentation module 420 in the apparatus 400 .

[0127] Specifically, the semantic segmentation model performs plane semantic segmentation on the image. This assigns a plane label value to each pixel in the image, resulting in a plane semantic label. The processed image shows the plane areas, thus extracting the plane areas from the image. For example, the plane semantic label of pixels in the plane area is non-zero, while the plane semantic label of pixels in the non-plane area is zero.

[0128] Exemplarily, the semantic segmentation model includes a feature encoder, a feature decoder, and a classifier. The feature encoder is used to perform downsampling processing, and the feature decoder is used to perform upsampling processing. For example, the planar semantic segmentation model can be a network model such as a fully convolutional neural network (FCN), SegNet, or Deeplab.

[0129] The following uses the FCN model as an example to illustrate the plane semantic segmentation process.

[0130] (1) Feature encoding of the image input to the FCN model.

[0131] The feature encoding process can also be understood as a downsampling process.

[0132] Specifically, the feature encoder in FCN extracts abstract semantic features from the input image.

[0133] For example, network structures such as VGGNet, ResNet, ResNeXt, and DenseNet can be used as feature encoders to extract planar semantic features.

[0134] (2) Feature decoding of the encoded result.

[0135] The feature decoding process can also be understood as an upsampling process.

[0136] Specifically, the feature decoder in FCN upsamples the semantic features extracted by different layers of the feature encoder to obtain semantic features of the same resolution, and then performs feature fusion on the semantic features of the same resolution to obtain decoded semantic features.

[0137] (3) Classify based on the decoded semantic features.

[0138] The classifier in FCN classifies the decoded semantic features to obtain the category corresponding to each pixel in the image. The category corresponding to each pixel is indicated by the planar semantic label of each pixel.

[0139] Exemplarily, the classifier may be a softmax classifier.

[0140] However, the feature expression capability of the FCN model is insufficient, and image details will be lost during the feature encoding and decoding stages, resulting in inaccurate image segmentation.

[0141] In the examples of this application, DeepLab-v3+ is used for planar semantic segmentation. The DeepLab-v3+ network model incorporates a spatial multi-pyramid pooling (ASPP) module based on dilated convolution, which introduces multi-scale information and further integrates low-level features with high-level features, improving the accuracy of semantic segmentation. Furthermore, the introduction of deep separable convolution processing in the ASPP module and feature decoding module improves the processing speed and robustness of the model.

[0142] The semantic labels obtained by image semantic segmentation can also be called initial semantic labels. That is, in method 1, the initial semantic labels of the multi-frame images are used as target semantic labels of the multi-frame images.

[0143] The target semantic labels of the multi-frame images are obtained based on semantic segmentation processing of the images themselves. However, the image semantic segmentation results may be inaccurate. For example, the same plane area in the image may be divided into two plane areas; or, in the initial depth map, the areas corresponding to some plane semantic labels may have fewer valid pixels or inaccurate pixel values, which leads to a decrease in the accuracy of the obtained plane parameters and affects the accuracy of 3D reconstruction.

[0144] Method 2

[0145] Method 2 provides a method for obtaining the target semantic label of the multi-frame image, merging the initial semantic labels of the multi-frame image so that the pixels belonging to the same plane in the multi-frame image have the same semantic label, or in other words, replacing different semantic labels indicating the same plane in the multi-frame image with the same semantic label.

[0146] Optionally, the multiple frames of images belong to an image sequence. The target semantic labels of the multiple frames of images are obtained by merging initial semantic labels in the image sequence, and the initial semantic labels of the images in the image sequence are obtained by performing image semantic segmentation on the images.

[0147] The first semantic label is determined based on a semantic label obtained by merging a first initial semantic label in the first image and a second semantic label in the second image in the image sequence. The merged semantic label is obtained by merging the first initial semantic label and the second initial semantic label when a first area corresponding to the first initial semantic label in the first image and a second area corresponding to the second initial semantic label in the second image at least partially overlap.

[0148] Optionally, the first image and the second image are two adjacent frames of images in the image sequence.

[0149] The initial semantic labels of the first image include a first initial semantic label, and the initial semantic labels of the second image include a second initial semantic label.

[0150] That is to say, if there is an overlapping area between the areas corresponding to the two initial semantic labels in the two frames of images, the planes indicated by the two initial semantic labels can be considered to be the same plane, and the two initial semantic labels are merged into the same label. In other words, the same label is assigned to the pixels in the areas corresponding to the two initial semantic labels in the two frames of images. This same label can be called a global label.

[0151] For example, using the order of the image sequence, take the first image A and the second image B. If there is an overlap between the area corresponding to label x (an example of the first initial semantic label) in image A (an example of the first image) and the area corresponding to label y (an example of the second initial semantic label) in image B (an example of the second image), then the pixels in the area corresponding to label x and the area corresponding to label y are assigned the same global label z.

[0152] Exemplarily, step S620 may be performed by the label merging module 440 in the apparatus 400 .

[0153] By processing all images in the image sequence in this manner, labels for all images in the sequence can be merged to obtain target semantic labels for all images in the sequence. In this case, pixels with the same target semantic label in different images in the sequence reside in the same plane region. That is, pixels with the same target semantic label in different images, after being converted into 3D points, belong to the same plane.

[0154] The initial semantic labels of an image can be obtained by performing image semantic segmentation on the image. That is, the initial semantic labels of the first image are obtained by performing image semantic segmentation on the first image, and the initial semantic labels of the second image are obtained by performing image semantic segmentation on the second image. For example, the initial semantic labels of an image can be obtained using Method 1; alternatively, the initial semantic labels of an image can be obtained from other devices, which is not limited in this embodiment of the present application.

[0155] The region corresponding to an initial semantic label in the image refers to the region where the pixels with the initial semantic label are located. In other words, all pixels in the region corresponding to an initial semantic label in the image have the initial semantic label.

[0156] The first region in the first image refers to a region in the first image where pixels having the first initial semantic label are located, or it can be understood that all pixels in the first region in the first image have the first initial semantic label.

[0157] The second region in the second image refers to a region in the second image where pixels having the second initial semantic label are located, or it can be understood that all pixels in the second region in the second image have the second initial semantic label.

[0158] Furthermore, when the ratio of the number of pixels in the overlapping area in the first area to the number of pixels in the first area is greater than or equal to a first threshold, and / or the ratio of the number of pixels in the overlapping area in the second area to the number of pixels in the second area is greater than or equal to the first threshold, the first initial semantic label and the second initial semantic label are replaced with the same semantic label.

[0159] That is to say, if there is an overlapping area between the areas corresponding to the two initial semantic labels in the two frames of images, and the ratio of the number of pixels in the overlapping area to the number of pixels in the area corresponding to one of the initial semantic labels is greater than or equal to the first threshold, then the planes indicated by the two initial semantic labels can be considered to be the same plane, and the two initial semantic labels are merged into the same label, or in other words, the pixels in the areas corresponding to the two initial semantic labels in the two frames of images are assigned the same label, and the same label can be called a global label.

[0160] For example, using the order of image sequences, take the first image A and the second image B. If there is overlap between the region corresponding to label x (an example of the first initial semantic label) in image A (an example of the first image) and the region corresponding to label y (an example of the second initial semantic label) in image B (an example of the second image), and the ratio of the number of pixels in the overlapping region to the number of pixels in the region corresponding to label x is greater than or equal to a first threshold, or the ratio of the number of pixels in the overlapping region to the number of pixels in the region corresponding to label y is greater than or equal to the first threshold, then the plane indicated by label x in image A and label y in image B can be considered to be the same plane. The same global label z is assigned to the pixels in the region corresponding to label x and the region corresponding to label y.

[0161] In this way, the initial semantic labels are merged when the number of pixels in the overlapping area meets the conditions, which improves the accuracy of the label merging results and thus improves the accuracy of 3D reconstruction.

[0162] It should be understood that the above method is only an example, and other conditions can be set to achieve label merging, which is not limited in the embodiments of the present application. For example, if the number of pixels in the overlapping area of ​​the first area is greater than or equal to the second threshold, and / or the number of pixels in the overlapping area of ​​the second area is greater than or equal to the second threshold, the first initial semantic label and the second initial semantic label are replaced with the same semantic label.

[0163] S630: Convert pixels with a first semantic label in the initial depth map corresponding to the multiple frames of images into three-dimensional points to obtain a point cloud of a first plane.

[0164] The coordinates of the three-dimensional point corresponding to the pixel in the depth map can be calculated based on the intrinsic and extrinsic parameters of the camera and the depth value of the pixel in the depth map, that is, the pixel in the depth map is converted into the corresponding three-dimensional point.

[0165] A plane in space can be represented by a corresponding point cloud, and the plane parameters of a plane in space can be determined by the plane parameters of the point cloud of the plane.

[0166] Based on the different target semantic labels of the multi-frame images, the pixels corresponding to the different target semantic labels in the initial depth map can be converted into point clouds of different planes. Alternatively, it can be understood that different planar regions in the initial depth map corresponding to the multi-frame images are converted into different point clouds. If the multi-frame images include multiple planar regions, multiple point clouds can be obtained after conversion to spatial points.

[0167] The target semantic label corresponds one-to-one to the point cloud of the plane. The first semantic label is used to indicate the first plane, and the point cloud corresponding to the first semantic label is the point cloud of the first plane. The point cloud of the first plane is the point cloud corresponding to the first semantic label. Pixels with the first semantic label belong to the first plane, and the set of 3D points obtained after conversion to 3D points is the point cloud of the first plane. The plane parameters of the first plane can be determined based on the plane parameters of the point cloud of the first plane.

[0168] Exemplarily, step S630 may be performed by the 3D plane fitting module 432 in the apparatus 400 .

[0169] As mentioned above, the target semantic label may be a plane semantic label. For example, the area where pixels with a plane semantic label value of 0 are located is a non-plane area, and the area where pixels with a plane semantic label value of non-zero are located is a plane area.

[0170] In this case, the pixels whose plane semantic labels have non-zero values ​​can be converted into point clouds, that is, the pixels in the plane area can be converted into point clouds.

[0171] In one implementation, the target semantic label is obtained by method 1 in step S620. The target semantic label in each frame of the multi-frame image is independent, and step S630 can be for processing the multi-frame image separately. Taking one frame of the multi-frame image as an example, step S630 can include: converting the pixels in the initial depth map corresponding to the frame image into point clouds of different planes according to the different target semantic labels of the frame image. For example, the pixels with the first semantic label are located in the first image, and step S630 can be: converting the pixels with the first semantic label in the initial depth map corresponding to the first image into point clouds of the first plane.

[0172] In another implementation, the target semantic label is obtained by method 2 in step S620. Different images in the multi-frame image may have the same target semantic label. Step S630 may include: converting the pixels in the initial depth map corresponding to the multi-frame image into point clouds of different planes according to the different target semantic labels of the multi-frame image. In other words, converting the pixels with the same target semantic label in the multi-image into point clouds of the same plane. In this case, the pixels with the first semantic label may be located in the multi-frame image, and step S630 may be: converting the pixels with the first semantic label in the initial depth map corresponding to the multi-frame image into point clouds of the first plane. In other words, the pixels with the first semantic label may be located in one frame image or in multiple frames image.

[0173] In this way, the global label of the multiple images is used as the target semantic label to extract three-dimensional points belonging to the same plane from the multiple frame images, that is, the pixels belonging to the same plane are converted into the same point cloud, which increases the number of pixels corresponding to a semantic label, solves the problem that there may be some plane semantic labels in a single frame image that correspond to areas with fewer effective pixels or inaccurate pixel values, improves the accuracy of the point cloud obtained after conversion, and then improves the accuracy of the plane parameters, and further improves the accuracy of three-dimensional reconstruction.

[0174] Optionally, pixels with a first semantic label in an initial depth map corresponding to multiple frames of images are converted into three-dimensional points to obtain a point cloud of a first plane, including: converting pixels with a first semantic label in an initial depth map corresponding to multiple frames of images into three-dimensional points, filtering the three-dimensional points obtained after the conversion, and obtaining a point cloud of the first plane.

[0175] That is, the point cloud obtained after the conversion is processed and used as the point cloud corresponding to the target semantic label. For example, the point cloud obtained after the conversion is filtered. In other words, the point cloud of the first plane can be the point cloud obtained after filtering.

[0176] For example, statistical filtering and radius filtering are performed on the point cloud obtained after conversion, which can remove peripheral flying points and improve the accuracy of the point cloud.

[0177] A 3D point and a normal vector can define a plane.

[0178] The plane parameters (q, normal) of the point cloud may include the 3D point q through which the plane corresponding to the point cloud passes and the normal vector normal of the plane corresponding to the point cloud.

[0179] Optionally, the plane parameters of the point cloud of the first plane include: a normal vector of the first plane and a target three-dimensional point in the first plane.

[0180] Exemplarily, the three-dimensional point that the plane corresponding to the point cloud passes through can be determined by the average value of the positions of multiple points in the point cloud.

[0181] Optionally, the target three-dimensional point in the first plane is obtained by calculating an average value of the three-dimensional points in the point cloud of the first plane.

[0182] This can improve the accuracy of plane parameters and thus improve the accuracy of three-dimensional reconstruction.

[0183] Alternatively, the 3D point that the plane corresponding to the point cloud passes through may also be a point in the point cloud.

[0184] Exemplarily, the normal vector of the plane corresponding to the point cloud can be determined by a PCA algorithm.

[0185] Specifically, the PCA algorithm is used to estimate the orientation of the point cloud. The point cloud's orientation is composed of three perpendicular axes: the x-axis, the y-axis, and the z-axis. The x-axis represents the longest distribution direction of the point cloud, the y-axis represents the widest distribution direction of the point cloud, and the z-axis represents the normal vector to the plane corresponding to the point cloud.

[0186] If the point cloud is filtered, the plane parameters of the point cloud are determined based on the filtered point cloud.

[0187] S640: Convert the initial depth map corresponding to the multiple frames of images into a point cloud corresponding to the multiple frames of images.

[0188] Among them, the multiple frames of images include a first image, the three-dimensional point corresponding to the first pixel in the first image is the intersection of the first straight line and the first plane, the first pixel has a first semantic label, and the first straight line is determined based on the posture of the first image and the position of the first pixel in the first image.

[0189] In one implementation, step S640 includes step S641 and step S642.

[0190] S641 , updating the depth values ​​of the initial depth map corresponding to the multiple frames of images to obtain a target depth map corresponding to the multiple frames of images.

[0191] The depth value of the first pixel in the target depth map corresponding to the first image is determined according to the three-dimensional point corresponding to the first pixel.

[0192] Specifically, the depth value of the first pixel in the target depth map corresponding to the first image is obtained by projecting the three-dimensional point corresponding to the first pixel onto the target depth map corresponding to the first image according to the posture of the first image.

[0193] It should be understood that updating the depth value of the initial depth map corresponding to the multi-frame images refers to updating the depth value of at least one pixel in the initial depth map corresponding to the multi-frame images. That is, updating the depth value of the initial depth map corresponding to the multi-frame images may be updating the depth value of some or all pixels in the initial depth map corresponding to the multi-frame images. For example, updating the depth value of the initial depth map corresponding to the multi-frame images may include updating the depth value of the first pixel in the initial depth map corresponding to the first image. Or updating the depth value of the initial depth map corresponding to the multi-frame images may include updating the depth values ​​of all pixels in the initial depth map corresponding to all images in the multi-frame images. The embodiment of the present application does not limit the number of pixels updated in the initial depth map.

[0194] Exemplarily, step S641 may be performed by the depth map completion module 433 in the apparatus 400 .

[0195] The plane area in the initial depth map may have problems such as fewer valid pixels or inaccurate pixel values. For example, the weak texture area such as the plane area in the initial depth map may have holes. When the plane parameters of the plane area in the initial depth map are known in space, completing the depth value of the plane area in the initial depth map can essentially be converted into solving the problem of solving the intersection of a line and a plane, such as Figure 7 Completing the depth map means updating the depth value in the initial depth to obtain the target depth map.

[0196] The pixels in the depth map can be converted into three-dimensional points based on the intrinsic and extrinsic parameters of the camera, and for a pixel in the depth map, different depth values ​​correspond to different three-dimensional points. In other words, the depth value of a pixel is different, and the three-dimensional points obtained by its conversion are also different. These different three-dimensional points are located on the same straight line, which passes through the optical center of the camera. The three-dimensional point corresponding to the depth value of a pixel in a plane area in the depth map is the intersection of the straight line and the plane in space. The depth value of a pixel in the target depth map corresponding to a frame of image can be obtained by projecting the three-dimensional point corresponding to the pixel onto the frame of image.

[0197] The following describes a process of completing the initial depth maps corresponding to the multiple frames of images to obtain the target depth map, taking one frame of image as an example.

[0198] S1: Determine the straight line where the target 3D point corresponds to the pixel in the image.

[0199] The “pixels in the image” in step S1 refer to pixels in a planar area in the image.

[0200] Whether a pixel in an image belongs to a planar region can be determined based on the pixel's target semantic label. For example, if the value of a pixel's target semantic label is 0, the pixel belongs to a non-planar region; if the value of a pixel's target semantic label is non-zero, the pixel belongs to a planar region. In this case, step S1 may include determining the straight line on which the target 3D point corresponding to the pixel in the image whose target semantic label value is non-zero lies.

[0201] Exemplarily, the pixels in the image are traversed. If the value of the target semantic label of the pixel is 0, the pixel is skipped. If the value of the target semantic label of the pixel is not 0, step S1 is executed.

[0202] Step S1 can also be understood as determining the parameters of the straight line where the three-dimensional point corresponding to the pixel in the image lies.

[0203] Optionally, the pose of the first image includes the position of the optical center of a device that captures the first image.

[0204] Furthermore, the first straight line is determined based on the optical center position of the device that captures the first image and the position of the first three-dimensional point, and the first three-dimensional point is obtained by converting the first pixel into space based on the position of the first pixel and the initial depth value of the first pixel.

[0205] Two points in space can determine a straight line. The straight line where the three-dimensional point corresponding to the pixel in the image is located must pass through the optical center position c of the camera. Therefore, it is only necessary to determine another point p on the straight line (an example of the first three-dimensional point) to determine the straight line where the three-dimensional point corresponding to the pixel is located.

[0206] The point p may be a three-dimensional point obtained by converting the pixel in the initial depth map into space.

[0207] Specifically, point p can be determined in a variety of ways. For example, the pixel is converted into a three-dimensional point based on the depth value of the pixel in the initial depth map, and the three-dimensional point is used as the spatial point p. However, the depth value of the pixel in the initial depth map may be 0. In this case, the pixel can be converted into a three-dimensional point based on the preset depth value of the pixel, and the three-dimensional point is used as the spatial point p. The preset depth value of the pixel can be an arbitrarily set value. Alternatively, as mentioned above, the plane parameters of the plane include a three-dimensional point and a normal vector. In this case, the 3D point in the plane parameters of the plane to which the pixel belongs can be projected into the initial depth map to obtain a depth value, and the depth value is used as the depth value of the pixel, and then the pixel is converted into a three-dimensional point, and the three-dimensional point is used as the spatial point p. Exemplarily, the plane parameters of the plane to which the pixel belongs can be determined based on the point cloud corresponding to the target semantic label of the pixel. As mentioned above, the target semantic label and the point cloud are one-to-one corresponding, and the plane parameters of the point cloud corresponding to the target semantic label can be determined based on the target semantic label of the pixel.

[0208] For example, Figure 7 As shown, the three-dimensional point (i.e. Figure 7 The straight line A (an example of the first straight line) where the intersection point n) is located must pass through the position of the optical center c of the camera. It is only necessary to determine another point p on the straight line to determine the straight line where the three-dimensional point corresponding to pixel A is located. Pixel A has a label A (an example of a first semantic label), and label A is used to indicate that pixel A belongs to plane A (an example of a first plane). The three-dimensional point q in the plane parameters of plane A is projected onto the image to obtain a depth value, which is used as the initial depth value of pixel A. Pixel A is converted to an initial three-dimensional point in three-dimensional space according to the camera pose and internal parameters, and the initial three-dimensional point is used as the spatial point p. The spatial point p and the optical center position c can be used as the parameters of straight line A.

[0209] S2: Determine the 3D point corresponding to the pixel.

[0210] The intersection of the straight line where the three-dimensional point corresponding to the pixel is located and the plane to which the pixel belongs is the three-dimensional point corresponding to the pixel.

[0211] For example, Figure 7 As shown, the intersection point n of the straight line A and the plane A is the target three-dimensional point corresponding to the pixel A, that is, the actual three-dimensional point of the pixel A in space.

[0212] S3: Projecting the three-dimensional point corresponding to the pixel onto the initial depth map to obtain the depth value of the pixel, that is, the depth value of the pixel in the target depth map.

[0213] Step S3 can also be understood as projecting the target three-dimensional point onto the target depth map to obtain the depth value of the pixel, that is, completing the completion operation on the pixel.

[0214] Furthermore, the above steps S1 to S3 can be performed on all pixels in the plane region of the image to complete all pixels in the plane region of the image. For example, the above steps S1 to S3 can be performed on all pixels in the image whose plane semantic label value is non-zero.

[0215] S642: Convert the target depth map corresponding to the multiple frames of images into a point cloud corresponding to the multiple frames of images.

[0216] Specifically, the target depth map corresponding to the multi-frame images is converted into a point cloud according to the internal parameters of the multi-frame images, so as to obtain the point cloud corresponding to the multi-frame images.

[0217] In another implementation, step S640 may include: converting pixels of a non-planar area in an initial depth map corresponding to the multi-frame images into space according to the internal parameters of the multi-frame images to obtain a non-planar point cloud; determining three-dimensional points corresponding to pixels of a planar area in the initial depth map corresponding to the multi-frame images according to target semantic labels of the multi-frame images to obtain a planar point cloud.

[0218] The following describes step S640 by taking the first image in the multi-frame image as an example. The corresponding point clouds can be obtained in the same manner for other images in the multi-frame image.

[0219] The point cloud corresponding to the first image may include a point cloud corresponding to a planar area in the first image and a point cloud corresponding to a non-planar area in the first image.

[0220] For pixels in the non-planar area, the pixels in the non-planar area in the initial depth map corresponding to the first image are converted into space according to the intrinsic parameters of the first image, so as to obtain a non-planar point cloud.

[0221] For a pixel in the plane area, reference may be made to S1 and S2 in step S641 to obtain the three-dimensional point corresponding to the pixel.

[0222] S650: Fusing point clouds corresponding to multiple frames of images to obtain a fused target point cloud.

[0223] Specifically, the point cloud corresponding to the at least one frame of image is transformed into a world coordinate system according to the pose of the at least one frame of image, and fused to obtain a target point cloud, i.e., a dense point cloud. The dense point cloud can describe the entire scene.

[0224] In existing 3D reconstruction schemes, the estimated depth values ​​in weakly textured areas such as planar areas are inaccurate and will be filtered out during the filtering stage, resulting in holes in the point cloud obtained after fusion, making it impossible to reconstruct an accurate point cloud.

[0225] In the solution of the embodiment of the present application, the planar area in the initial depth map is converted into a corresponding planar point cloud according to the target semantic label of the image, the plane parameters of the plane can be determined according to the point cloud of the plane, and then the three-dimensional points corresponding to the pixels in the planar area in space are obtained according to the plane parameters. In this way, more accurate three-dimensional points corresponding to the planar area can be obtained, avoiding the occurrence of holes in the point cloud of the planar area with weak texture, which is conducive to improving the accuracy and density of three-dimensional reconstruction.

[0226] For example, the initial depth map in the embodiment of the present application may be a depth map obtained by an existing solution, in which the plane area in the depth map has fewer valid pixels and the depth values ​​of the pixels are not accurate enough. According to the solution of the embodiment of the present application, the plane area in the initial depth map is converted into a corresponding point cloud based on the semantic label, and then the actual depth values ​​of the pixels in the plane area are obtained based on the plane parameters of the point cloud to update the depth values ​​of the plane area in the initial depth map, thereby obtaining more accurate depth information in the plane area with weak texture, which is conducive to improving the accuracy and density of three-dimensional reconstruction.

[0227] The process of obtaining the initial depth map corresponding to at least one frame of image in step S610 is exemplarily described below.

[0228] Step S610 includes steps S611 to S612, which are described below.

[0229] S611, obtaining the pose and sparse point cloud of the image sequence.

[0230] The image sequence includes the at least one frame of image. The image sequence is a two-dimensional image sequence, for example, the image sequence can be an RGB image sequence.

[0231] Step S611 may include receiving the pose and sparse point cloud of the image sequence from another device; or step S610 may also include processing the image sequence to obtain the pose and sparse point cloud of the image sequence. This embodiment of the application does not limit the specific method of "obtaining" in step S611.

[0232] For example, step S611 may be performed by Figure 4 The image pose and sparse point cloud estimation module 410 in is executed.

[0233] Specifically, the image pose and sparse point cloud estimation module 410 may obtain the pose and sparse point cloud of the image sequence through SFM.

[0234] For example, the pose and sparse point cloud of an image sequence can be obtained by incremental SFM.

[0235] SFM refers to the process of reconstructing the 3D structure of a scene in an image from an image sequence.

[0236] For example, the image sequence may be a plurality of RGB images of a scene captured by an electronic device with a camera, or a plurality of RGB images received from another device.

[0237] The pose of an image can also be referred to as the pose of the camera that captured the image, or simply the camera pose. The camera pose can be understood as the transformation of the camera coordinate system to the world coordinate system, namely, rotation and offset. In other words, the camera pose can be represented by the camera pose matrix, which includes the transformation matrix for offset and rotation. The camera pose can also be understood as the camera's parameters, such as the camera's intrinsic parameters or the camera's extrinsic parameters.

[0238] The camera's pose relative to the object—that is, the pose of the camera coordinate system relative to the world coordinate system—can be determined based on the object's multiple 3D points in the world coordinate system and the 2D points projected onto the image. Furthermore, for multiple frames in an image sequence, the pose transformation of the current frame relative to the previous frame can be determined based on the multiple 3D points in the previous frame and the 2D points projected onto the current frame.

[0239] The sparse point cloud of the image sequence is used to indicate the geometric information of the scene.

[0240] The following describes step S611 by taking the incremental SFM as an example.

[0241] Incremental SFM usually consists of two stages: the correspondence search stage and the incremental reconstruction stage. Figure 8A schematic diagram of the incremental SFM process is shown. Figure 8 As shown in the figure, in the association search phase, the image is processed sequentially through feature extraction, feature matching, and geometric verification. In the incremental reconstruction phase, initialization is performed, followed by image registration, triangulation, and bundle adjustment.

[0242] The association search phase includes steps S11 to S13 , and the incremental reconstruction phase includes steps S21 to S24 .

[0243] S11, feature extraction.

[0244] Specifically, local feature points are detected in images in an image sequence to obtain descriptors of the local feature points.

[0245] Local feature points have illumination invariance and geometric invariance. For example, geometric invariance includes scale invariance and rotation invariance. For the sake of simplicity, local feature points will be collectively described as feature points in the following text.

[0246] The descriptor of a feature point can be represented by a vector. The descriptor of a feature point can be called a feature vector.

[0247] For example, the feature extracted in step S1 is a scale-invariant feature transform (SIFT) feature vector, and the SIFT feature vector is a descriptor of a SIFT feature point.

[0248] S12, feature matching.

[0249] Whether two feature points are a matching point pair is determined based on the Euclidean distance between the two feature points on two images in the image sequence.

[0250] For example, if the Euclidean distance between two feature points is greater than or equal to a set threshold, the two feature points are not a matching point pair. If the Euclidean distance between two feature points is less than the set threshold, the two feature points are a matching pair.

[0251] The Euclidean distance between two feature points can be determined by the Euclidean distance between the descriptors of the two feature points.

[0252] If there is a matching pair between two images, then there is an association relationship between the two images.

[0253] S13, geometric verification.

[0254] The motion of the camera that captured the two images is determined based on the correspondence between the matching pairs in the two images, and then the incorrect matching pairs are deleted based on the camera motion and the epipolar constraint relationship.

[0255] Specifically, the essential matrix, fundamental matrix, and homography matrix are estimated using the random sample consensus (RANSAC) algorithm based on the epipolar geometric relationship of the matching pairs. Based on these matrices, the epipolar constraint relationship is used to delete incorrect matching pairs.

[0256] The essential matrix, fundamental matrix, and homography matrix are used to represent the motion of the camera. The essential matrix can also be called the E matrix, the fundamental matrix can also be called the F matrix, and the homography matrix can also be called the H matrix.

[0257] Through geometric verification, incorrect matching pairs are removed from the matching point pairs to obtain accurate matching pairs.

[0258] The output results of the association search phase include matching pairs of any two images in the input image sequence. For example, the output results include matching pairs of every two images in the input image sequence. The matching pairs of any two images refer to matching pairs obtained after the geometric verification process.

[0259] S21, initialization.

[0260] Two frames of images are selected from the output results of the association search stage as the initial pair, the initial pose is calculated based on the initial pair, and the matching pairs in the two frames of images are triangulated to obtain the initial point cloud.

[0261] The initial pair must meet at least the following conditions:

[0262] The number of inlier matching is greater than or equal to a first threshold t1, the front-to-back translation distance z between the two frames of image is less than a second threshold t2, and the triangulation angle is greater than a third threshold t3.

[0263] Exemplarily, a search may be performed in the output results of the association search phase until two frames of images satisfying the above conditions are obtained, ie, the search is stopped and the two frames of images satisfying the above conditions are taken as an initial pair.

[0264] Alternatively, a search may be performed in the output results of the association search phase, and an image pair may be arbitrarily selected as an initial pair from a plurality of image pairs that meet the above conditions.

[0265] S22, image registration.

[0266] Register the initial pair; sort the images that are associated with the registered images; obtain the best next-frame image associated with the registered image, register it using the Perspective-3-Points (P3P) random sampling consensus algorithm, and calculate the pose of this frame. Repeat this process until all images in the image sequence are registered, and the pose of each frame in the image sequence is obtained.

[0267] The best next frame image associated with the registered image at least satisfies the condition in step S4.

[0268] S23, triangulation.

[0269] The corresponding 3D map points are obtained by triangulating the poses of the current frame and the registered image, as well as the matching pairs between the two.

[0270] S24, bundle adjustment.

[0271] Bundle adjustment is a joint nonlinear optimization method that optimizes poses and 3D map points by minimizing the reprojection error.

[0272] Specifically, the registered image frames, poses, and 3D map points are bundled and adjusted to obtain optimized poses and 3D map points.

[0273] Furthermore, the 3D map points are retriangulated and filtered to obtain a sparse point cloud, which can further improve the accuracy.

[0274] The pose of the image sequence and the sparse point cloud corresponding to the image sequence can be obtained through incremental SFM processing.

[0275] It should be noted that the above is only an example of the incremental SFM method. The pose and sparse point cloud of the image sequence can also be obtained by other methods, and the embodiments of the present application are not limited to this.

[0276] S612: Obtain an initial depth map corresponding to the image according to the pose of the image sequence and the sparse point cloud.

[0277] Exemplarily, step S612 may be performed by the depth map estimation filtering module 431 in the apparatus 400 .

[0278] Specifically, the MVS algorithm is used to perform stereo matching based on the pose of the image sequence and the sparse point cloud to obtain a depth map corresponding to the image in the image sequence, and the depth map is used as the initial depth map corresponding to the image.

[0279] Furthermore, the depth map corresponding to the image can be filtered to obtain a depth map of the filtered image, which is used as the initial depth map corresponding to the image. The depth map of the filtered image has higher quality, which is conducive to achieving better 3D reconstruction effect.

[0280] Exemplarily, step S612 includes steps S6121 to S6123.

[0281] S6121, view selection.

[0282] According to the pose and sparse point cloud of the image sequence, the neighborhood frame of each frame in the image sequence is obtained.

[0283] The length of the baseline between the image frame and the neighboring frame is greater than the set threshold, which can improve the accuracy of the target point cloud. The overlapping area between the image frame and the neighboring frame is greater than the set threshold.

[0284] The baseline of two frames refers to the line connecting the positions of the cameras that captured the two frames.

[0285] The neighborhood frames of a frame of image can be one frame or multiple frames.

[0286] S6122, Depth map estimation.

[0287] Initialize the depth map, estimate the pixel values ​​of the depth map, and then optimize the estimated depth map.

[0288] Specifically, initializing the depth map includes assigning an initial value to each pixel in the depth map based on the sparse point cloud to obtain an initial pixel value of the depth map. In the depth map, the pixel value of a pixel is the depth value of the pixel.

[0289] The pixel values ​​of the depth map are estimated, including: traversing all pixels of the depth map, using the normalized cross correlation (NCC) similarity score of the image block where the pixel point is located as the criterion, iteratively optimizing the pixel value of each pixel through neighborhood propagation and random perturbation, deleting unreliable pixel values, and obtaining the depth map corresponding to the image frame.

[0290] The size of the image block where the pixel point is located can be set as needed. For example, the image block where the pixel point is located can be an image block of size 7*7.

[0291] The reliability of a pixel value can be determined by its confidence level. A pixel value with a confidence level lower than a set threshold is considered an unreliable pixel value. The confidence level of a pixel value can be determined by NCC.

[0292] The estimated depth map is optimized, including: performing image-level processing on the estimated depth map, such as removing depth fragments and filling small holes / gaps, to obtain an optimized depth map.

[0293] S6123, depth filtering.

[0294] The optimized depth map is filtered to obtain a depth map of the filtered image.

[0295] Exemplarily, a neighborhood frame projection method may be used to perform depth filtering.

[0296] Specifically, the depth map of the neighboring frame of the current frame obtained in step S6122 is projected to the current frame according to the posture to obtain a projected depth map, and the confidence of the pixel value in the depth map obtained in step S6122 of the neighboring frame is used as the confidence of the pixel value in the projected depth map.

[0297] The depth value of a pixel in the projected depth map is determined to be similar to the depth value of the corresponding pixel in the depth map of the current frame. If the similarity is above a set threshold, the weighted depth of the pixel in the neighboring frame is accumulated, the positive frame counter is updated, and the positive confidence is accumulated. Otherwise, the negative frame counter is updated and the negative confidence is accumulated. For example, if the similarity is above a set threshold, the positive frame counter is incremented by one; if the similarity is below or equal to the set threshold, the negative frame counter is incremented by one. The weighted depth is calculated by multiplying the depth value of the pixel in the neighboring frame by the confidence level of that depth value.

[0298] If the number of positive frames is greater than the set threshold, that is, the value of the positive frame counter is greater than the set threshold, and the sum of the positive confidences of the pixels in the current frame is greater than the sum of the negative confidences, and the weighted depth is between the minimum depth and the maximum depth, then the weighted depth is used to update the depth value and confidence of the pixel. Otherwise, the depth value and confidence of the pixel are discarded, that is, the pixel value is set to 0, the confidence is set to 0, and the filtered depth map is obtained.

[0299] It should be understood that the above method of obtaining the initial depth map corresponding to the at least one frame of image is only an example, and the initial depth map corresponding to the at least one frame of image can also be obtained by other methods, which is not limited in this embodiment of the present application.

[0300] The solution of the embodiment of the present application introduces the shape semantic information of the image, and reconstructs dense 3D points based on the sparse 3D points of the planar area with weak texture, which can solve the problem of poor reconstruction effect of the planar area when performing three-dimensional reconstruction based on two-dimensional images.

[0301] Figure 9 (a) shows the plane semantic segmentation result of a frame image. Figure 9(b) shows a schematic diagram of converting the depth map corresponding to the image into a point cloud using an existing solution. Figure 9 (c) shows a schematic diagram of converting the target depth map corresponding to the image into a point cloud using the solution of the present application.

[0302] like Figure 9 As shown in (a), after the image is processed with semantic segmentation, multiple plane areas such as the ceiling, the ground and the wall are obtained. Figure 9 As shown in (b), the existing solution will result in holes in the depth map corresponding to the image, and the point cloud converted from the depth map is relatively sparse. The solution of this application completes the planar area in the initial depth map corresponding to the image, and then converts the target depth map obtained after completion into a point cloud, as shown in Figure 9 As shown in (c), in the point cloud obtained by the solution of this application, the ceiling, the ground and the left wall complete more points, which is obviously better than Figure 9 The point cloud shown in (b) is denser.

[0303] Figure 10 (a) shows the dense point cloud corresponding to the image sequence obtained using the existing scheme. Figure 10 (b) shows the dense point cloud corresponding to the image sequence obtained using the solution of the present application. Figure 10 There are a lot of holes in the point cloud in (a). Figure 10 The point cloud in (b) has fewer holes, much less than Figure 10 The point cloud in (a) is denser, especially the ceiling, ground, and wall areas are almost completely covered by the point cloud.

[0304] In addition, the dense point cloud obtained by 3D reconstruction can be applied to the RGB-VPS system, in which the dense point cloud can also be called a map. The solution of the embodiment of the present application can improve the accuracy of the reconstructed dense point cloud, which is conducive to improving the accuracy and robustness of positioning. Moreover, the solution of the embodiment of the present application can increase the density of the reconstructed point cloud, which is conducive to improving the experience of virtual and real occlusion.

[0305] Figure 11 (a) shows the virtual and real occlusion effect of the dense point cloud obtained by the existing solution. Figure 11 (b) shows the virtual and real occlusion effect of the dense point cloud obtained by using the solution of this application. Figure 11 As shown, a virtual object is placed in the space of the RGB-VPS system, namely Figure 11The block in the figure is located behind the wall. This virtual object should not be observed in the map of the RGB-VPS system. When the dense point cloud obtained by the prior art is used as a map, the point cloud is relatively sparse and the wall in the actual scene is not reconstructed, so the virtual object cannot be blocked and the user can observe the virtual object. When the dense point cloud obtained by the solution of the embodiment of the present application is used as a map, the point cloud is relatively dense and the wall in the actual scene is reconstructed, so the virtual object can be blocked.

[0306] Figure 12 (a) shows the effect of not using a globally unified planar semantic label on the point cloud. Figure 12 (b) shows the effect of adopting globally unified planar semantic labels on the point cloud. Figure 12 The point cloud of the right wall in (b) is significantly denser. This means that using globally consistent plane semantic labels can yield more accurate semantic segmentation results, improving the inaccurate or impossible estimation of plane parameters for some planes. This allows for more accurate plane parameters, enhances depth map completion, and further improves 3D reconstruction accuracy.

[0307] The following combination Figures 13 and 14 The apparatus of the embodiment of the present application is described. It should be understood that the apparatus described below can execute the method of the embodiment of the present application, and in order to avoid unnecessary repetition, repeated descriptions are appropriately omitted when introducing the apparatus of the embodiment of the present application.

[0308] Figure 13 It is a schematic block diagram of a three-dimensional reconstruction device provided in an embodiment of the present application. Figure 13 The device 4000 shown includes an acquisition unit 4010 and a processing unit 4020 .

[0309] The acquisition unit 4010 and the processing unit 4020 can be used to execute the three-dimensional reconstruction method of the embodiment of the present application, for example, can be used to execute method 600.

[0310] The acquisition unit 4010 is used to: obtain an initial depth map corresponding to multiple frame images; obtain a target semantic label of the initial depth map corresponding to the multiple frame images, the target semantic label includes a first semantic label, and the first semantic label is used to indicate that the pixels with the first semantic label in the initial depth map corresponding to the multiple frame images belong to the first plane.

[0311] The processing unit 4020 is used to: convert pixels with a first semantic label in an initial depth map corresponding to the multi-frame images into three-dimensional points to obtain a point cloud of the first plane; convert the initial depth map corresponding to the multi-frame images into a point cloud corresponding to the multi-frame images, wherein the multi-frame images include a first image, the three-dimensional point corresponding to the first pixel in the first image is the intersection of the first straight line and the first plane, the first pixel has a first semantic label, and the first straight line is determined based on the posture of the first image and the position of the first pixel in the first image; fuse the point clouds corresponding to the multi-frame images to obtain a fused target point cloud.

[0312] Optionally, as an embodiment, the processing unit 4020 is specifically used to: update the depth value of the initial depth map corresponding to the multiple-frame images to obtain the target depth map corresponding to the multiple-frame images, wherein the depth value of the first pixel in the target depth map corresponding to the first image is obtained by projecting the three-dimensional point corresponding to the first pixel onto the target depth map corresponding to the first image according to the posture of the first image; and convert the target depth map corresponding to the multiple-frame images into a point cloud corresponding to the multiple-frame images.

[0313] Optionally, as an embodiment, the target semantic label is obtained by performing image semantic segmentation processing on multiple frames of images.

[0314] Optionally, as an embodiment, multiple frames of images belong to an image sequence, and the target semantic label is obtained by merging the initial semantic labels in the image sequence, and the initial semantic labels of the images in the image sequence are obtained by performing image semantic segmentation processing on the images, wherein the first semantic label is determined based on the semantic label obtained by merging the first initial semantic label in the first image and the second semantic label in the second image adjacent to the first image in the image sequence, and the semantic label obtained after the merger is obtained by merging the first initial semantic label and the second initial semantic label when the first area corresponding to the first initial semantic label in the first image and the second area corresponding to the second initial semantic label in the second image at least partially overlap.

[0315] Optionally, as an embodiment, the semantic label obtained after merging is obtained by merging the first initial semantic label and the second initial semantic label when the first area corresponding to the first initial semantic label in the first image and the second area corresponding to the second initial semantic label in the second image at least partially overlap, including: the semantic label obtained after merging is obtained by merging the first initial semantic label and the second initial semantic label when the ratio of the number of pixels in the overlapping area in the first area to the number of pixels in the first area is greater than or equal to a first threshold, and / or the ratio of the number of pixels in the overlapping area in the second area to the number of pixels in the second area is greater than or equal to the first threshold.

[0316] Optionally, as an embodiment, the plane parameters of the point cloud of the first plane include: a normal vector of the first plane and a target three-dimensional point in the first plane.

[0317] Optionally, as an embodiment, the target three-dimensional point in the first plane is obtained by calculating an average value of the three-dimensional points in the point cloud of the first plane.

[0318] Optionally, as an embodiment, the posture of the first image includes the optical center position of a device that captures the first image.

[0319] Optionally, as an embodiment, the first straight line is determined based on the optical center position of the device that captures the first image and the position of the first three-dimensional point, and the first three-dimensional point is obtained by converting the first pixel into space based on the position of the first pixel in the first image and the initial depth value of the first pixel.

[0320] It should be noted that the above-mentioned device 4000 is embodied in the form of a functional unit. The term "unit" here can be implemented in the form of software and / or hardware, and is not specifically limited to this.

[0321] For example, a "unit" may be a software program, a hardware circuit, or a combination of the two that implements the aforementioned functionality. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (e.g., a shared processor, a dedicated processor, or a group processor) and memory for executing one or more software or firmware programs, combined logic circuits, and / or other suitable components that support the described functionality.

[0322] Therefore, the units of each example described in the embodiments of this application can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0323] Figure 14 Schematic diagram of the hardware structure of the three-dimensional reconstruction device according to an embodiment of the present application. Figure 14 The device 6000 shown includes a memory 6001, a processor 6002, a communication interface 6003, and a bus 6004. The memory 6001, the processor 6002, and the communication interface 6003 are communicatively connected to each other via the bus 6004.

[0324] The memory 6001 can be a ROM, a static storage device, or a RAM. The memory 6001 can store programs. When the programs stored in the memory 6001 are executed by the processor 6002, the processor 6002 and the communication interface 6003 are used to execute the various steps of the three-dimensional reconstruction method of the embodiment of the present application. Specifically, the processor 6002 can execute the above Figure 6 Steps S630 to S650 in the method shown.

[0325] The processor 6002 can be a general-purpose CPU, microprocessor, ASIC, GPU or one or more integrated circuits, which is used to execute relevant programs to implement the functions required to be performed by the units in the three-dimensional reconstruction device of the embodiment of the present application, or to execute the three-dimensional reconstruction method of the method embodiment of the present application.

[0326] The processor 6002 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the 3D reconstruction method of the embodiment of the present application may be completed by hardware integrated logic circuits in the processor 6002 or software instructions.

[0327] The above-mentioned processor 6002 can also be a general-purpose processor, DSP, ASIC, FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The various methods, steps and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of this application can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 6001, and the processor 6002 reads the information in the memory 6001 and completes it in combination with its hardware. Figure 13 The functions required to be performed by the units included in the three-dimensional reconstruction device shown, or the functions required to be performed Figure 6 The 3D reconstruction method shown.

[0328] The communication interface 6003 uses a transceiver such as, but not limited to, a transceiver to implement communication between the device 6000 and other devices or a communication network. For example, the communication interface 6003 can be used to obtain initial depth maps or target semantic labels corresponding to multiple frames of images.

[0329] The bus 6004 may include a path for transmitting information between various components of the device 6000 (eg, the memory 6001 , the processor 6002 , and the communication interface 6003 ).

[0330] It should be noted that although the above-mentioned device 6000 only shows a memory, a processor, and a communication interface, in the specific implementation process, those skilled in the art should understand that the device 6000 may also include other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the device 6000 may also include hardware devices that implement other additional functions. In addition, those skilled in the art should understand that the device 6000 may also only include the devices necessary to implement the embodiments of the present application, and does not necessarily include Figure 14 All devices shown in .

[0331] It should be understood that the processor in the embodiments of the present application may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0332] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0333] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0334] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0335] In this application, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0336] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0337] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0338] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0339] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0340] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0341] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0342] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0343] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A three-dimensional reconstruction method, characterized in that: include: Obtain the initial depth map corresponding to multiple frames of images; Obtaining a target semantic label of the initial depth map corresponding to the multiple-frame image, the target semantic label including a first semantic label, the first semantic label being used to indicate that a pixel having the first semantic label in the initial depth map corresponding to the multiple-frame image belongs to a first plane; Converting pixels having the first semantic label in the initial depth map corresponding to the multiple frames of image into three-dimensional points to obtain a point cloud of the first plane; Converting an initial depth map corresponding to the multiple image frames into a point cloud corresponding to the multiple image frames, wherein the multiple image frames include a first image, a three-dimensional point corresponding to a first pixel in the first image is an intersection of a first straight line and the first plane, the first pixel has the first semantic label, and the first straight line is determined based on a pose of the first image and a position of the first pixel in the first image; The point clouds corresponding to the multiple frames of images are fused to obtain a fused target point cloud.

2. The method according to claim 1, characterized in that The converting the initial depth map corresponding to the multiple frames of image into a point cloud corresponding to the multiple frames of image comprises: Updating the depth values ​​of the initial depth maps corresponding to the multiple frames of images to obtain a target depth map corresponding to the multiple frames of images, wherein the depth value of a first pixel in the target depth map corresponding to the first image is obtained by projecting a three-dimensional point corresponding to the first pixel onto the target depth map corresponding to the first image according to the pose of the first image; The target depth map corresponding to the multiple frames of images is converted into a point cloud corresponding to the multiple frames of images.

3. The method according to claim 1 or 2, characterized in that The target semantic label is obtained by performing image semantic segmentation processing on the multiple frames of images.

4. The method according to claim 1 or 2, characterized in that The multiple frames of images belong to an image sequence, and The target semantic label is obtained by merging initial semantic labels in the image sequence, and the initial semantic labels of the images in the image sequence are obtained by performing image semantic segmentation processing on the images. The first semantic label is determined based on a semantic label obtained by merging a first initial semantic label in the first image and a second initial semantic label in a second image adjacent to the first image in the image sequence. The merged semantic label is obtained by merging the first initial semantic label and the second initial semantic label when a first area corresponding to the first initial semantic label in the first image and a second area corresponding to the second initial semantic label in the second image at least partially overlap.

5. The method according to claim 4, characterized in that The merged semantic label is obtained by merging the first initial semantic label and the second initial semantic label when a first area corresponding to the first initial semantic label in the first image and a second area corresponding to the second initial semantic label in the second image at least partially overlap, and includes: The semantic label obtained after merging is obtained by merging the first initial semantic label and the second initial semantic label when a ratio of the number of pixels in the overlapping area of ​​the first area to the number of pixels in the first area is greater than or equal to a first threshold, and / or a ratio of the number of pixels in the overlapping area of ​​the second area to the number of pixels in the second area is greater than or equal to the first threshold.

6. The method according to any one of claims 1 to 5, characterized in that The plane parameters of the point cloud of the first plane include: a normal vector of the first plane and a target three-dimensional point in the first plane.

7. The method according to claim 6, characterized in that The target three-dimensional point in the first plane is obtained by calculating an average value of the three-dimensional points in the point cloud of the first plane.

8. The method according to any one of claims 1 to 7, characterized in that The pose of the first image includes the position of the optical center of a device that captures the first image.

9. The method according to claim 8, characterized in that The first straight line is determined based on the optical center position of the device that captures the first image and the position of the first three-dimensional point, and the first three-dimensional point is obtained by converting the first pixel into space based on the position of the first pixel in the first image and the initial depth value of the first pixel.

10. A three-dimensional reconstruction device, characterized in that: include: Get unit for: Obtain the initial depth map corresponding to multiple frames of images; Obtaining a target semantic label of the initial depth map corresponding to the multiple-frame image, the target semantic label including a first semantic label, the first semantic label being used to indicate that a pixel having the first semantic label in the initial depth map corresponding to the multiple-frame image belongs to a first plane; A processing unit for: Converting pixels having the first semantic label in the initial depth map corresponding to the multiple frames of image into three-dimensional points to obtain a point cloud of the first plane; Converting an initial depth map corresponding to the multiple image frames into a point cloud corresponding to the multiple image frames, wherein the multiple image frames include a first image, a three-dimensional point corresponding to a first pixel in the first image is an intersection of a first straight line and the first plane, the first pixel has the first semantic label, and the first straight line is determined based on a pose of the first image and a position of the first pixel in the first image; The point clouds corresponding to the multiple frames of images are fused to obtain a fused target point cloud.

11. The device according to claim 10, characterized in that The processing unit is specifically configured to: Updating the depth values ​​of the initial depth maps corresponding to the multiple frames of images to obtain a target depth map corresponding to the multiple frames of images, wherein the depth value of a first pixel in the target depth map corresponding to the first image is obtained by projecting a three-dimensional point corresponding to the first pixel onto the target depth map corresponding to the first image according to the pose of the first image; The target depth map corresponding to the multiple frames of images is converted into a point cloud corresponding to the multiple frames of images.

12. The device according to claim 10 or 11, characterized in that The target semantic label is obtained by performing image semantic segmentation processing on the multiple frames of images.

13. The device according to claim 10 or 11, characterized in that The multiple frames of images belong to an image sequence, and The target semantic label is obtained by merging initial semantic labels in the image sequence, and the initial semantic labels of the images in the image sequence are obtained by performing image semantic segmentation processing on the images. The first semantic label is determined based on a semantic label obtained by merging a first initial semantic label in the first image and a second initial semantic label in a second image adjacent to the first image in the image sequence. The merged semantic label is obtained by merging the first initial semantic label and the second initial semantic label when a first area corresponding to the first initial semantic label in the first image and a second area corresponding to the second initial semantic label in the second image at least partially overlap.

14. The device according to claim 13, characterized in that The merged semantic label is obtained by merging the first initial semantic label and the second initial semantic label when a first area corresponding to the first initial semantic label in the first image and a second area corresponding to the second initial semantic label in the second image at least partially overlap, and includes: The semantic label obtained after merging is obtained by merging the first initial semantic label and the second initial semantic label when a ratio of the number of pixels in the overlapping area of ​​the first area to the number of pixels in the first area is greater than or equal to a first threshold, and / or a ratio of the number of pixels in the overlapping area of ​​the second area to the number of pixels in the second area is greater than or equal to the first threshold.

15. The device according to any one of claims 10 to 14, characterized in that The plane parameters of the point cloud of the first plane include: a normal vector of the first plane and a target three-dimensional point in the first plane.

16. The device according to claim 15, characterized in that The target three-dimensional point in the first plane is obtained by calculating an average value of the three-dimensional points in the point cloud of the first plane.

17. The device according to any one of claims 10 to 16, characterized in that The pose of the first image includes the position of the optical center of a device that captures the first image.

18. The device according to claim 17, characterized in that The first straight line is determined based on the optical center position of the device that captures the first image and the position of the first three-dimensional point, and the first three-dimensional point is obtained by converting the first pixel into space based on the position of the first pixel in the first image and the initial depth value of the first pixel.

19. A three-dimensional reconstruction device, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the program instructions to execute the method according to any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that The computer-readable storage medium stores program codes for execution by a device, wherein the program codes include codes for executing the method according to any one of claims 1 to 9.

Images