Three-dimensional scene reconstruction method, device, electronic device and readable storage medium

Abstract

The present invention discloses a 3D scene reconstruction method, device, electronic device and readable storage medium, which are applied to the technical field of 3D reconstruction. In order to solve the problem of low accuracy of existing 3D reconstruction, it is proposed to update a current 3D geometric model based on a current frame image to obtain an updated 3D geometric model; use a brightness map obtained based on the current frame image to update the existing texture image stored inside the surface voxels in the updated 3D geometric model to obtain an updated texture image; judge whether the current frame image is the last frame image, and if so, use the updated 3D geometric model as the reconstructed 3D geometric model, and use the updated texture image corresponding to each surface voxel as the reconstructed texture image to constitute a reconstructed 3D scene; otherwise, use the next frame image as the current frame image and return to execute the step of updating the current 3D geometric model based on the current frame image; the accuracy and quality of reconstruction can be improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of three-dimensional reconstruction technology, and in particular to a three-dimensional scene reconstruction method, device, electronic device, and computer-readable storage medium. Background Art

[0002] Three-dimensional reconstruction is the technology that uses various sensor devices to scan and process real physical scenes and generate corresponding digital models. It is one of the key issues in three-dimensional vision research and has achieved many research results in the past few decades. In recent years, with the advancement of sensor and computing technology, three-dimensional reconstruction has shown some new changes and characteristics. Real-time, precise, and multi-factor 3D reconstruction methods that combine textures / materials have become the goal pursued by industry insiders. Three-dimensional reconstruction needs to consider the coupling of multiple factors such as geometry, texture, material, and lighting. Simple reconstruction of geometry currently has relatively complete theoretical support and engineering verification. Texture reconstruction can be affected by factors such as geometry and lighting. However, existing texture reconstruction schemes only perform texture reconstruction based on texture images, and do not consider the impact of geometry and lighting on the color of the acquired image during the reconstruction process, which affects the accuracy of three-dimensional scene reconstruction.

[0003] In view of this, how to improve the accuracy of three-dimensional scene reconstruction has become a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] The purpose of the embodiments of the present invention is to provide a three-dimensional scene reconstruction method, device, electronic device and computer-readable storage medium, which, during use, combines geometric and brightness factors to reconstruct texture images, thereby improving the accuracy of three-dimensional scene reconstruction and the quality of reconstructed textures.

[0005] To solve the above technical problems, an embodiment of the present invention provides a three-dimensional scene reconstruction method, comprising:

[0006] updating the current three-dimensional geometric model based on the current frame image to obtain an updated three-dimensional geometric model;

[0007] Using a brightness map obtained based on the current frame image, an existing texture image stored in a surface voxel in the updated three-dimensional geometric model is updated to obtain an updated texture image;

[0008] Determine whether the current frame image is the last frame image. If so, use the updated three-dimensional geometric model as the reconstructed three-dimensional geometric model, use the updated texture images corresponding to each of the surface voxels as the reconstructed texture image, and obtain a reconstructed three-dimensional scene based on the reconstructed three-dimensional geometric model and the reconstructed texture image; otherwise, use the next frame image as the current frame image and return to execute the step of updating the current three-dimensional geometric model based on the current frame image.

[0009] In one embodiment, updating the current three-dimensional geometric model based on the current frame image to obtain an updated three-dimensional geometric model includes:

[0010] The signed distance stored in each voxel in the current three-dimensional geometric model is updated according to the current frame image to obtain an updated three-dimensional geometric model.

[0011] In one embodiment, updating the signed distance stored in each voxel in the current three-dimensional geometric model according to the current frame image to obtain an updated three-dimensional geometric model includes:

[0012] Based on the current frame image and the previous frame image, obtaining a current camera pose corresponding to the current frame image;

[0013] Calculating the three-dimensional coordinates corresponding to each pixel in the current frame image based on the current camera pose and the image pixel position of each pixel in the current frame image;

[0014] Determining, based on each voxel of the current three-dimensional geometric model, a voxel corresponding to each pixel in the current frame image;

[0015] For each voxel, calculating a current signed distance according to an image pixel position of a pixel corresponding to the voxel;

[0016] The current signed distance and the stored signed distance stored in the voxel are used to obtain an updated signed distance, and the updated signed distance is used to replace the stored signed distance until the stored signed distances of all voxels are updated to obtain an updated three-dimensional geometric model.

[0017] In one embodiment, obtaining a current camera pose corresponding to the current frame image based on the current frame image and the previous frame image includes:

[0018] Performing feature description extraction on the current frame image to obtain multiple current image features;

[0019] Performing feature matching on a plurality of current image features of the current frame image and a plurality of image features of the previous frame image to obtain a correspondence between the image features;

[0020] Based on the correspondence between the image features, an iterative closest point algorithm is used to determine the current camera pose corresponding to the current frame image.

[0021] In one embodiment, the updating of the stored texture image stored in the surface voxels in the updated three-dimensional geometric model using the brightness map obtained based on the current frame image to obtain the updated texture image includes:

[0022] determining each surface voxel from each voxel of the updated three-dimensional geometric model;

[0023] Determining, according to the current frame image, a current texture image of the current frame image for each of the surface voxels;

[0024] Calculating a brightness level value of a sampling area corresponding to the surface voxel according to a brightness map obtained based on the current frame image;

[0025] According to the magnitude relationship between the brightness level value and the preset brightness level value, using the current frame image for the current texture image of the surface voxel, updating the existing texture image stored in the surface voxel to obtain an updated texture image corresponding to the surface voxel;

[0026] An overall updated texture image is obtained according to the updated texture images corresponding to the respective surface voxels.

[0027] In one embodiment, determining, based on the current frame image, a current texture image of the current frame image for each surface voxel includes:

[0028] For each of the surface voxels, determining the pixels corresponding to the surface voxel according to the three-dimensional coordinates corresponding to the pixels of the current frame image;

[0029] Projecting each pixel corresponding to the surface voxel along the surface normal direction of the surface voxel to obtain a projection area relative to the surface voxel;

[0030] The projection area relative to the surface voxel is sampled using preset pixels to obtain a pixel value of each sampling pixel in the sampling area, so as to obtain a current texture image of the current frame image for the surface voxel.

[0031] In one embodiment, projecting each pixel corresponding to the surface voxel along the surface normal direction of the surface voxel to obtain a projection area relative to the surface voxel includes:

[0032] Calculating a surface normal of the surface voxel based on the stored signed distance stored in the surface voxel;

[0033] Calculating the angle between the viewpoint direction of the current frame image and the surface normal of the surface voxel;

[0034] Based on the included angle, each pixel corresponding to the surface voxel is projected along the surface normal direction of the surface voxel to obtain a corresponding projection area.

[0035] In one embodiment, the viewpoint direction of the current frame image is determined according to the current camera pose corresponding to the current frame image.

[0036] In one embodiment, sampling the projection area relative to the surface voxel using preset pixels to obtain the pixel value of each sampled pixel in the sampling area includes:

[0037] Determining the pixel positions of each vertex in the sampling area according to the size of the preset pixel and the projection area;

[0038] Calculating the pixel coordinates of the corresponding sampling pixels in the sampling area based on a preset pixel size, pixel positions of each vertex in the sampling area, and image pixel positions of pixels corresponding to the surface voxels in the original sampling image in combination with a pixel coordinate calculation formula;

[0039] The pixel value of the sampling pixel at the pixel coordinates in the sampling area is calculated according to the pixel value of the target pixel corresponding to the pixel coordinates in the original sampling image.

[0040] In one embodiment, calculating the pixel value of the sampling pixel at the pixel coordinates in the sampling area according to the pixel value of the target pixel corresponding to the pixel coordinates in the original sampling image includes:

[0041] When there are multiple target pixels corresponding to the pixel coordinates, a weighted average is performed on the pixel values ​​of the multiple target pixels in the original sampling image to obtain the pixel value of the sampling pixel at the pixel coordinates in the sampling area.

[0042] In one embodiment, the pixel coordinate calculation relationship is:

[0043] i=(UU min )*w / (U max -U min ),

[0044] j=(VV min )*h / (V max -V min), where i represents the horizontal coordinate of the sampled pixel, j represents the vertical coordinate of the sampled pixel, and U max Indicates the maximum value of the horizontal coordinate of the sampling area, U min Indicates the minimum value of the horizontal coordinate of the sampling area, V max Indicates the maximum value of the vertical coordinate of the sampling area, V min Indicates the minimum value of the horizontal coordinate of the sampling area, U indicates the horizontal coordinate of the image pixel corresponding to the pixel, and V indicates the vertical coordinate of the image pixel corresponding to the pixel.

[0045] In one embodiment, calculating the brightness level value of the sampling area corresponding to the surface voxel according to the brightness map obtained based on the current frame image includes:

[0046] Obtaining brightness level values ​​of respective pixels corresponding to the surface voxels according to a brightness map obtained based on the current frame image;

[0047] The brightness level values ​​of the pixels corresponding to the surface voxels are averaged to obtain the brightness level value of the surface voxels.

[0048] In one embodiment, before calculating the brightness level value of the sampling area corresponding to the surface voxel according to the brightness map obtained based on the current frame image, the method further includes:

[0049] A brightness image corresponding to the current frame image is obtained according to the color image of the current frame image.

[0050] In one embodiment, obtaining a brightness map corresponding to the current frame image based on the color image of the current frame image includes:

[0051] Converting the color image of the current frame image into a grayscale image;

[0052] For each pixel in the grayscale image, determining a brightness level value of the pixel according to the grayscale value of the pixel and a preset number of brightness levels;

[0053] According to the brightness level value of each pixel, a brightness map corresponding to the current frame image is obtained.

[0054] In one embodiment, determining the brightness level value of the pixel according to the grayscale value of the pixel and the number of preset brightness levels includes:

[0055] The brightness level value of the pixel is calculated based on the grayscale value of the pixel, the number of preset brightness levels and the brightness level calculation formula; wherein:

[0056] The brightness level calculation formula is: h=A*n / 255, where h represents the brightness level value of the pixel, A represents the grayscale value of the pixel, and n represents the number of preset brightness levels.

[0057] In one embodiment, the updating of the existing texture image stored in the surface voxel using the current frame image for the current texture image of the surface voxel according to the magnitude relationship between the brightness level value and the preset brightness level value to obtain the updated texture image corresponding to the surface voxel includes:

[0058] When the brightness level value is greater than the preset brightness level value, directly using the existing texture image stored in the surface voxel as the updated texture image corresponding to the surface voxel;

[0059] When the brightness level value is less than the preset brightness level value, updating the existing texture image stored in the surface voxel to the current texture image of the current frame image for the surface voxel, to obtain an updated texture image corresponding to the surface voxel;

[0060] When the brightness level value is equal to the preset brightness level value, the current frame image is fused with the current texture image of the surface voxel and the stored texture image stored in the surface voxel, and the fused texture image is used as the updated texture image corresponding to the surface voxel.

[0061] In one embodiment, fusing the current texture image of the current frame image for the surface voxel with the stored texture image stored in the surface voxel includes:

[0062] Calculating the angle between the viewpoint direction of the current frame image and the surface normal of the surface voxel;

[0063] Calculating a camera motion distance between the current frame image and the previous frame image according to a current camera pose corresponding to the current frame image and a camera pose corresponding to the previous frame image;

[0064] Normalizing the camera motion distance to obtain a normalized camera motion distance;

[0065] Based on the normalized movement distance, the angle, the current texture image of the current frame image for the surface voxel and the stored texture image stored in the surface voxel, a fusion calculation is performed according to a texture image fusion relationship.

[0066] In one embodiment, the texture image fusion relationship is:

[0067] I_new(i, j) = (1-|cos(a)|*cos(s))*I(i, j) + |cos(a)|*cos(s)*I_input(i, j), where (i, j) is the pixel coordinate, I_new(i, j) represents the pixel value of the fused pixel (i, j), I(i, j) represents the pixel value of the pixel (i, j) in the stored texture image stored in the surface voxel, I_input(i, j) represents the pixel value of the pixel (i, j) in the current texture image of the current frame image for the surface voxel, a represents the angle between the viewpoint direction of the current frame image and the surface normal of the surface voxel, and S represents the normalized camera motion distance.

[0068] In one embodiment, the calculating the three-dimensional coordinates corresponding to each pixel in the current frame image based on the current camera pose and the image pixel position of each pixel in the current frame image includes:

[0069] For each pixel in the current frame image, the three-dimensional coordinates of the pixel are calculated using a three-dimensional coordinate calculation formula based on the current camera pose and the image pixel position of the pixel in the current frame image; wherein the three-dimensional coordinate calculation formula is:

[0070] p=P*I rgb (u, v), where p represents the three-dimensional coordinates of the pixel, P represents the current camera pose, and I rgb (u, v) represents the pixel at the image pixel position (u, v).

[0071] An embodiment of the present invention further provides a three-dimensional scene reconstruction device, comprising:

[0072] A first updating module is configured to update the current three-dimensional geometric model based on the current frame image to obtain an updated three-dimensional geometric model;

[0073] A second updating module is configured to update an existing texture image stored in a surface voxel in the updated three-dimensional geometric model using a brightness image obtained based on the current frame image to obtain an updated texture image;

[0074] A judging module, configured to judge whether the current frame image is the last frame image, and if so, trigger the determining module; otherwise, trigger the returning module;

[0075] The determining module is configured to use the updated three-dimensional geometric model as the reconstructed three-dimensional geometric model, and use the updated texture images corresponding to the surface voxels as the reconstructed texture images;

[0076] The return module is used to take the next frame image as the current frame image and return to execute the step of fusing the current three-dimensional geometric model based on the current frame image.

[0077] An embodiment of the present invention further provides an electronic device, including:

[0078] memory for storing computer programs;

[0079] A processor is used to implement the steps of the three-dimensional scene reconstruction method as described above when executing the computer program.

[0080] A computer-readable storage medium is also provided in an embodiment of the present invention. A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the steps of the three-dimensional scene reconstruction method described above are implemented.

[0081] An embodiment of the present invention provides a three-dimensional scene reconstruction method, device, electronic device and computer-readable storage medium, the method comprising: updating a current three-dimensional geometric model based on a current frame image to obtain an updated three-dimensional geometric model; using a brightness map obtained based on the current frame image to update an existing texture image stored inside a surface voxel in the updated three-dimensional geometric model to obtain an updated texture image; determining whether the current frame image is the last frame image, and if so, using the updated three-dimensional geometric model as the reconstructed three-dimensional geometric model, using the updated texture image corresponding to each surface voxel as the reconstructed texture image, and obtaining a reconstructed three-dimensional scene based on the reconstructed three-dimensional geometric model and the reconstructed texture image; otherwise, using the next frame image as the current frame image and returning to execute the step of updating the current three-dimensional geometric model based on the current frame image.

[0082] It can be seen that in the embodiment of the present invention, the current 3D geometric model is updated according to the current frame image to reconstruct the 3D geometric model, and a corresponding texture image is stored in each surface voxel in the current 3D geometric model, which is an existing texture image, thereby combining the texture image with the geometric factors, and updating the existing texture image stored in each surface voxel in the updated 3D geometric model according to the brightness map obtained based on the current texture image, thereby obtaining updated texture images corresponding to each surface voxel, and continuing to obtain the next frame image to continuously update the new current 3D geometric model, and update the existing texture image of each surface voxel, until all images complete the above operations to obtain the reconstructed 3D geometric model and texture image, and form a reconstructed 3D scene. In the present invention, the texture image is reconstructed by combining the geometric and brightness factors in the 3D reconstruction process, which is beneficial to improving the accuracy of 3D scene reconstruction and the quality of reconstructed texture. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the prior art and the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0084] Figure 1 A schematic flow chart of a three-dimensional scene reconstruction method provided by an embodiment of the present invention;

[0085] Figure 2 A schematic diagram of a process for updating a three-dimensional geometric model provided by an embodiment of the present invention;

[0086] Figure 3 A schematic diagram of a texture image update process provided by an embodiment of the present invention;

[0087] Figure 4 A schematic diagram of the surface normal of a surface voxel provided in an embodiment of the present invention;

[0088] Figure 5 A schematic diagram of a secondary projection sampling calculation provided by an embodiment of the present invention;

[0089] Figure 6 A schematic structural diagram of a three-dimensional scene reconstruction device provided by an embodiment of the present invention;

[0090] Figure 7 A schematic structural diagram of an electronic device provided by an embodiment of the present invention;

[0091] Figure 8 A schematic structural diagram of a computer-readable storage medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0092] Embodiments of the present invention provide a three-dimensional scene reconstruction method, device, electronic device and computer-readable storage medium. During use, texture images are reconstructed by combining geometric and brightness factors, which is beneficial to improving the accuracy of three-dimensional scene reconstruction and the quality of reconstructed textures.

[0093] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0094] Please refer to Figure 1 , Figure 1 A schematic flow chart of a three-dimensional scene reconstruction method provided by an embodiment of the present invention. The method includes:

[0095] S110: updating the current three-dimensional geometric model based on the current frame image to obtain an updated three-dimensional geometric model;

[0096] It should be noted that in the embodiment of the present invention, a series of image data can be scanned and collected in chronological order, and the steps of the present invention are executed every time a frame of image is collected, until the last frame of image is processed to obtain a reconstructed three-dimensional geometric model and texture image.

[0097] Specifically, in an embodiment of the present invention, the current frame image currently captured is used to update the current three-dimensional geometric model, thereby obtaining an updated three-dimensional geometric model, wherein the current three-dimensional geometric model is reconstructed based on a series of previously captured images.

[0098] S120: using the brightness image obtained based on the current frame image to update the existing texture image stored in the surface voxels in the updated three-dimensional geometric model to obtain an updated texture image;

[0099] Specifically, in an embodiment of the present invention, each surface voxel in the current three-dimensional geometric model stores a corresponding texture image, and the texture image currently stored in the surface voxel is a stored texture image. After acquiring the current frame image, a brightness image corresponding to the current frame image can also be obtained based on the current frame image. Then, for each surface voxel in the updated three-dimensional geometric model, the stored texture image in each surface voxel is updated using the brightness image of the current three-dimensional geometric model, thereby obtaining an updated texture image corresponding to each surface voxel.

[0100] S130: Determine whether the current frame image is the last frame image. If so, proceed to S140; otherwise, proceed to S150;

[0101] It should be noted that after the current three-dimensional geometric model and the stored texture images in each surface voxel are updated based on the current frame image, it is further determined whether the current frame image is the last frame image. If the current frame image is the last frame image, enter S140; if the current frame image is not the last frame image, enter S150.

[0102] S140: Using the updated 3D geometric model as a reconstructed 3D geometric model, using the updated texture images corresponding to the respective surface voxels as reconstructed texture images, and obtaining a reconstructed 3D scene based on the reconstructed 3D geometric model and the reconstructed texture images;

[0103] Specifically, when it is determined that the current frame image is the last frame image, the updated three-dimensional geometric model obtained above is the final reconstructed three-dimensional geometric model, and the updated texture images in each of the surface voxels above constitute the reconstructed texture image as a whole. The reconstructed three-dimensional geometric model and the reconstructed texture image constitute the reconstructed three-dimensional scene, and the reconstruction is completed, and the operation ends. The three-dimensional geometric model in the three-dimensional scene has multiple surface voxels, each of which stores a corresponding texture image. The texture images of each surface voxel constitute the surface texture of the three-dimensional scene. Each voxel in the three-dimensional geometric model stores a corresponding distance symbol, and the distance symbols of each voxel constitute the three-dimensional geometric model.

[0104] S150: taking the next frame image as the current frame image and returning to execute the step of updating the current three-dimensional geometric model based on the current frame image.

[0105] Specifically, when the current frame image is not the last frame image, the next frame image is used as the current frame image, and returns to S110 to update the three-dimensional geometric model and texture image of the new frame image until the last frame image operation is completed, and the final reconstructed three-dimensional geometric model and texture image are obtained, and the operation is ended.

[0106] It can be seen that in the embodiment of the present invention, the current 3D geometric model is updated according to the current frame image to reconstruct the 3D geometric model, and a corresponding texture image is stored in each surface voxel in the current 3D geometric model, which is an existing texture image, thereby combining the texture image with the geometric factors, and updating the existing texture images stored in each surface voxel in the current 3D geometric model according to the brightness map obtained based on the current texture image, thereby obtaining updated texture images corresponding to each surface voxel, and continuing to obtain the next frame image to continuously update the new current 3D geometric model, and update the existing texture image of each surface voxel, until all images complete the above operations to obtain the reconstructed 3D geometric model and texture image. In the present invention, the texture image is reconstructed by combining the geometric and brightness factors in the 3D reconstruction process, which is beneficial to improving the accuracy of 3D scene reconstruction and the quality of reconstructed texture.

[0107] Based on the above embodiment, the technical solution is further introduced and explained below, specifically as follows:

[0108] In one embodiment, Figure 2 As shown, the process of updating the current 3D geometric model based on the current frame image in S110 to obtain the updated 3D geometric model may specifically include:

[0109] S210: updating the signed distance stored in each voxel in the current three-dimensional geometric model according to the current frame image to obtain an updated three-dimensional geometric model.

[0110] It is understood that in practical applications, multiple voxels are pre-divided, and for each voxel in the current 3D geometric model, a signed distance is stored. Specifically, this may be the original signed distance in the TSDF (truncated signed distance function) model, where the signed distance refers to the closest distance from the voxel to the reconstructed object surface. Specifically, the signed distance stored in each voxel in the current 3D geometric model can be updated based on the current frame image, thereby obtaining an updated signed distance corresponding to each voxel, thereby obtaining an updated 3D geometric model.

[0111] Furthermore, the process of updating the signed distance stored in each voxel in the current three-dimensional geometric model according to the current frame image to obtain the updated three-dimensional geometric model may specifically include:

[0112] S220: Based on the current frame image and the previous frame image, obtain a current camera pose corresponding to the current frame image;

[0113] Specifically, by extracting feature description of the current frame image, multiple current image features can be obtained, and then the multiple current image features of the current frame image are feature matched with the multiple image features of the previous frame image to obtain the correspondence between the image features. Based on the correspondence between the image features, the iterative nearest point algorithm is used to determine the current camera pose corresponding to the current frame image.

[0114] In other words, after capturing the current frame image, a feature description is first extracted for the current frame image. This feature description can be a manually designed feature such as SIFT (Scale-invariant feature transform) or an image feature generated through machine learning, which is not specifically limited in the embodiments of the present invention. After extracting the feature description for the current frame image, multiple current image features can be extracted. The previous frame image also corresponds to multiple image features. Therefore, feature matching calculations can be performed on the multiple current image features of the current frame image and the multiple image features of the previous frame image to obtain a corresponding relationship between the image features of the current frame image and the previous frame image. For example, current image feature 01 of the current frame image corresponds to image feature 11 of the previous frame image, current image feature 02 of the current frame image corresponds to image feature 12 of the previous frame image, and so on. After obtaining the corresponding relationship between the image features, the six-degree-of-freedom pose of the camera corresponding to the current frame image in three-dimensional space can be further estimated based on the corresponding relationship between the image features using an algorithm such as ICP (Iterative Closest Point algorithm). In other words, the current camera pose corresponding to the current frame image is obtained.

[0115] S230: Calculating the three-dimensional coordinates corresponding to each pixel in the current frame image based on the current camera pose and the image pixel position of each pixel in the current frame image;

[0116] It should be noted that after obtaining the current camera pose of the current frame image, the three-dimensional coordinates of each pixel in the current frame image in the three-dimensional space can be further calculated based on the current camera pose combined with the image pixel position of each pixel in the current frame image (that is, the pixel coordinates of the pixel in the current frame image).

[0117] Specifically, for each pixel in the current frame image, the three-dimensional coordinates of the pixel in the three-dimensional space can be calculated using a three-dimensional coordinate calculation formula based on the current camera pose and the image pixel position of the pixel in the current frame image; wherein the three-dimensional coordinate calculation formula is:

[0118] p=P*I rgb (u, v), where p represents the three-dimensional coordinates of the pixel, P represents the current camera pose, and I rgb (u, v) represents the pixel at the image pixel position (u, v).

[0119] It is understandable that, in the embodiment of the present invention, for each pixel in the current frame image, the pixel I at the image pixel position (u, v) can be obtained according to the current frame image. rgb(u, v), and then the three-dimensional coordinate p corresponding to the pixel can be calculated by combining the above relationship.

[0120] S240: Determine, based on the voxels of the current three-dimensional geometric model, the voxels corresponding to each pixel in the current frame image;

[0121] Specifically, after calculating the three-dimensional coordinates of the pixel, the voxel G corresponding to the pixel in the TSDF model can be determined based on the three-dimensional coordinates of the pixel, so that the voxel corresponding to each pixel in the current frame image can be determined.

[0122] S250: For each voxel, calculate the current symbol distance according to the three-dimensional coordinates of the pixel corresponding to the voxel;

[0123] Specifically, for each voxel in the current three-dimensional geometric model, the current signed distance of the voxel can be calculated based on the three-dimensional coordinates of each pixel in the current frame image corresponding to the voxel. For example, pixel 1 and pixel 2 in the current frame image both correspond to voxel G', then the current signed distance of voxel G' can be calculated based on the three-dimensional coordinates of pixel 1 and the three-dimensional coordinates of pixel 2.

[0124] S260: Using the current signed distance and the stored signed distance in the voxel to obtain an updated signed distance, and using the updated signed distance to replace the stored signed distance, until the stored signed distances of all voxels are updated, thereby obtaining an updated three-dimensional geometric model.

[0125] It should be noted that after calculating the current signed distance of a voxel, to improve update accuracy, a weighted average can be performed based on the current signed distance and the previously stored signed distance for that voxel. The weighted average result is used as the new signed distance to update the previously stored signed distance for that voxel, thereby updating the previously stored signed distance for that voxel to the new signed distance. The above method is used to update the signed distance for each voxel, thereby obtaining an updated three-dimensional geometric model.

[0126] In one embodiment, please refer to Figure 3 The process of updating the stored texture image stored in the surface voxels in the updated three-dimensional geometric model using the brightness image obtained based on the current frame image in S120 to obtain the updated texture image may specifically include:

[0127] S310: determining each surface voxel from each voxel of the updated three-dimensional geometric model;

[0128] It should be noted that, based on the updated three-dimensional geometric model, the voxels covered by the updated three-dimensional geometric model are determined, and then the zero-value surface of the updated three-dimensional geometric model can be determined based on the signed distance stored in each voxel. The voxels through which the zero-value surface passes are surface voxels.

[0129] S320: Determine, according to the current frame image, a current texture image for each surface voxel of the current frame image;

[0130] Specifically, after each surface voxel of the updated three-dimensional geometric model is determined, a current texture image corresponding to each surface voxel may be determined according to the current frame image.

[0131] It can be understood that, specifically, for each surface voxel, the pixels corresponding to the surface voxel can be determined according to the three-dimensional coordinates corresponding to each pixel of the current frame image. For example, the pixels corresponding to surface voxel 1 are pixel 1 and pixel 2, and the pixels corresponding to surface voxel 2 are pixel 3 and pixel 4, etc. Then, the surface normal of the surface voxel is calculated based on the stored signed distance stored in the surface voxel. Specifically, the surface normal of the surface voxel can be calculated based on the signed distances corresponding to the eight vertices of the surface voxel, and the angle between the viewpoint direction of the current frame image and the surface normal of the surface voxel is further calculated, as shown in FIG. Figure 4 As shown, the viewpoint direction is Figure 4 The angle between the viewpoint direction of the current frame image and the surface normal of the surface voxel is a. It should also be noted that in the embodiment of the present invention, the viewpoint direction of the current frame image can be obtained according to the camera pose of the current frame image.

[0132] Specifically, after obtaining the angle a between the viewpoint direction of the current frame image and the surface normal of the surface voxel, each pixel corresponding to the surface voxel is projected along the surface normal direction of the surface voxel according to the angle to obtain the corresponding projection area. That is, for each pixel corresponding to the surface voxel, each pixel is projected along the surface normal direction of the surface voxel (that is, secondary projection mapping) to obtain the corresponding projection area, and then the projection area relative to the surface voxel is sampled using preset pixels, and the pixel value of each sampled pixel in the sampling area is obtained to obtain the current texture image of the current frame image for the surface voxel, that is, the current texture image projected onto the surface voxel is obtained. Since the camera is not perpendicular to the surface of the object in actual applications, in order to make the image seen when observing perpendicular to the surface of the object a front view, the projection area is sampled by the sampling area of ​​preset pixels in the embodiment of the present invention, so that the quality of the texture image can be improved. Among them, for the specific positional relationship between the projection area and the sampling area, please refer to Figure 5 Schematic diagram of the secondary projection sampling calculation shown.

[0133] Furthermore, the process of sampling the projection area of ​​the relative surface voxel using the preset pixels to obtain the pixel value of each sampled pixel in the sampling area may include:

[0134] Determine the pixel positions of each vertex in the sampling area based on the preset pixel size and projection area;

[0135] The pixel coordinates of the corresponding sampling pixels in the sampling area are calculated based on the preset pixel size, the pixel positions of each vertex in the sampling area, and the image pixel positions of the pixels corresponding to the surface voxels in the original sampling image, in combination with the pixel coordinate calculation relationship.

[0136] The pixel value of the sampling pixel at the pixel coordinate in the sampling area is calculated according to the pixel value of the target pixel corresponding to the pixel coordinate in the original sampling image.

[0137] It should be noted that if Figure 5 As shown, the vertex pixel positions of each vertex in the sampling area can be further determined according to the preset pixel size (w×h) and the position of the projection area, for example, (U min ,V min )、(U min ,V max )、(U max ,V min ) and (U max ,V max ), then the four vertices (U min ,V min )、(U min ,V max )、(U max ,V min ) and (U max ,V max ) is used as the base pixel. Other sampled pixels are mapped to a w*h pixel area based on their distance from the four vertex pixels in the U and V directions. Combined with the image pixel position of the pixel corresponding to the surface voxel in the original sampled image, the pixel coordinates of each sampled pixel in each sampling area are determined. Then, for each pixel coordinate, the pixel value of the sampled pixel at the pixel coordinate in the sampling area is calculated based on the pixel value of the target pixel corresponding to the pixel coordinate in the original sampled image. The target pixel is the pixel whose pixel coordinates are obtained by the above calculation. For example, if the pixel coordinates of pixel 1 calculated by the pixel coordinate calculation relationship are pixel coordinates 1, then the target pixel corresponding to pixel coordinates 1 is pixel 1.

[0138] It should be noted that the pixel coordinate calculation relationship in the embodiment of the present invention can be:

[0139] i=(UU min )*w / (U max -U min ),

[0140] j=(VV min )*h / (V max -V min ), where i represents the horizontal coordinate of the sampled pixel, j represents the vertical coordinate of the sampled pixel, and U max Indicates the maximum value of the horizontal coordinate of the sampling area, U min Indicates the minimum value of the horizontal coordinate of the sampling area, V max Indicates the maximum value of the vertical coordinate of the sampling area, V min Indicates the minimum value of the horizontal coordinate of the sampling area, U indicates the horizontal coordinate of the image pixel corresponding to the pixel, and V indicates the vertical coordinate of the image pixel corresponding to the pixel.

[0141] Furthermore, in practical applications, multiple pixels may have the same calculated pixel coordinates. In this case, the pixel coordinates correspond to multiple target pixels. Therefore, for a pixel coordinate, when there are multiple target pixels corresponding to the pixel coordinates, a weighted average of the pixel values ​​of the multiple target pixels in the original sampled image can be performed to obtain the pixel value of the sampled pixel at the pixel coordinates in the sampling area.

[0142] Specifically, after calculating the pixel value of each sampling pixel in the sampling area, the current texture image corresponding to the sampling area can be obtained, that is, the current texture image corresponding to the corresponding surface voxel can be obtained.

[0143] S330: Calculating the brightness level value of the sampling area corresponding to the surface voxel according to the brightness map obtained based on the current frame image;

[0144] It should be noted that in the embodiment of the present invention, the brightness and texture of the current value image are separated. Then, in order to further determine the impact of brightness on the texture image, the brightness level value of the sampling area corresponding to the surface voxel can be calculated based on the brightness map obtained based on the current frame image.

[0145] Specifically, the directly captured image is a color image. Therefore, after the current frame image is captured, a brightness image corresponding to the current frame image can be further obtained based on the color image of the current frame image.

[0146] Specifically, the color image of the current frame image can be converted into a grayscale image I with a value of 0-255. b , specifically, it can be based on the following formula based on the general grayscale conversion method:

[0147] I h =0.299*I rgb (R)+0.587*Irgb (G)+0.114*I rgb (B), where I rgb (R),

[0148] I rgb (G), I rgb (B) represent the color images I rgb The R, G, and B color components of each pixel.

[0149] Specifically, after obtaining the grayscale image, the brightness level value of each pixel in the grayscale image can be determined according to the grayscale value of the pixel and the number of preset brightness levels, and then a brightness map corresponding to the current frame image can be obtained based on the brightness level value of each pixel. Specifically, the brightness level value of the pixel can be calculated based on the grayscale value of the pixel, the number of preset brightness levels, and the brightness level calculation relationship; where:

[0150] The brightness level calculation formula is: h=A*n / 255, where h represents the brightness level value of the pixel, A represents the grayscale value of the pixel, and n represents the number of preset brightness levels.

[0151] That is, 0-255 can be divided into n levels in advance, for example, n=10, and then 0-255 can be evenly divided into ten brightness levels of 0-9, so that the brightness level corresponding to each pixel can be calculated according to the above relationship. After obtaining the brightness level of each pixel, the brightness map corresponding to the current frame image can be obtained.

[0152] Specifically, after obtaining the brightness map corresponding to the current frame image, the brightness level values ​​of each pixel corresponding to the surface voxel can be obtained according to the brightness map, and then the brightness level values ​​of each pixel corresponding to the surface voxel are averaged to obtain the brightness level values ​​of the surface voxel.

[0153] That is, for each surface voxel, the pixels corresponding to the surface voxel in the current frame image can be determined, and then the brightness level values ​​of each pixel can be obtained according to the brightness map, and the average brightness level value can be further calculated based on the brightness level values ​​of each pixel, and the average brightness level value can be used as the brightness level value of the surface voxel.

[0154] S340: updating the existing texture image stored in the surface voxel using the current frame image according to the magnitude relationship between the brightness level value and the preset brightness level value, thereby obtaining an updated texture image corresponding to the surface voxel;

[0155] It should be noted that, for each surface voxel, the brightness level value h_avg of the surface voxel can be compared with the preset brightness level H to determine the size relationship between the two, and then further determine how to update the stored texture image stored in the surface voxel based on the size relationship.

[0156] Specifically, when the brightness level value is greater than the preset brightness level value, the texture image stored in the surface voxel is directly used as the updated texture image corresponding to the surface voxel; that is, if h_avg is greater than H, it can be considered that the current frame image I rgb The texture map corresponding to the surface voxel area is a highlight area. At this time, the external light interference is relatively large, and the texture map of the current voxel G is not fused, that is, the existing texture image stored in the surface voxel is retained.

[0157] When the brightness level value is less than the preset brightness level value, the stored texture image stored in the surface voxel is updated to the current texture image of the current frame image for the surface voxel, and the updated texture image corresponding to the surface voxel is obtained; that is, if h_avg is less than H, it is determined that the stored texture image stored in the surface voxel is the texture of the highlight area relative to the current texture image of the current frame image for the surface voxel, and the stored texture image stored in the surface voxel is replaced with the texture image of the current frame image I rgb The calculated w*h texture image block corresponding to the voxel (ie, replaced with the calculated current texture image for the surface voxel of the current frame image), and H=h_avg may also be set.

[0158] When the brightness level value is equal to the preset brightness level value, the current texture image of the surface voxel in the current frame image and the stored texture image stored in the surface voxel are fused, and the fused texture image is used as the updated texture image corresponding to the surface voxel.

[0159] It can be understood that when h_avg is equal to H, it means that the current texture image of the current frame image for the surface voxel and the stored texture image stored in the surface voxel are not highlight areas. At this time, the current texture image of the current frame image for the surface voxel and the stored texture image stored in the surface voxel can be fused.

[0160] Specifically, the camera motion distance between the current frame image and the previous frame image can be calculated based on the current camera pose corresponding to the current frame image and the camera pose corresponding to the previous frame image, and then the camera motion distance is normalized to obtain the normalized camera motion distance; for example, the camera motion distance between the current frame image and the previous frame image is s_cur, then the preset value s_max can be used to normalize the camera motion distance s_cur to the [0,1] interval: s = min(1,s_cur / s_max).

[0161] Specifically, based on the normalized motion distance, the angle between the viewpoint direction of the current frame image and the surface normal of the surface voxel, the current texture image of the current frame image for the surface voxel, and the stored texture image stored in the surface voxel, the fusion calculation is performed according to the texture image fusion relationship. The texture image fusion relationship is:

[0162] I_new(i, j) = (1-|cos(a)|*cos(s))*I(i, j) + |cos(a)|*cos(s)*I_input(i, j), where (i, j) is the pixel coordinate, I_new(i, j) represents the pixel value of the fused pixel (i, j), I(i, j) represents the pixel value of the pixel (i, j) in the existing texture image stored in the surface voxel, I_input(i, j) represents the pixel value of the pixel (i, j) in the current texture image for the surface voxel in the current frame image, a represents the angle between the viewpoint direction of the current frame image and the surface normal of the surface voxel, and S represents the normalized camera motion distance. That is, after obtaining the corresponding pixel value for each pixel, the fused texture image can be obtained, and the fused texture image is used as the updated texture image of the surface voxel.

[0163] S350: Obtaining an overall updated texture image based on the updated texture images corresponding to the respective surface voxels.

[0164] It should be noted that after obtaining the updated texture image corresponding to each surface voxel, the updated texture image corresponding to the current frame image can be obtained. The above steps are performed on all collected images to obtain the final reconstructed texture image.

[0165] Thus, the embodiments of the present invention address the problem of high-quality texture reconstruction in 3D reconstruction by designing a texture reconstruction method based on brightness separation. By separately calculating and processing the texture image and brightness map, and simultaneously detecting and removing highlights during the texture image generation process, high-quality texture images can be generated during 3D reconstruction. Specifically, by separately calculating and mapping image brightness, color and brightness interference factors, such as highlights, are separated during texture reconstruction. This solves the problem of poor reconstructed texture images due to coupled ambient light illumination in the input color image, resulting in a texture image that more closely resembles the object's original color. Furthermore, the texture mapping generation method based on a reprojection mechanism in the embodiments of the present invention, by performing reprojection mapping along the surface normal direction of surface voxels, can make the reconstructed texture closer to the front view image and reduce distortion in the texture image. Furthermore, the texture fusion method based on the texture map and brightness map in the embodiments of the present invention identifies highlight regions in the image by analyzing brightness variations in the same region across different input images, and removes these regions during the texture fusion process. This solves the problem of image highlight regions affecting the quality of the reconstructed texture, thereby improving the quality of the reconstructed texture.

[0166] On the basis of the above embodiments, a three-dimensional scene reconstruction device is also provided in the embodiment of the present invention. Figure 6 , the device comprises:

[0167] A first updating module 11 is configured to update the current three-dimensional geometric model based on the current frame image to obtain an updated three-dimensional geometric model;

[0168] A second updating module 12 is configured to update the existing texture image stored in the surface voxels in the updated three-dimensional geometric model using the brightness image obtained based on the current frame image to obtain an updated texture image;

[0169] The judging module 13 is used to judge whether the current frame image is the last frame image. If so, the determining module 14 is triggered; otherwise, the returning module 15 is triggered.

[0170] a determination module 14 configured to use the updated 3D geometric model as a reconstructed 3D geometric model, use the updated texture images corresponding to the respective surface voxels as reconstructed texture images, and obtain a reconstructed 3D scene based on the reconstructed 3D geometric model and the reconstructed texture images;

[0171] The return module 15 is used to return the next frame image as the current frame image to execute the step of updating the current three-dimensional geometric model based on the current frame image.

[0172] In one embodiment, the first updating module 11 is configured to:

[0173] The signed distance stored in each voxel in the current three-dimensional geometric model is updated according to the current frame image to obtain an updated three-dimensional geometric model.

[0174] In one embodiment, the first update module 11 includes:

[0175] A first calculation unit is used to obtain a current camera pose corresponding to the current frame image based on the current frame image and the previous frame image;

[0176] A second calculation unit is configured to calculate the three-dimensional coordinates corresponding to each pixel in the current frame image based on the current camera pose and the image pixel position of each pixel in the current frame image;

[0177] A first determining unit is configured to determine, based on each voxel of the current three-dimensional geometric model, a voxel corresponding to each pixel in the current frame image;

[0178] a third calculation unit, configured to calculate, for each voxel, a current signed distance according to an image pixel position of a pixel corresponding to the voxel;

[0179] A first updating unit is configured to obtain an updated signed distance using the current signed distance and the stored signed distance stored in the voxel, and to replace the stored signed distance with the updated signed distance until the stored signed distances of all voxels are updated, thereby obtaining an updated three-dimensional geometric model.

[0180] In one embodiment, the first computing unit includes:

[0181] an extraction subunit, configured to extract feature descriptions of the current frame image to obtain a plurality of current image features;

[0182] a matching subunit, configured to perform feature matching on a plurality of current image features of the current frame image and a plurality of image features of the previous frame image to obtain a correspondence between the image features;

[0183] The first calculation subunit is used to determine the current camera pose corresponding to the current frame image by using an iterative closest point algorithm based on the correspondence between the image features.

[0184] In one embodiment, the second updating module 12 includes:

[0185] A second determining unit, configured to determine each surface voxel from each voxel of the updated three-dimensional geometric model;

[0186] a third determining unit, configured to determine, based on the current frame image, a current texture image of the current frame image for each of the surface voxels;

[0187] a fourth calculation unit, configured to calculate a brightness level value of a sampling area corresponding to the surface voxel according to a brightness map obtained based on the current frame image;

[0188] a second updating unit, configured to update an existing texture image stored in the surface voxel using the current frame image according to a magnitude relationship between the brightness level value and a preset brightness level value, to obtain an updated texture image corresponding to the surface voxel;

[0189] The fifth determining unit is configured to obtain an overall updated texture image according to the updated texture images corresponding to the respective surface voxels.

[0190] In one embodiment, the third determining unit includes:

[0191] A first determining subunit is configured to determine, for each of the surface voxels, the pixels corresponding to the surface voxel according to the three-dimensional coordinates corresponding to the pixels of the current frame image;

[0192] a projection subunit, configured to project each pixel corresponding to the surface voxel along a surface normal direction of the surface voxel to obtain a projection area relative to the surface voxel;

[0193] The sampling subunit is configured to sample the projection area relative to the surface voxel using preset pixels to obtain a pixel value of each sampling pixel in the sampling area, so as to obtain a current texture image of the current frame image for the surface voxel.

[0194] In one embodiment, the projection subunit includes:

[0195] a second calculation subunit, configured to calculate a surface normal of the surface voxel according to the stored signed distance stored in the surface voxel;

[0196] A third calculation subunit is used to calculate the angle between the viewpoint direction of the current frame image and the surface normal of the surface voxel;

[0197] The first projection subunit is configured to project each pixel corresponding to the surface voxel along a surface normal direction of the surface voxel based on the included angle to obtain a corresponding projection area.

[0198] In one embodiment, the third calculation subunit is used to determine the viewpoint direction of the current frame image according to the current camera pose corresponding to the current frame image, and calculate the angle between the viewpoint direction of the current frame image and the surface normal of the surface voxel.

[0199] In one embodiment, the sampling subunit includes:

[0200] A first determining subunit, configured to determine a pixel position of each vertex of a sampling area according to the size of the preset pixel and the projection area;

[0201] a fourth calculation subunit, configured to calculate pixel coordinates of corresponding sampling pixels in the sampling area based on a preset pixel size, pixel positions of vertices of the sampling area, and image pixel positions of pixels corresponding to the surface voxels in the original sampling image, in combination with a pixel coordinate calculation formula;

[0202] The fifth calculation subunit is configured to calculate the pixel value of the sampling pixel at the pixel coordinates in the sampling area according to the pixel value of the target pixel corresponding to the pixel coordinates in the original sampling image.

[0203] In one embodiment, the fifth computing subunit is configured to:

[0204] When there are multiple target pixels corresponding to the pixel coordinates, a weighted average is performed on the pixel values ​​of the multiple target pixels in the original sampling image to obtain the pixel value of the sampling pixel at the pixel coordinates in the sampling area.

[0205] In one embodiment, the pixel coordinate calculation relationship is:

[0206] i=(UU min )*w / (U max -U min ),

[0207] j=(VV min )*h / (V max -V min ), where i represents the horizontal coordinate of the sampled pixel, j represents the vertical coordinate of the sampled pixel, and U max Indicates the maximum value of the horizontal coordinate of the sampling area, U min Indicates the minimum value of the horizontal coordinate of the sampling area, V max Indicates the maximum value of the vertical coordinate of the sampling area, V min Indicates the minimum value of the horizontal coordinate of the sampling area, U indicates the horizontal coordinate of the image pixel corresponding to the pixel, and V indicates the vertical coordinate of the image pixel corresponding to the pixel.

[0208] In one embodiment, the fourth computing unit includes:

[0209] an acquisition subunit, configured to acquire a brightness level value of each pixel corresponding to the surface voxel according to a brightness map obtained based on the current frame image;

[0210] The sixth calculation subunit is configured to average the brightness level values ​​of the pixels corresponding to the surface voxel to obtain the brightness level value of the surface voxel.

[0211] In one embodiment, it further includes:

[0212] The conversion unit obtains a brightness image corresponding to the current frame image according to the color image of the current frame image.

[0213] In one embodiment, the conversion unit includes:

[0214] a conversion subunit, configured to convert the color image of the current frame image into a grayscale image;

[0215] a seventh calculation subunit, configured to determine, for each pixel in the grayscale image, a brightness level value of the pixel according to the grayscale value of the pixel and a preset number of brightness levels;

[0216] The eighth calculation subunit is used to obtain a brightness map corresponding to the current frame image according to the brightness level value of each pixel.

[0217] In one embodiment, the eighth computing subunit is configured to:

[0218] The brightness level value of the pixel is calculated based on the grayscale value of the pixel, the number of preset brightness levels and the brightness level calculation formula; wherein:

[0219] The brightness level calculation formula is: h=A*n / 255, where h represents the brightness level value of the pixel, A represents the grayscale value of the pixel, and n represents the number of preset brightness levels.

[0220] In one embodiment, the second updating unit includes:

[0221] a first updating subunit, configured to directly use the existing texture image stored in the surface voxel as an updated texture image corresponding to the surface voxel when the brightness level value is greater than the preset brightness level value;

[0222] a second updating subunit, configured to update the existing texture image stored in the surface voxel to the current texture image of the current frame image for the surface voxel when the brightness level value is less than the preset brightness level value, to obtain an updated texture image corresponding to the surface voxel;

[0223] A third updating subunit is configured to fuse the current texture image of the surface voxel with the existing texture image stored in the surface voxel in the current frame image when the brightness level value is equal to the preset brightness level value, and use the fused texture image as the updated texture image corresponding to the surface voxel.

[0224] In one embodiment, the third updating subunit includes:

[0225] a ninth calculation subunit, configured to calculate an angle between a viewpoint direction of the current frame image and a surface normal of the surface voxel;

[0226] a tenth calculating subunit, configured to calculate a camera motion distance between the current frame image and the previous frame image according to a current camera pose corresponding to the current frame image and a camera pose corresponding to the previous frame image;

[0227] a normalization subunit, configured to normalize the camera motion distance to obtain a normalized camera motion distance;

[0228] An eleventh calculation subunit is configured to perform fusion calculation according to a texture image fusion relationship based on the normalized motion distance, the angle, the current texture image of the surface voxel of the current frame image, and the stored texture image stored in the surface voxel.

[0229] In one embodiment, the texture image fusion relationship is:

[0230] I_new(i, j) = (1-|cos(a)|*cos(s))*I(i, j) + |cos(a)|*cos(s)*I_input(i, j), where (i, j) is the pixel coordinate, I_new(i, j) represents the pixel value of the fused pixel (i, j), I(i, j) represents the pixel value of the pixel (i, j) in the stored texture image stored in the surface voxel, I_input(i, j) represents the pixel value of the pixel (i, j) in the current texture image of the current frame image for the surface voxel, a represents the angle between the viewpoint direction of the current frame image and the surface normal of the surface voxel, and S represents the normalized camera motion distance.

[0231] In one embodiment, the second computing unit is configured to:

[0232] For each pixel in the current frame image, the three-dimensional coordinates of the pixel are calculated using a three-dimensional coordinate calculation formula based on the current camera pose and the image pixel position of the pixel in the current frame image; wherein the three-dimensional coordinate calculation formula is:

[0233] p=P*Irgb (u, v), where p represents the three-dimensional coordinates of the pixel, P represents the current camera pose, and I rgb (u, v) represents the pixel at the image pixel position (u, v).

[0234] It should be noted that the three-dimensional scene reconstruction device provided in the embodiment of the present invention has the same beneficial effects as the three-dimensional scene reconstruction method provided in the above embodiment, and for the specific introduction of the three-dimensional scene reconstruction method involved in the embodiment of the present invention, please refer to the above embodiment, and the present invention will not go into details here.

[0235] Figure 7 A structural diagram of an electronic device provided by an embodiment of the present invention, such as Figure 7 As shown, the electronic device includes: a memory 20 for storing computer programs;

[0236] The processor 21 is configured to implement the steps of the three-dimensional scene reconstruction method of the above embodiment when executing a computer program.

[0237] The electronic device provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a laptop computer, or a desktop computer.

[0238] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0239] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 20 is at least used to store the following computer program 201, wherein, after the computer program is loaded and executed by the processor 21, it can implement the relevant steps of the three-dimensional scene reconstruction method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 202 may include Windows, Unix, Linux, etc. The data 203 may include but is not limited to a set offset, etc.

[0240] In some embodiments, the electronic device may further include a display screen 22 , an input / output interface 23 , a communication interface 24 , a power supply 25 , and a communication bus 26 .

[0241] Those skilled in the art will understand that Figure 7 The structure shown in the figure does not constitute a limitation of the electronic device, and may include more or fewer components than shown in the figure.

[0242] It is understandable that if the three-dimensional scene reconstruction method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium and executes all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, magnetic disk or optical disk, etc. Various media that can store program code.

[0243] Based on this, Figure 8 As shown, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program 31 is stored. When the computer program 31 is executed by a processor, the steps of the above-mentioned three-dimensional scene reconstruction method are implemented.

[0244] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0245] It should also be noted that, in this specification, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0246] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A three-dimensional scene reconstruction method, characterized in that: include: updating the current three-dimensional geometric model based on the current frame image to obtain an updated three-dimensional geometric model; Using a brightness map obtained based on the current frame image, an existing texture image stored in a surface voxel in the updated three-dimensional geometric model is updated to obtain an updated texture image; Determine whether the current frame image is the last frame image. If so, use the updated 3D geometric model as the reconstructed 3D geometric model, use the updated texture images corresponding to each of the surface voxels as the reconstructed texture image, and obtain a reconstructed 3D scene based on the reconstructed 3D geometric model and the reconstructed texture image; otherwise, use the next frame image as the current frame image and return to execute the step of updating the current 3D geometric model based on the current frame image.

2. The three-dimensional scene reconstruction method according to claim 1, characterized in that: The updating of the current three-dimensional geometric model based on the current frame image to obtain an updated three-dimensional geometric model includes: The signed distance stored in each voxel in the current three-dimensional geometric model is updated according to the current frame image to obtain an updated three-dimensional geometric model.

3. The three-dimensional scene reconstruction method according to claim 2, characterized in that: The updating of the signed distance stored in each voxel in the current three-dimensional geometric model according to the current frame image to obtain an updated three-dimensional geometric model includes: Based on the current frame image and the previous frame image, obtaining a current camera pose corresponding to the current frame image; Calculating the three-dimensional coordinates corresponding to each pixel in the current frame image based on the current camera pose and the image pixel position of each pixel in the current frame image; Determining, based on each voxel of the current three-dimensional geometric model, a voxel corresponding to each pixel in the current frame image; For each voxel, calculating a current signed distance according to an image pixel position of a pixel corresponding to the voxel; The current signed distance and the stored signed distance stored in the voxel are used to obtain an updated signed distance, and the updated signed distance is used to replace the stored signed distance until the stored signed distances of all voxels are updated to obtain an updated three-dimensional geometric model.

4. The three-dimensional scene reconstruction method according to claim 3, characterized in that: The obtaining of a current camera pose corresponding to the current frame image based on the current frame image and the previous frame image includes: Performing feature description extraction on the current frame image to obtain multiple current image features; Performing feature matching on a plurality of current image features of the current frame image and a plurality of image features of the previous frame image to obtain a correspondence between the image features; Based on the correspondence between the image features, an iterative closest point algorithm is used to determine the current camera pose corresponding to the current frame image.

5. The three-dimensional scene reconstruction method according to any one of claims 1 to 4, characterized in that: The method of updating the stored texture image stored in the surface voxels in the updated three-dimensional geometric model using the brightness map obtained based on the current frame image to obtain the updated texture image includes: determining each surface voxel from each voxel of the updated three-dimensional geometric model; Determining, according to the current frame image, a current texture image of the current frame image for each of the surface voxels; Calculating a brightness level value of a sampling area corresponding to the surface voxel according to a brightness map obtained based on the current frame image; According to the magnitude relationship between the brightness level value and the preset brightness level value, using the current frame image for the current texture image of the surface voxel, updating the existing texture image stored in the surface voxel to obtain an updated texture image corresponding to the surface voxel; An overall updated texture image is obtained according to the updated texture images corresponding to the respective surface voxels.

6. The three-dimensional scene reconstruction method according to claim 5, characterized in that: The determining, based on the current frame image, a current texture image of each surface voxel of the current frame image includes: For each of the surface voxels, determining the pixels corresponding to the surface voxel according to the three-dimensional coordinates corresponding to the pixels of the current frame image; Projecting each pixel corresponding to the surface voxel along the surface normal direction of the surface voxel to obtain a projection area relative to the surface voxel; The projection area relative to the surface voxel is sampled using preset pixels to obtain a pixel value of each sampling pixel in the sampling area, so as to obtain a current texture image of the current frame image for the surface voxel.

7. The three-dimensional scene reconstruction method according to claim 6, characterized in that: The step of projecting each pixel corresponding to the surface voxel along the surface normal direction of the surface voxel to obtain a projection area relative to the surface voxel includes: Calculating a surface normal of the surface voxel based on the stored signed distance stored in the surface voxel; Calculating the angle between the viewpoint direction of the current frame image and the surface normal of the surface voxel; Based on the included angle, each pixel corresponding to the surface voxel is projected along the surface normal direction of the surface voxel to obtain a corresponding projection area.

8. The three-dimensional scene reconstruction method according to claim 7, characterized in that: The viewpoint direction of the current frame image is determined according to the current camera pose corresponding to the current frame image.

9. The three-dimensional scene reconstruction method according to claim 6, characterized in that: The step of sampling the projection area relative to the surface voxel using preset pixels to obtain a pixel value of each sampling pixel in the sampling area includes: Determining the pixel positions of each vertex in the sampling area according to the size of the preset pixel and the projection area; Calculating the pixel coordinates of the corresponding sampling pixels in the sampling area based on a preset pixel size, pixel positions of each vertex in the sampling area, and image pixel positions of pixels corresponding to the surface voxels in the original sampling image in combination with a pixel coordinate calculation formula; The pixel value of the sampling pixel at the pixel coordinates in the sampling area is calculated according to the pixel value of the target pixel corresponding to the pixel coordinates in the original sampling image.

10. The three-dimensional scene reconstruction method according to claim 9, characterized in that: The step of calculating the pixel value of the sampling pixel at the pixel coordinates in the sampling area according to the pixel value of the target pixel corresponding to the pixel coordinates in the original sampling image includes: When there are multiple target pixels corresponding to the pixel coordinates, a weighted average is performed on the pixel values ​​of the multiple target pixels in the original sampling image to obtain the pixel value of the sampling pixel at the pixel coordinates in the sampling area.

11. The three-dimensional scene reconstruction method according to claim 9, characterized in that: The pixel coordinate calculation relationship is: i=(UU min )*w / (U max -U min ), j=(VV min )*h / (V max -V min ), where i represents the horizontal coordinate of the sampled pixel, j represents the vertical coordinate of the sampled pixel, and U max Indicates the maximum value of the horizontal coordinate of the sampling area, U min Indicates the minimum value of the horizontal coordinate of the sampling area, V max Indicates the maximum value of the vertical coordinate of the sampling area, V min Indicates the minimum value of the horizontal coordinate of the sampling area, U indicates the horizontal coordinate of the image pixel corresponding to the pixel, and V indicates the vertical coordinate of the image pixel corresponding to the pixel.

12. The three-dimensional scene reconstruction method according to claim 5, characterized in that: The calculating, based on the brightness map obtained based on the current frame image, the brightness level value of the sampling area corresponding to the surface voxel includes: Obtaining brightness level values ​​of respective pixels corresponding to the surface voxels according to a brightness map obtained based on the current frame image; The brightness level values ​​of the pixels corresponding to the surface voxels are averaged to obtain the brightness level value of the surface voxels.

13. The three-dimensional scene reconstruction method according to claim 12, characterized in that: Before calculating the brightness level value of the sampling area corresponding to the surface voxel according to the brightness map obtained based on the current frame image, the method further includes: A brightness image corresponding to the current frame image is obtained according to the color image of the current frame image.

14. The three-dimensional scene reconstruction method according to claim 13, characterized in that: The step of obtaining a brightness map corresponding to the current frame image according to the color image of the current frame image includes: Converting the color image of the current frame image into a grayscale image; For each pixel in the grayscale image, determining a brightness level value of the pixel according to the grayscale value of the pixel and a preset number of brightness levels; According to the brightness level value of each pixel, a brightness map corresponding to the current frame image is obtained.

15. The three-dimensional scene reconstruction method according to claim 14, characterized in that: Determining the brightness level value of the pixel according to the grayscale value of the pixel and the number of preset brightness levels includes: The brightness level value of the pixel is calculated based on the grayscale value of the pixel, the number of preset brightness levels and the brightness level calculation formula; wherein: The brightness level calculation formula is: h=A*n / 255, where h represents the brightness level value of the pixel, A represents the grayscale value of the pixel, and n represents the number of preset brightness levels.

16. The three-dimensional scene reconstruction method according to claim 5, characterized in that: The updating of the texture image stored in the surface voxel using the current frame image according to the magnitude relationship between the brightness level value and the preset brightness level value to obtain an updated texture image corresponding to the surface voxel includes: When the brightness level value is greater than the preset brightness level value, directly using the existing texture image stored in the surface voxel as the updated texture image corresponding to the surface voxel; When the brightness level value is less than the preset brightness level value, updating the existing texture image stored in the surface voxel to the current texture image of the current frame image for the surface voxel, to obtain an updated texture image corresponding to the surface voxel; When the brightness level value is equal to the preset brightness level value, the current frame image is fused with the current texture image of the surface voxel and the stored texture image stored in the surface voxel, and the fused texture image is used as the updated texture image corresponding to the surface voxel.

17. The three-dimensional scene reconstruction method according to claim 16, characterized in that: The fusing of the current texture image of the surface voxel with the stored texture image stored in the surface voxel in the current frame image includes: Calculating the angle between the viewpoint direction of the current frame image and the surface normal of the surface voxel; Calculating a camera motion distance between the current frame image and the previous frame image according to a current camera pose corresponding to the current frame image and a camera pose corresponding to the previous frame image; Normalizing the camera motion distance to obtain a normalized camera motion distance; Based on the normalized movement distance, the angle, the current texture image of the current frame image for the surface voxel and the stored texture image stored in the surface voxel, a fusion calculation is performed according to a texture image fusion relationship.

18. The three-dimensional scene reconstruction method according to claim 17, characterized in that: The texture image fusion relationship is: I_new(i, j) = (1-|cos(a)|*cos(s))*I(i, j) + cos(a)|*cos(s)I_input(i, j), where (i, j) is the pixel coordinate, I_new(i, j) represents the pixel value of the fused pixel (i, j), I(i, j) represents the pixel value of the pixel (i, j) in the stored texture image stored in the surface voxel, I_input(i, j) represents the pixel value of the pixel (i, j) in the current texture image of the current frame image for the surface voxel, a represents the angle between the viewpoint direction of the current frame image and the surface normal of the surface voxel, and S represents the normalized camera motion distance.

19. The three-dimensional scene reconstruction method according to claim 3, characterized in that: The calculating, based on the current camera pose and the image pixel position of each pixel in the current frame image, the three-dimensional coordinates corresponding to each pixel in the current frame image includes: For each pixel in the current frame image, the three-dimensional coordinates of the pixel are calculated using a three-dimensional coordinate calculation formula based on the current camera pose and the image pixel position of the pixel in the current frame image; wherein the three-dimensional coordinate calculation formula is: p=P*I rgb (u, v), where p represents the three-dimensional coordinates of the pixel, P represents the current camera pose, and I rgb (u, v) represents the pixel at the image pixel position (u, v).

20. A three-dimensional scene reconstruction device, characterized in that: include: A first updating module is configured to update the current three-dimensional geometric model based on the current frame image to obtain an updated three-dimensional geometric model; A second updating module is configured to update an existing texture image stored in a surface voxel in the updated three-dimensional geometric model using a brightness image obtained based on the current frame image to obtain an updated texture image; A judging module, configured to judge whether the current frame image is the last frame image, and if so, trigger the determining module; otherwise, trigger the returning module; The determining module is configured to use the updated three-dimensional geometric model as a reconstructed three-dimensional geometric model, use the updated texture images corresponding to the surface voxels as reconstructed texture images, and obtain a reconstructed three-dimensional scene based on the reconstructed three-dimensional geometric model and the reconstructed texture images; The return module is used to take the next frame image as the current frame image and return to execute the step of updating the current three-dimensional geometric model based on the current frame image.

21. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the three-dimensional scene reconstruction method according to any one of claims 1 to 19 when executing the computer program.

22. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the three-dimensional scene reconstruction method according to any one of claims 1 to 19.

Images