3D Reconstruction Method, Device, Electronic Device, and Readable Storage Medium

Abstract

The present application discloses a three-dimensional reconstruction method, apparatus, electronic device, and readable storage medium, belonging to the technical field of image data processing. The three-dimensional reconstruction method provided by the present application includes: matching texture information for multiple patches in a three-dimensional mesh model, and determining a first virtual point position in the case where the first patch in the three-dimensional mesh model fails to match the corresponding texture information; rendering a first panoramic view of the three-dimensional mesh model based on the first virtual point position, and determining a first area in the first panoramic view, where the first area corresponds to the first patch; determining target texture information according to the first panoramic view and the first area, and assigning the target texture information to the first patch.

Description

Technical Field

[0001] This application belongs to the technical field of image data processing, and particularly relates to a three-dimensional reconstruction method, a three-dimensional reconstruction device, an electronic device, and a readable storage medium. Background Art

[0002] In the prior art, creating a three-dimensional mesh model of a target scene is the most intuitive way to display the scene. During the process of collecting data for the scene, due to the limitations of the image acquisition device (discrete, finite, and fixed acquisition points), problems such as occlusion and self-occlusion, as well as the limitation of the number of finite acquisition points, often lead to incomplete image data being collected, making it impossible to construct a complete three-dimensional mesh model. Summary of the Invention

[0003] The purpose of the embodiments of this application is to provide a three-dimensional reconstruction method, a three-dimensional reconstruction device, an electronic device, and a readable storage medium. When the collected image data (texture information) is incomplete, by constructing virtual points, estimating the missing texture information, and filling the estimated texture information into the three-dimensional mesh model, the texture information can be effectively supplemented to ensure the integrity of model construction.

[0004] In a first aspect, the embodiments of this application provide a three-dimensional reconstruction method, which includes: matching texture information for multiple patches in a three-dimensional mesh model; when the first patch in the three-dimensional mesh model fails to match the corresponding texture information, determining a first virtual point; rendering a first panoramic view of the three-dimensional mesh model based on the first virtual point, and determining a first area in the first panoramic view, where the first area corresponds to the first patch; determining target texture information according to the first panoramic view and the first area, and assigning the target texture information to the first patch.

[0005] In a second aspect, the embodiments of this application provide a three-dimensional reconstruction device, which includes: a processing unit for matching texture information for multiple patches in a three-dimensional mesh model, and determining a first virtual point when the first patch in the three-dimensional mesh model fails to match the corresponding texture information; the processing unit is further configured to render a first panoramic view of the three-dimensional mesh model based on the first virtual point, and determine a first area in the first panoramic view, where the first area corresponds to the first patch; the processing unit is further configured to determine target texture information according to the first panoramic view and the first area, and assign the target texture information to the first patch.

[0006] In a third aspect, the embodiments of this application provide an electronic device, including a memory, a processor, and a program or instruction stored on the memory and executable on the processor. When the program or instruction is executed by the processor, the steps of the three-dimensional reconstruction method as described in the first aspect are implemented.

[0007] Fourthly, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the 3D reconstruction method as described in the first aspect are implemented.

[0008] Fifthly, an embodiment of the present application provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the steps of the 3D reconstruction method as described in the first aspect.

[0009] Sixthly, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the 3D reconstruction method as described in the first aspect.

[0010] In the embodiment of the present application, during the reconstruction process of the 3D mesh model of the target scene, for the first patch lacking texture information in the model, a first virtual point is constructed. According to the first panoramic view obtained at the first virtual point and the first region corresponding to the patch lacking texture information in the first panoramic view, the target texture information of the first patch is re-estimated, and the estimated target texture information is filled into the 3D mesh model, thereby complementing the missing texture information in the 3D mesh model, effectively ensuring the integrity of the 3D mesh model, and improving the display effect of the 3D mesh model of the target scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 Fig. 1 shows one of the schematic flowcharts of the 3D reconstruction method provided by the embodiment of the present application;

[0012] Figure 2 Fig. 2 shows another schematic flowchart of the 3D reconstruction method provided by the embodiment of the present application;

[0013] Figure 3 Fig. 3 shows one of the schematic diagrams of the 3D reconstruction method provided by the embodiment of the present application;

[0014] Figure 4 Fig. 4 shows another schematic diagram of the 3D reconstruction method provided by the embodiment of the present application;

[0015] Figure 5 Fig. 5 shows a third schematic diagram of the 3D reconstruction method provided by the embodiment of the present application;

[0016] Figure 6 Fig. 6 shows a fourth schematic diagram of the 3D reconstruction method provided by the embodiment of the present application;

[0017] Figure 7 Fig. 7 shows a fifth schematic diagram of the 3D reconstruction method provided by the embodiment of the present application;

[0018] Figure 8Shows the third flowchart of the 3D reconstruction method provided by the embodiments of the present application;

[0019] Figure 9 Shows the structural block diagram of the 3D reconstruction device provided by the embodiments of the present application;

[0020] Figure 10 Shows the structural block diagram of the electronic device provided by the embodiments of the present application;

[0021] Figure 11 Shows the hardware structure diagram of an electronic device provided by the embodiments of the present application. Detailed implementation manners

[0022] Next, the technical solutions in the embodiments of the present application will be clearly described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art belong to the scope of protection of the present application.

[0023] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are usually of the same category, and the number of objects is not limited. For example, the first object can be one or multiple. In addition, "and / or" in the specification and claims means at least one of the connected objects, and the character " / " generally represents an "or" relationship between the associated objects before and after.

[0024] For the convenience of understanding, some technical terms related to the present application are briefly introduced below:

[0025] 3D mesh model: A 3D mesh model is a polygonal representation of an object, usually displayed using a computer or other video device. The displayed object can be an entity in the real world or a fictional object. Anything that exists in the physical nature can be represented by a 3D model. In the embodiments of the present application, the 3D model of the object is used to indicate the 3D structure and size information of the target scene (such as a house). There are various data storage forms of the 3D model, such as in the form of 3D point clouds, meshes, or voxels, etc., and specific details are not limited here.

[0026] Patch: A patch refers to the smallest planar building unit in a 3D mesh model. Usually in rendering, the model in space needs to be divided into countless tiny planes. These planes are also called patches, and they can be any polygon, commonly triangles and quadrilaterals. The intersection points of the sides of these patches are the vertices of each patch. Patches can be randomly divided according to information such as the material or color of the model.

[0027] Panorama: In a broad sense, a panorama refers to a wide-angle image, that is, an image with a large viewing angle. Panoramas can be achieved through different projection methods, common ones including: equiangular projection, equirectangular projection, orthographic projection, and equal-area projection, etc., which are not specifically limited here.

[0028] The following combines the attached Figures 1 to 11 , and through specific embodiments and their application scenarios, the 3D reconstruction method, 3D reconstruction device, electronic device, and readable storage medium provided by the embodiments of the present application are described in detail.

[0029] In the embodiments of the present application, a 3D reconstruction method is provided. Figure 1 FIG. shows one of the schematic flowcharts of the 3D reconstruction method provided by the embodiments of the present application. As Figure 1 shown, the 3D reconstruction method includes:

[0030] Step 102: Match texture information for multiple patches in the 3D mesh model. In the case where the first patch in the 3D mesh model fails to match the corresponding texture information, determine the first virtual point position.

[0031] Specifically, there are multiple patches in the 3D mesh model. After the 3D mesh model is constructed, texture mapping needs to be performed on the 3D mesh model. During the texture mapping process, texture information needs to be matched for each patch in the 3D mesh model.

[0032] Specifically, a texture estimation algorithm can be used to match texture information for multiple patches in the 3D mesh model.

[0033] Further, the first patch among the multiple patches can be a patch in the 3D mesh model that lacks texture information. Usually, the number of the above first patches is multiple. The lack of texture information for multiple first patches may cause the 3D mesh model after texture mapping to be incompletely expressed. Therefore, it is necessary to fill and assign values to the first patches lacking texture information to make the 3D mesh model more complete.

[0034] Further, the first patch can be a 3D point set P(x’, y’, z’), and the points in the set cannot correctly estimate the texture information.

[0035] Further, since the first patch lacks texture information, a virtual point can be determined for each first patch based on the spatial parameter information of each first patch. The above-mentioned first virtual point is the viewpoint in the three-dimensional mesh model. Through the coordinate position of the first virtual point, most of the patches in the three-dimensional mesh model can be observed well, including the first patch corresponding to the first virtual point that lacks texture information.

[0036] It can be understood that if there is a virtual point among the above virtual points that can observe all the first patches lacking texture information, then this virtual point can be determined as the first virtual point, that is, this one first virtual point can observe all the patches lacking texture. If there is no virtual point among the above virtual points that can observe all the first patches lacking texture, then multiple first virtual points need to be determined to ensure that all the first patches lacking texture can be observed.

[0037] It should be noted that whether the first patch lacking texture information corresponds to one or multiple first virtual points only affects the number of times of obtaining the first panoramic image and the number of times of estimating the lacking texture information in the subsequent process. In the embodiments of the present application, only the case where multiple first patches lacking texture correspond to one first virtual point is described, and the method of estimating the lacking texture information through multiple first virtual points will not be elaborated.

[0038] Step 104: Render the first panoramic image of the three-dimensional mesh model based on the first virtual point, and determine the first area in the first panoramic image;

[0039] Among them, the first area corresponds to the first patch;

[0040] It should be noted that the first panoramic image includes a 360-degree panoramic image corresponding to the three-dimensional mesh model. The panoramic image can be divided into multiple areas, and there is a corresponding relationship between the multiple areas in the first panoramic image and the multiple patches in the three-dimensional mesh model. That is, there is a corresponding relationship between the first area in the first panoramic image and the first patch in the three-dimensional mesh model.

[0041] Further, the size of the above first panoramic image is limited, specifically including: the width of the first panoramic image, that is, the latitude, ranges from [0, 2×Pai], and the height of the first panoramic image, that is, the longitude, ranges from [-Pai / 2, Pai / 2], where Pai = 3.1415926.

[0042] Further, the formula for calculating the first area in the above first panoramic image is:

[0043] P Y = arcsin(P_z’ / R);

[0044] P x= arcsin(P_x’ / R×cos(P Y ));

[0045] Wherein, R is the radius when the first panoramic view is drawn, (P_x’, P_y’, P_z’) are the coordinate values of each point of the first patch, and (Px, Py) are the coordinate values of a point within the first region.

[0046] It can be understood that obtaining the first panoramic view of the above three-dimensional mesh model based on the first virtual point position can more accurately reflect the position information of the first patch, and can also determine the hole situation caused by the first patch lacking texture information in the target scene image.

[0047] Step 106: Determine the target texture information according to the first panoramic view and the first region, and assign the target texture information to the first patch.

[0048] Specifically, the above target texture information includes depth data and color data.

[0049] It can be understood that through the above first panoramic view rendered by the first virtual point position, and the first region determined from multiple regions in the first panoramic view, through parsing and estimation, the target texture information can be determined, and the target texture information is used to assign values to the first patch in the three-dimensional mesh model to fill the hole area caused by the lack of texture information, so that the three-dimensional mesh model is more complete and beautiful in display. Exemplarily, as Figure 6 shown, in the three-dimensional mesh model of the target scene, some areas (black areas) of the "bed" in the image lack texture information. The patch corresponding to the area lacking texture information is set as the first patch (target patch). Based on the first patch, a first virtual point position is constructed. At the coordinate position of the first virtual point position, the three-dimensional mesh model of the target scene is re-rendered to obtain a 360-degree first panoramic view, and the first panoramic view includes Figure 6 the "bed" in. The black area in this first panoramic view is the first region corresponding to the first patch.

[0050] Further exemplarily, in the re-rendered first panoramic view, the first region (black area) corresponding to the first patch is determined. According to the above information such as the first panoramic view and the first region, the target texture information of the first patch is estimated, and the estimated target texture information is re-filled into the first patch of the three-dimensional mesh model of the target scene. The target texture information is filled into the Figure 6 black hole area in. The filling effect is as Figure 6 shown. By filling the target texture information into the area lacking texture information, the three-dimensional mesh model is more complete and beautiful in display.

[0051] The 3D reconstruction method provided by the embodiments of the present application constructs a first virtual point for the first patch lacking texture information, re-renders the first panoramic view, estimates the corresponding target texture information, and fills it into the first patch of the 3D mesh model. Through the 3D reconstruction method provided by the present application, the texture information missing in the 3D mesh model can be effectively supplemented, ensuring the integrity of the texture information in the 3D mesh model of the target scene.

[0052] In some embodiments of the present application, step 102 of determining the first virtual point specifically includes:

[0053] Step 102a: Determine the first virtual point corresponding to the first patch according to the spatial parameters of the first patch.

[0054] In the embodiments of the present application, the spatial parameters of the first patch inside the 3D mesh model are determined, and the spatial parameters include the three-dimensional coordinate parameters of the first patch in the 3D mesh model.

[0055] Specifically, according to the spatial parameters of the first patch in the 3D mesh model, the spatial coordinate parameters of the first virtual point in the 3D mesh model are calculated.

[0056] Furthermore, the three-dimensional coordinate value of the center point of the first patch in the 3D mesh model can be determined first, and according to the three-dimensional coordinate value of this center point, the spatial coordinate parameters of the first virtual point corresponding to the first patch are calculated. The 3D reconstruction method provided by the embodiments of the present application determines the virtual point corresponding to the first patch according to the spatial parameters of the first patch, ensuring the accuracy of the positioning of the first virtual point, and further ensuring the display effect of the first panoramic view rendered at the first virtual point.

[0057] In some embodiments of the present application, step 102a specifically may further include:

[0058] Step 1021: Determine the first virtual point according to the normal vector of the first patch;

[0059] Wherein, the included angle between the vector from the center of the first patch to the first virtual point and the normal vector of the first patch is less than 90°.

[0060] In the above embodiments, the normal vector of the first patch can be determined according to the three-dimensional parameters of the first patch in the 3D mesh model, and then according to the normal vector of the first patch, the spatial coordinate parameters of the first virtual point are calculated.

[0061] Furthermore, the included angle between the vector determined by the center point coordinates and the first virtual point coordinates of the first patch and the normal vector of the first patch is less than 90°.

[0062] It can be understood that limiting the angle between the vector determined by the central point coordinates of the first patch and the first virtual point coordinates and the normal vector of the first patch can enable the first virtual point to render the first panoramic view from a more accurate perspective.

[0063] Specifically, the position points of the above-mentioned first virtual points are not unique. The first virtual point can be located on the extension line of the normal vector of the center point of the first patch, or on the extension line of the normal vector of other points on the first patch. As long as the angle between the vector from the center of the first patch to the first virtual point and the normal vector of the first patch is within the range less than 90°, it can be used as the first virtual point.

[0064] Furthermore, the coordinate calculation formula for the first virtual point is as follows:

[0065] V x = D × avg(N_x’) + P_x’;

[0066] V y = D × avg(N_y’) + P_y’;

[0067] V z = D × avg(N_z’) + P_z’;

[0068] Where, (Vx, Vy, V z ) is the coordinate value of the first virtual point, (P_x’, P_y’, P_z’) is the coordinate value of the center point of the first patch, (N_x’, N_y’, N_z’) is the coordinate value of a point on the normal vector of the first patch, and D is the spatial distance between the first virtual point and the center point of the first patch.

[0069] Furthermore, the first virtual point can be the virtual point with the largest number corresponding to the first patch within the above range.

[0070] The 3D reconstruction method provided by the embodiments of the present application finds the first patch with missing texture in the 3D mesh model of the target scene, determines the normal vector of the target patch, calculates the coordinate value of the first virtual point corresponding to the target patch according to the normal vector of the target patch, and then determines the virtual point corresponding to the first patch to ensure the accuracy of the spatial parameters of the virtual point.

[0071] In some embodiments of the present application, Figure 2 shows the second flow diagram of the 3D reconstruction method provided by the embodiments of the present application. As Figure 2 shown, the above 3D reconstruction method specifically includes:

[0072] Step 202: Match texture information for multiple patches in the 3D mesh model. In the case where the first patch in the 3D mesh model does not match the corresponding texture information, determine the first virtual point;

[0073] Step 204: Render the first panoramic view of the three-dimensional mesh model based on the first virtual point position, and determine the first area in the first panoramic view, where the first area corresponds to the first patch;

[0074] Step 206: Render the first area into the target color, and generate a mask image in the area corresponding to the target color;

[0075] Step 208: Determine the target texture information according to the rendering result of the first panoramic view and the mask image;

[0076] Step 210: Assign the target texture information to the first patch.

[0077] Specifically, determine the first area in the first panoramic view, where the first area corresponds to the first patch, and the above-mentioned first area may correspond to multiple first patches.

[0078] Further, render the color of the first area into the target color, such as pure black, pure green or other solid colors.

[0079] Further, the above mask image may define the contour shape information of the target texture information.

[0080] Exemplarily, as Figure 3 shown, the three-dimensional reconstruction method provided by the present application can render the first area lacking texture information black in the 360-degree first panoramic view. According to Figure 3 the black area corresponding to the "door" in Figure 3 generate the corresponding mask image. The mask image is as shown in Figure 4 . The mask image can determine the edge contour of the target texture information. After estimating the target texture information, fill the target texture information into the first patch corresponding to the black area in Figure 3 to complete the missing texture of the "door". The effect is as shown in Figure 5 so as to make the content of the three-dimensional mesh model more complete.

[0081] The three-dimensional reconstruction method provided by the embodiments of the present application renders the color of the first area into the target color and generates a mask image in the area rendered into the target color, which is convenient for estimating the texture information of the texture missing area and ensures the accuracy of the target texture information.

[0082] In some embodiments of the present application, step 204 in the three-dimensional reconstruction method provided by the embodiments of the present application may specifically include:

[0083] Step 204a: Input the parameter information of the rendering result of the first panoramic view and the mask image into the first prediction model to output the target texture information through the first prediction model.

[0084] In the above embodiment, parameter information such as the rendering result of the first panoramic image and the mask image is input into the first prediction model. After the operation of the first prediction model, target texture information is obtained.

[0085] Furthermore, the mask image includes contour shape information, and the edge contour of the target texture information can be determined.

[0086] Specifically, the first prediction model can be a deep learning model. Exemplarily, the deep learning model can be a model with an encoder-decoder architecture. The rendering result of the panoramic image and the mask image are input into the above first prediction model. After the operation of the first prediction model, texture information is output, and this texture information can be the target texture information suitable for assigning values to the first patch.

[0087] Exemplarily, according to Figure 6 the above, a corresponding mask image is generated. The mask image can determine the edge contour of the target texture information. The first panoramic image and the mask image are used as input parameters and input into the first prediction model. The first prediction model outputs the target texture information after operation. The contour information of the target texture information conforms to Figure 6 the black area in. The target texture information is filled onto the first patch corresponding to the black area, so that the missing texture at the edge of the "bed" is completed, and the effect is as Figure 7 shown, thereby making the content of the three-dimensional mesh model more complete.

[0088] The 3D reconstruction method provided by the embodiments of the present application determines the target texture information by using a prediction model, ensures the accuracy of the estimated target texture information, and further ensures the display integrity of the 3D mesh model, enriching the information parameters of the 3D mesh model.

[0089] In some embodiments of the present application, before step 102 in the 3D reconstruction method provided by the present application, the 3D reconstruction method further includes:

[0090] Step 101: Obtain multiple depth maps of the shooting scene and multiple color maps corresponding to the multiple depth maps, and construct a 3D mesh model according to the multiple depth maps and the multiple color maps.

[0091] In the embodiments of the present application, multiple depth maps of the shooting scene and multiple color maps corresponding to the multiple depth maps are obtained, and a three-dimensional mesh model is constructed based on the obtained multiple depth maps and multiple color maps. In a possible implementation manner, constructing a three-dimensional mesh model of the target scene requires multiple color two-dimensional images and depth images of the target scene. After obtaining the depth maps and color maps taken inside the target scene, the point estimation of the RGBD multi-point data is performed on the multiple depth maps and multiple color maps, and a three-dimensional point cloud of the target scene is generated based on the obtained points. Then, a mesh of the target scene is generated based on the three-dimensional point cloud data, and the texture information is estimated. Finally, a three-dimensional mesh model of the target scene is constructed.

[0092] Construct a three-dimensional mesh model of the target scene based on these depth maps and color two-dimensional maps.

[0093] The three-dimensional reconstruction method provided by the embodiments of the present application constructs a three-dimensional mesh model based on multiple depth maps and multiple color maps, ensuring the accuracy and integrity of the three-dimensional mesh model.

[0094] In some embodiments of the present application, Figure 8 FIG. 3 shows a schematic flow chart of the three-dimensional reconstruction method provided by the embodiments of the present application. As Figure 8 shown, the above three-dimensional reconstruction method specifically includes:

[0095] Step 802: Obtain multiple patches in the three-dimensional mesh model;

[0096] Step 804: Determine the texture information corresponding to any one of the multiple patches through a texture estimation algorithm;

[0097] Step 806: Render a first panoramic view of the three-dimensional mesh model based on the first virtual point position, and determine a first area in the first panoramic view, where the first area corresponds to the first patch;

[0098] Step 808: Determine the target texture information according to the first panoramic view and the first area, and assign the target texture information to the first patch.

[0099] In the embodiments of the present application, the three-dimensional mesh model includes multiple patches inside, and through the texture estimation algorithm, the texture information of each patch can be calculated.

[0100] In a possible implementation manner, the texture estimation algorithm may be a Markov random number algorithm. The Markov random number algorithm needs to set a data item and a smoothing item. The calculation formula of the data item is:

[0101] Data(F,v)=S / |dt-d|;

[0102] Where, v is the first patch, S is the area of the projection of the first patch onto the two-dimensional view, dt is the depth parameter of the projection of the first patch onto the two-dimensional view, and d is the depth parameter of the first patch in the corresponding depth map.

[0103] The calculation formula for the smoothness term is:

[0104]

[0105] Where, I k is the two-dimensional semantics of the first patch, and I p is the two-dimensional semantics of the adjacent patch of the first patch. If the semantics of two adjacent patches are the same, the smoothness term is set to 1; otherwise, the smoothness term is set to 0.

[0106] The calculation formula for the Markov random field is:

[0107]

[0108] Based on the data term and the smoothness term, the texture information E(v) of the first patch is calculated.

[0109] The 3D reconstruction method provided by the embodiments of the present application calculates the texture information of each patch in the 3D mesh model through a texture estimation algorithm, and then can identify the first patch lacking texture information and fill the texture information of the first patch to ensure the integrity of the display of the 3D mesh model.

[0110] In some embodiments of the present application, a 3D reconstruction device is provided. Figure 9 The structural block diagram of the 3D reconstruction device provided by the embodiments of the present application is shown. As Figure 9 shown, the 3D reconstruction device 900 includes:

[0111] A processing unit 902, configured to match texture information for multiple patches in the 3D mesh model, and determine a first virtual point position when the first patch in the 3D mesh model fails to match the corresponding texture information;

[0112] The processing unit 902 is further configured to render a first panoramic view of the 3D mesh model based on the first virtual point position and determine a first area in the first panoramic view, where the first area corresponds to the first patch;

[0113] The processing unit 902 is further configured to determine target texture information according to the first panoramic view and the first area, and assign the target texture information to the first patch.

[0114] Specifically, during the process of constructing a three-dimensional grid model of the target scene, the interior of the three-dimensional grid model includes multiple patches. After the processing unit 902 confirms all the patches inside the three-dimensional grid model, it confirms the texture information of each patch, sets the patches lacking texture information as the first patches, and constructs the first virtual points based on the coordinate parameters of the first patches; the processing unit 902 re-renders the first panoramic view of the three-dimensional grid model at the position of the first virtual points, and determines the first area corresponding to the first patches in the first panoramic view; the processing unit 902 estimates the target texture information of the first patches according to the first panoramic view and the first area, and re-assigns the target texture information to the first patches.

[0115] The three-dimensional reconstruction device provided by the embodiments of the present application constructs virtual points for the patches lacking texture information through the processing unit, re-renders the panoramic view, and estimates the corresponding texture information to fill it into the three-dimensional grid model. It effectively supplements the lacking texture information in the three-dimensional grid model and ensures the integrity of the texture information in the three-dimensional grid model of the target scene.

[0116] In some embodiments of the present application, the processing unit 902 is further configured to determine the first virtual points corresponding to the first patches according to the spatial parameters of the first patches.

[0117] The three-dimensional reconstruction device provided by the embodiments of the present application determines the first virtual points corresponding to the first patches according to the spatial parameters of the first patches through the processing unit, ensuring the accuracy of the virtual points.

[0118] In some embodiments of the present application, the processing unit 902 is further configured to determine the first virtual points according to the normal vector of the first patches; wherein, the included angle between the vector from the center of the first patch to the first virtual point and the normal vector of the first patch is less than 90°.

[0119] The three-dimensional reconstruction device provided by the embodiments of the present application determines the virtual points corresponding to the first patches according to the normal vector of the first patches through the processing unit, ensuring the accuracy of the spatial parameters of the virtual points, and further ensuring the display effect of the first panoramic view rendered at the first virtual points.

[0120] In some embodiments of the present application, the processing unit 902 is further configured to render the first area into a target color and generate a mask image for the area corresponding to the target color; the processing unit 902 is further configured to determine the target texture information according to the rendering result of the first panoramic view and the mask image.

[0121] The three-dimensional reconstruction device provided by the embodiments of the present application renders the color of the first area into a target color through the processing unit, generates a mask image, and estimates the texture information of the texture missing area, ensuring the accuracy of the target texture information.

[0122] In some embodiments of the present application, the processing unit 902 is further configured to input the rendering result of the first panoramic image and the parameter information of the mask image into the first prediction model, so as to output the target texture information through the first prediction model.

[0123] The 3D reconstruction device provided by the embodiments of the present application uses a prediction model through the processing unit to determine texture information, ensuring the accuracy of the estimated target texture information, thereby ensuring the display integrity of the 3D mesh model and enriching the information parameters of the 3D mesh model.

[0124] In some embodiments of the present application, the processing unit 902 is further configured to obtain multiple depth maps of the shooting scene and multiple color maps corresponding to the multiple depth maps, and construct a 3D mesh model according to the multiple depth maps and the multiple color maps.

[0125] The 3D reconstruction device provided by the embodiments of the present application constructs a 3D mesh model according to multiple depth maps and multiple color maps through the processing unit, ensuring the accuracy and integrity of the 3D mesh model.

[0126] In some embodiments of the present application, the 3D reconstruction device 900 includes an acquisition unit configured to acquire multiple patches in the 3D mesh model;

[0127] The processing unit 902 is further configured to determine the texture information corresponding to any one of the multiple patches through a texture estimation algorithm.

[0128] The 3D reconstruction device provided by the embodiments of the present application uses a texture estimation algorithm through the processing unit to calculate the texture information of each patch in the 3D mesh model and identify the target patch lacking texture information.

[0129] The 3D reconstruction device in the embodiments of the present application can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices other than terminals. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a handheld computer, a vehicle-mounted electronic device, a Mobile Internet Device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. It can also be a server, a Network Attached Storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, etc. The embodiments of the present application do not make specific limitations.

[0130] The 3D reconstruction device in the embodiments of the present application can be a device with an operating system. The operating system can be the Android operating system, the iOS operating system, or other possible operating systems. The embodiments of the present application do not make specific limitations.

[0131] The 3D reconstruction device provided in the embodiments of the present application can implement each process implemented in the above method embodiments. To avoid repetition, it will not be elaborated here.

[0132] Optionally, the embodiments of the present application further provide an electronic device. Figure 10 The structural block diagram of the electronic device according to the embodiments of the present application is shown. As Figure 10 shown, the electronic device 1000 includes a processor 1002 and a memory 1004. A program or instruction that can run on the processor 1002 is stored on the memory 1004. When the program or instruction is executed by the processor 1002, each step of the above method embodiments is implemented, and the same technical effects can be achieved. To avoid repetition, it will not be elaborated here.

[0133] It should be noted that the electronic device in the embodiments of the present application includes the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0134] Figure 11 It is a schematic diagram of the hardware structure of an electronic device for implementing the embodiments of the present application.

[0135] The electronic device 1100 includes, but is not limited to, components such as a radio frequency unit 1101, a network module 1102, an audio output unit 1103, an input unit 1104, a sensor 1105, a display unit 1106, a user input unit 1107, an interface unit 1108, a memory 1109, and a processor 1110.

[0136] Those skilled in the art can understand that the electronic device 1100 may further include a power source (such as a battery) for supplying power to each component. The power source can be logically connected to the processor 1110 through a power management system, so as to implement functions such as management of charging, discharging, and power consumption management through the power management system. Figure 11 The structure of the electronic device shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0137] Among them, the processor 1110 is used to match texture information for multiple patches in the three-dimensional mesh model. In the case where the first patch in the three-dimensional mesh model does not match the corresponding texture information, a first virtual point position is determined.

[0138] The processor 1110 is used to render a first panoramic view of the three-dimensional mesh model based on the first virtual point position and determine a first area in the first panoramic view, where the first area corresponds to the first patch.

[0139] The processor 1110 is used to determine target texture information according to the first panoramic view and the first area, and assign the target texture information to the first patch.

[0140] The electronic device provided by the embodiment of the present application constructs a virtual point position for the patch lacking texture information, re-renders the panoramic view, and estimates the corresponding texture information to fill it into the three-dimensional mesh model. It effectively supplements the missing texture information in the three-dimensional mesh model and ensures the integrity of the texture information in the three-dimensional mesh model of the target scene.

[0141] Further, the processor 1110 is used to determine the first virtual point position corresponding to the first patch according to the spatial parameters of the first patch.

[0142] The electronic device provided by the embodiment of the present application determines the virtual point position corresponding to the first patch according to the spatial parameters of the first patch, ensuring the accuracy of the virtual point position.

[0143] Further, the processor 1110 is used to determine the first virtual point position according to the normal vector of the first patch.

[0144] Among them, the included angle between the vector from the center of the first patch to the first virtual point position and the normal vector of the first patch is less than 90°.

[0145] The electronic device provided by the embodiment of the present application determines the virtual point corresponding to the first patch according to the normal vector of the first patch, ensuring the accuracy of the spatial parameters of the virtual point.

[0146] Further, the processor 1110 is configured to render the first region into a target color, generate a mask image for the region corresponding to the target color, and determine the target texture information according to the rendering result of the first panorama and the mask image.

[0147] The electronic device provided by the embodiment of the present application renders the color of the first region into a target color, generates a mask image, and estimates the texture information of the texture missing region.

[0148] Further, the processor 1110 is configured to input the parameter information of the rendering result of the first panorama and the mask image into the first prediction model, so as to output the target texture information through the first prediction model.

[0149] The electronic device provided by the embodiment of the present application uses a prediction model to determine texture information, ensuring the accuracy of the estimated texture image.

[0150] Further, the processor 1110 is configured to obtain multiple depth maps of the shooting scene and multiple color maps corresponding to the multiple depth maps, and construct a three-dimensional mesh model according to the multiple depth maps and the multiple color maps.

[0151] The electronic device provided by the embodiment of the present application constructs a three-dimensional mesh model according to multiple depth maps and multiple color maps, ensuring the accuracy and integrity of the three-dimensional mesh model.

[0152] Further, the processor 1110 is configured to determine the third patch corresponding to the boundary of the invisible region in the three-dimensional mesh model, fill the texture color of the fourth patch into the third patch, where the fourth patch is an adjacent patch to the third patch and located outside the boundary of the invisible region; configured to obtain multiple patches in the three-dimensional mesh model; configured to determine the texture information corresponding to any one of the multiple patches through a texture estimation algorithm.

[0153] The electronic device provided by the embodiment of the present application uses a texture estimation algorithm to calculate the texture information of each patch in the three-dimensional mesh model and identify the target patch lacking texture information.

[0154] It should be understood that in the embodiments of the present application, the input unit 1104 may include a Graphics Processing Unit (GPU) 11041 and a microphone 11042. The graphics processor 11041 processes the image data of still pictures or videos obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 1106 may include a display panel 11061, and the display panel 11061 may be configured in the form of, for example, a liquid crystal display, an organic light emitting diode, etc. The user input unit 1107 includes at least one of a touch panel 11071 and other input devices 11072. The touch panel 11071 is also referred to as a touch screen. The touch panel 11071 may include two parts: a touch detection device and a touch controller. The other input devices 11072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be elaborated herein.

[0155] The memory 1109 can be used to store software programs and various data. The memory 1109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data. Among them, the first storage area may store an operating system, application programs or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 1109 may include a volatile memory or a non-volatile memory, or the memory 1109 may include both a volatile memory and a non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDR SDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synch link dynamic random access memory (SLDRAM), and a direct rambus random access memory (DRRAM). The memory 1109 in the embodiments of the present application includes, but is not limited to, these and any other suitable types of memories.

[0156] The processor 1110 may include one or more processing units; optionally, the processor 1110 integrates an application processor and a modem processor. Among them, the application processor mainly processes operations related to the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above modem processor may not be integrated into the processor 1110 either.

[0157] The embodiments of the present application also provide a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, it implements each process of the above-mentioned three-dimensional reconstruction method embodiments and can achieve the same technical effects. To avoid repetition, it will not be elaborated here.

[0158] Among them, the processor is the processor in the electronic device in the above-mentioned embodiment. The readable storage medium includes computer-readable storage media, such as computer read-only memory ROM, random access memory RAM, magnetic disks, or optical discs, etc.

[0159] Another embodiment of the present application provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement each process of the above-mentioned three-dimensional reconstruction method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0160] It should be understood that the chip mentioned in the embodiments of the present application can also be referred to as a system-on-chip, system chip, chip system, or system-on-chip, etc.

[0161] The embodiments of the present application provide a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement each process of the above-mentioned three-dimensional reconstruction method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0162] It should be noted that in this article, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of another identical element in the process, method, article or device including the element. In addition, it should be pointed out that the methods and devices in the embodiments of the present application are not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in a reverse order according to the functions involved. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0163] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc), and includes several instructions to enable a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of the present application.

[0164] The embodiments of the present application have been described above in conjunction with the accompanying drawings. However, the present application is not limited to the above specific implementation manners. The above specific implementation manners are merely illustrative rather than restrictive. Under the inspiration of the present application, those of ordinary skill in the art can also make many forms without departing from the purpose of the present application and the scope protected by the claims, and all of them fall within the protection scope of the present application.

Claims

1. A three-dimensional reconstruction method, characterized in that, The three-dimensional reconstruction method includes: Matching texture information for multiple patches in the three-dimensional mesh model, and determining a first virtual point position in the case where the first patch in the three-dimensional mesh model fails to match the corresponding texture information; Rendering a first panoramic view of the three-dimensional mesh model based on the first virtual point position, and determining a first area in the first panoramic view, where the first area corresponds to the first patch; Determining target texture information according to the first panoramic view and the first area, and assigning the target texture information to the first patch; The step of determining the target texture information according to the first panoramic view and the first area specifically includes: Rendering the first area into a target color, and generating a mask image in the area corresponding to the target color; Determining the target texture information according to the rendering result of the first panoramic view and the mask image; The mask image is used to define the contour shape information of the target texture information.

2. The three-dimensional reconstruction method according to claim 1, wherein The step of determining the first virtual point position specifically includes: Determining the first virtual point position corresponding to the first patch according to the spatial parameters of the first patch.

3. The three-dimensional reconstruction method according to claim 2, characterized in that The spatial parameters include a normal vector. The step of determining the first virtual point position corresponding to the first patch according to the spatial parameters of the first patch specifically includes: Determining the first virtual point position according to the normal vector of the first patch; Wherein, the included angle between the vector from the center of the first patch to the first virtual point position and the normal vector of the first patch is less than 90°.

4. The three-dimensional reconstruction method according to claim 3, wherein The step of determining the target texture information according to the rendering result of the first panoramic view and the mask image specifically includes: Inputting the rendering result of the first panoramic view and the parameter information of the mask image into a first prediction model to output the target texture information through the first prediction model.

5. The three-dimensional reconstruction method according to any one of claims 1 to 4, characterized in that Before matching texture information for multiple patches in the three-dimensional mesh model, the three-dimensional reconstruction method further includes: Obtaining multiple depth maps of the shooting scene and multiple color maps corresponding to the multiple depth maps, and constructing the three-dimensional mesh model according to the multiple depth maps and the multiple color maps.

6. The three-dimensional reconstruction method according to any one of claims 1 to 4, characterized in that The step of matching texture information for multiple patches in the three-dimensional mesh model specifically includes: Obtaining multiple patches in the three-dimensional mesh model; Determining that any one of the multiple patches matches the corresponding texture information through a texture estimation algorithm.

7. A three-dimensional reconstruction device, characterized in that, The three-dimensional reconstruction device includes: A processing unit, configured to match texture information for multiple patches in the three-dimensional mesh model, and determine a first virtual point position in the case where the first patch in the three-dimensional mesh model fails to match the corresponding texture information; The processing unit is further configured to render a first panoramic view of the three-dimensional mesh model based on the first virtual point position, and determine a first area in the first panoramic view, where the first area corresponds to the first patch; The processing unit is further configured to determine target texture information according to the first panoramic view and the first area, and assign the target texture information to the first patch.

8. An electronic device, characterized in that, Including: A memory, on which programs or instructions are stored; A processor, configured to implement the steps of the 3D reconstruction method according to any one of claims 1 to 6 when executing the program or instructions.

9. A readable storage medium having a program or instructions stored thereon, characterized in that, When the program or instructions are executed by the processor, the steps of the 3D reconstruction method according to any one of claims 1 to 6 are implemented.

Images