Three-dimensional grid model acquisition method and device and electronic equipment

Abstract

The invention discloses a three-dimensional grid model obtaining method and device and electronic equipment, and relates to the field of three-dimensional reconstruction, modeling is carried out on an acquired image of a to-be-modeled object through a Gaussian splashing algorithm, after an initial Gaussian point cloud model is obtained, requirements are formed on the basis of initial attribute characteristics of each initial Gaussian ellipsoid and a three-dimensional grid model, and a three-dimensional grid model is obtained. And performing point cloud processing on the initial Gaussian point cloud model to obtain a to-be-determined point cloud model with a three-dimensional gridding characteristic, then performing reconstruction through a Gaussian splash algorithm on the basis to obtain a target Gaussian point cloud model meeting a transparency constraint condition, and rapidly and accurately extracting a three-dimensional mesh patch from the target Gaussian point cloud model. Therefore, the three-dimensional grid model which is complete in structure and accurate in surface feature is formed, further manual modification by technicians is not needed, the obtaining efficiency and quality of the three-dimensional grid model are improved, and the requirement of downstream application for a high-quality three-dimensional model is reliably met.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional reconstruction, and in particular to a method, device and electronic equipment for acquiring a three-dimensional mesh model. Background Art

[0002] In recent years, 3D reconstruction technology has been widely used in many fields such as game development, virtual reality, metaverse, architectural design, film and television production, medical image processing, and cultural heritage protection. Gaussian Splatting (3DGS) technology is usually used to quickly reconstruct high-quality 3D models to meet the downstream application needs of the corresponding fields.

[0003] Among them, since the three-dimensional model reconstructed based on 3DGS technology is a Gaussian point cloud model, this model format cannot be suitable for downstream applications such as model editing, animation production, bone binding and other tasks. It is necessary to extract the three-dimensional mesh model from it to ensure that the reconstructed three-dimensional model can be quickly applied to downstream applications.

[0004] However, in the three-dimensional mesh model extraction method of Gaussian point cloud model based on 3DGS technology, there are still problems such as incomplete extracted three-dimensional mesh structure and too smooth or too rough surface, resulting in poor quality of the obtained three-dimensional mesh model. It requires technicians to spend a lot of time and energy to modify it manually, and it cannot reliably meet the high-quality requirements of downstream applications for three-dimensional models. Summary of the invention

[0005] In view of the above problems, this application provides the following solutions:

[0006] The first aspect of the present application provides a method for acquiring a three-dimensional mesh model, the method comprising:

[0007] Acquire an initial Gaussian point cloud model for the object to be modeled; the initial Gaussian point cloud model is obtained by modeling the collected image of the object to be modeled using a Gaussian splash algorithm;

[0008] Based on the requirements for the composition of the three-dimensional grid model and the initial attribute characteristics of each initial Gaussian ellipsoid constituting the initial Gaussian point cloud model, performing point cloud processing on the initial Gaussian point cloud model to obtain a pending point cloud model;

[0009] Based on the Gaussian splash algorithm and the undetermined point cloud model, the object to be modeled is three-dimensionally reconstructed to obtain a target Gaussian point cloud model that satisfies a transparency constraint condition; the transparency constraint condition indicates that the transparency of target Gaussian ellipsoids in the target Gaussian point cloud model that correspond one-to-one to each surface area of ​​the object to be modeled and are independent of each other tends to a first value;

[0010] A three-dimensional mesh surface is extracted from the target Gaussian point cloud model, and a three-dimensional mesh model of the object to be modeled is constructed using the extracted multiple three-dimensional mesh surface patches.

[0011] In a possible implementation, the step of performing point cloud processing on the initial Gaussian point cloud model based on the three-dimensional grid model composition requirements and the initial attribute characteristics of each initial Gaussian ellipsoid constituting the initial Gaussian point cloud model to obtain a pending point cloud model includes:

[0012] Determine initial attribute features of each initial Gaussian ellipsoid constituting the initial Gaussian point cloud model; the initial attribute features include spatial information, color value and transparency of the corresponding initial Gaussian ellipsoid in three-dimensional space;

[0013] Based on the spatial information corresponding to each of the initial Gaussian ellipsoids, the overlapping areas between different initial Gaussian ellipsoids are separated to obtain a plurality of independent Gaussian ellipsoids to be processed;

[0014] Performing two-dimensional processing on the multiple Gaussian ellipsoids to be processed to obtain multiple Gaussian ellipses; the color value and transparency of the Gaussian ellipses are obtained based on the conversion processing of the color value and transparency of the Gaussian ellipsoids to be processed;

[0015] Based on the respective transparency of the multiple Gaussian ellipses, the multiple Gaussian ellipses are subjected to transparency elimination to obtain a pending point cloud model; wherein the transparency of each of the Gaussian ellipses constituting the pending point cloud model is the first value.

[0016] In a possible implementation, the three-dimensional reconstruction of the object to be modeled based on the Gaussian splash algorithm and the point cloud model to be determined to obtain a target Gaussian point cloud model that satisfies the transparency constraint condition includes:

[0017] Based on the Gaussian splash algorithm and the to-be-determined point cloud model, converting the captured image of the to-be-modeled object into a rendered image under the same camera extrinsic parameters;

[0018] Obtaining a rendering loss between the rendered image and the corresponding acquired image;

[0019] Obtaining transparency loss based on the transparency of each Gaussian ellipse of the to-be-determined point cloud model and the total number of the Gaussian ellipses;

[0020] Performing a weighted summation on the rendering loss and the transparency loss to obtain a target loss;

[0021] By minimizing the target loss, the color value and transparency of each Gaussian ellipse of the undetermined point cloud model are optimized to obtain a target Gaussian point cloud model that satisfies the transparency constraint condition.

[0022] In a possible implementation, in the process of optimizing the color value and transparency of each Gaussian ellipse of the to-be-determined point cloud model:

[0023] The spatial information of each Gaussian ellipse remains unchanged;

[0024] The transparency of the Gaussian ellipse corresponding to each surface area of ​​the object to be modeled is optimized in an increasing direction, and the transparency of the Gaussian ellipse not corresponding to any surface area of ​​the object to be modeled is optimized in a decreasing direction.

[0025] In a possible implementation, obtaining the transparency loss based on the transparency of each Gaussian ellipse of the to-be-determined point cloud model and the total number of the Gaussian ellipses includes:

[0026] Determine the difference between the transparency optimization target value and the transparency of each Gaussian ellipse of the undetermined point cloud model to be optimized;

[0027] A mean operation is performed on the product of the transparency of each Gaussian ellipse and the difference to obtain the transparency loss.

[0028] In a possible implementation, extracting three-dimensional mesh facets from the target Gaussian point cloud model and using the extracted multiple three-dimensional mesh facets to construct a three-dimensional mesh model of the object to be modeled includes:

[0029] Based on the octree algorithm, the three-dimensional space where the target Gaussian point cloud model is located is divided into a plurality of cubes; the number of target Gaussian ellipsoids in each of the cubes is less than the ellipsoid threshold;

[0030] Performing local surface fitting on all the target Gaussian ellipsoids of each cube to obtain a fitting surface of the corresponding cube;

[0031] Based on the fitted surfaces of the cubes, meshing the corresponding cubes to obtain a plurality of three-dimensional mesh facets;

[0032] Based on the fitting surfaces of each of the cubes, the multiple three-dimensional mesh facets are spliced, and the spliced ​​three-dimensional meshes are smoothed to obtain a three-dimensional mesh model of the object to be modeled.

[0033] A second aspect of the present application provides a three-dimensional mesh model acquisition device, the device comprising:

[0034] An initial Gaussian point cloud model acquisition module is used to acquire an initial Gaussian point cloud model for the object to be modeled; the initial Gaussian point cloud model is obtained by modeling the collected image of the object to be modeled using a Gaussian splash algorithm;

[0035] A point cloud processing module, configured to perform point cloud processing on the initial Gaussian point cloud model based on the three-dimensional grid model composition requirements and the initial attribute characteristics of each initial Gaussian ellipsoid constituting the initial Gaussian point cloud model, to obtain a pending point cloud model;

[0036] A three-dimensional reconstruction module, configured to perform three-dimensional reconstruction of the object to be modeled based on the Gaussian splash algorithm and the point cloud model to be determined, and obtain a target Gaussian point cloud model that satisfies a transparency constraint condition; the transparency constraint condition indicates that the transparency of target Gaussian ellipsoids in the target Gaussian point cloud model that correspond one-to-one to each surface area of ​​the object to be modeled and are independent of each other tends to a first value;

[0037] The three-dimensional mesh extraction module is used to extract three-dimensional mesh facets from the target Gaussian point cloud model, and use the extracted multiple three-dimensional mesh facets to construct a three-dimensional mesh model of the object to be modeled.

[0038] A third aspect of the present application provides an electronic device, the electronic device comprising: at least one communication element, at least one memory and at least one processor;

[0039] The communication element is used to obtain a captured image of the object to be modeled;

[0040] The processor is used to load and execute multiple computer instructions stored in the memory to implement the following steps:

[0041] Acquire an initial Gaussian point cloud model for the object to be modeled; the initial Gaussian point cloud model is obtained by modeling the collected image of the object to be modeled using a Gaussian splash algorithm;

[0042] Based on the requirements for the composition of the three-dimensional grid model and the initial attribute characteristics of each initial Gaussian ellipsoid constituting the initial Gaussian point cloud model, performing point cloud processing on the initial Gaussian point cloud model to obtain a pending point cloud model;

[0043] Based on the Gaussian splash algorithm and the undetermined point cloud model, the object to be modeled is three-dimensionally reconstructed to obtain a target Gaussian point cloud model that satisfies a transparency constraint condition; the transparency constraint condition indicates that the transparency of target Gaussian ellipsoids in the target Gaussian point cloud model that correspond one-to-one to each surface area of ​​the object to be modeled and are independent of each other tends to a first value;

[0044] A three-dimensional mesh surface is extracted from the target Gaussian point cloud model, and a three-dimensional mesh model of the object to be modeled is constructed using the extracted multiple three-dimensional mesh surface patches.

[0045] It can be seen that in the three-dimensional mesh model acquisition method provided in the present application, the captured image of the object to be modeled is modeled by the Gaussian splash algorithm to obtain the initial Gaussian point cloud model, and then the initial Gaussian point cloud model is processed based on the initial attribute characteristics of each initial Gaussian ellipsoid and the three-dimensional mesh model composition requirements to obtain a pending point cloud model with three-dimensional meshing features. On this basis, it is reconstructed by the Gaussian splash algorithm to obtain a target Gaussian point cloud model that meets the transparency constraint condition, which can better characterize the surface distribution of the object to be modeled and align with the surface of the object to be modeled, and can quickly and accurately extract three-dimensional mesh facets therefrom, thereby constituting a three-dimensional mesh model with a complete structure and accurate surface features. No further manual modification by technical personnel is required, which can provide more efficient and high-quality content support for downstream application tasks that require three-dimensional models, thereby improving the efficiency and accuracy of three-dimensional mesh model acquisition. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0047] Figure 1 A schematic diagram of the process of obtaining a three-dimensional mesh model according to the first embodiment of the present application;

[0048] Figure 2 A schematic diagram of the process of obtaining a three-dimensional mesh model according to the second embodiment of the present application;

[0049] Figure 3 This is a flow chart of the three-dimensional mesh model acquisition method proposed in Example 3 of the present application;

[0050] Figure 4 A schematic diagram of the structure of a three-dimensional grid model acquisition device proposed in an embodiment of the present application;

[0051] Figure 5 A schematic diagram of the hardware structure of the electronic device proposed in the embodiment of the present application. DETAILED DESCRIPTION

[0052] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation mode of the present application are only used to explain the specific embodiments of the present application and are not intended to limit the present application. The embodiments of the present application will be described below in conjunction with the drawings. It is known to those of ordinary skill in the art that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0053] Moreover, the terms "first", "second" etc. in the full text description content and the accompanying drawings of the present application are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0054] Regarding the description of the background technology, after analysis, it is found that the Gaussian point cloud model is caused by the obvious difference in the composition of the ordinary point cloud model. The Gaussian point cloud model reconstructed based on the Gaussian Splatting (3DGS) technology represents and renders the three-dimensional scene through 3DGS, so that it is not an ordinary point cloud that only has position, color and other information but no size. Points in space, but are composed of ellipsoids in space with size, rotation, scale and anisotropy in all directions. This will cause the three-dimensional mesh model extraction method applied to ordinary point cloud models to be unable to meet the extraction requirements of high-quality three-dimensional mesh models based on Gaussian point cloud models reconstructed by 3DGS.

[0055] In order to solve the above technical problems, the embodiment of the present application provides a three-dimensional mesh model acquisition method. By optimizing the 3DGS algorithm process, the Gaussian point cloud model (i.e., the target Gaussian point cloud model) constructed for the three-dimensional scene naturally has the characteristics of easy extraction of the three-dimensional mesh model, ensuring that a high-quality three-dimensional mesh model is extracted from the Gaussian point cloud model, and can meet the task requirements of downstream applications without modification by technical personnel, thereby improving the acquisition efficiency and reliability of the three-dimensional mesh model. The implementation steps of the three-dimensional mesh model acquisition method proposed in the embodiment of the present application will be described in detail below.

[0056] Reference Figure 1 , is a flow chart of the three-dimensional mesh model acquisition method proposed in the first embodiment of the present application. The method proposed in the embodiment of the present application can be applied to electronic devices, such as terminal devices or servers with data processing capabilities, and the three-dimensional mesh model acquisition method can also be implemented by the terminal device and the server in cooperation. Based on this, Figure 1 As shown, the three-dimensional mesh model acquisition method proposed in this embodiment may include but is not limited to the following steps:

[0057] Step S11, obtaining an initial Gaussian point cloud model for the object to be modeled; the initial Gaussian point cloud model is obtained by modeling the collected image of the object to be modeled using a Gaussian splash algorithm;

[0058] In the embodiments of the present application, the object to be modeled can be a task object of any downstream application in the fields of game development, virtual reality, metaverse, architectural design, film and television production, medical image processing, and cultural heritage protection, such as game / film characters, buildings, medical research objects, or cultural relics, etc., which can be determined based on actual conditions. The present application does not limit the type of object to be modeled.

[0059] After determining the object to be modeled, multiple acquired images of the object to be modeled can be obtained by scanning, taking pictures, etc., which may include images acquired from different perspectives of the object to be modeled, to ensure that these acquired images contain each point of the object to be modeled. In order to improve the reliability of modeling, it can be required that each point is presented in at least two or three acquired images. The present application does not limit the method for acquiring the acquired images of the object to be modeled and the number of images, which can be flexibly adjusted according to the situation.

[0060] Afterwards, this application adopts the Gaussian Splatting (3DGS) algorithm to construct a Gaussian point cloud model of the object to be modeled. Therefore, the embodiment of the present application can estimate the point cloud data from a set of collected images of the object to be modeled through the COLMAP software, and then use the 3DGS algorithm to obtain the initial Gaussian point cloud model of the object to be modeled. This application does not elaborate on the implementation process of how to use the 3DGS algorithm to construct and render the Gaussian point cloud model of the object to be modeled.

[0061] It should be understood that, relative to ordinary point cloud models, the initial Gaussian point cloud model describes the uncertainty in the point cloud by introducing probability distribution, in which each point or a group of points is represented as a Gaussian distribution, so that the initial Gaussian point cloud model contains not only position information, but also probability information about the point distribution. In practical applications, when constructing a Gaussian point cloud model, a unique representation method can be provided by a Gaussian ellipsoid, that is, by representing each point in the Gaussian point cloud as an ellipsoid with specific spatial information such as position, size, direction, and color and transparency. In order to distinguish the description, the Gaussian ellipsoid constituting the initial Gaussian point cloud model is recorded as an initial Gaussian ellipsoid in the embodiment of the present application. This information for each Gaussian ellipsoid is determined during the Gaussian splashing process, and the implementation process is not described in detail in this application.

[0062] Step S12, based on the three-dimensional grid model composition requirements and the initial attribute characteristics of each initial Gaussian ellipsoid constituting the initial Gaussian point cloud model, performing point cloud processing on the initial Gaussian point cloud model to obtain a pending point cloud model;

[0063] Combined with the above analysis of the differences between the ordinary point cloud model and the initial Gaussian point cloud model, it can be seen that in the three-dimensional meshing process, a surface independent of the viewing angle needs to be determined to characterize the object to be modeled, and the initial Gaussian ellipsoids that constitute the initial Gaussian point cloud model have spatial size and anisotropy. This plane is naturally difficult to characterize, which makes the initial Gaussian point cloud model lack the characteristics that can be three-dimensionally meshed, which leads to the poor quality of the three-dimensional mesh model directly extracted from the initial Gaussian point cloud model.

[0064] In order to improve the above-mentioned problem, the present application proposes to perform post-processing on the initial Gaussian point cloud model obtained above before performing the three-dimensional mesh model extraction operation, and obtain a target Gaussian point cloud model that has features that can be three-dimensionally meshed and is aligned with the surface of the object to be modeled, and then perform the three-dimensional mesh model extraction operation. Therefore, in order to obtain a pending point cloud model that has features that can be three-dimensionally meshed, the present application can determine the three-dimensional mesh model composition requirements (i.e., three-dimensional meshing requirements), such as the requirements for the positional correspondence between each Gaussian ellipsoid and each surface area contained in the object to be modeled, the transparency requirements of the Gaussian ellipsoid corresponding to the surface area, the representation requirements of each surface area, etc., in order to clarify what kind of point cloud model can be three-dimensionally meshed with high quality. The present application does not limit the content of the three-dimensional mesh model composition requirements and the description method, which can be determined according to the circumstances.

[0065] Combined with the above analysis of the requirements for the composition of the three-dimensional mesh model, the attribute characteristics of each initial Gaussian ellipsoid that constitutes the initial Gaussian point cloud model (for the convenience of distinguishing descriptions, they can be recorded as initial attribute characteristics), such as the above-mentioned spatial information, color values, and transparency, can be compared with the requirements for the composition of the three-dimensional mesh model. After determining how to adjust the corresponding attribute characteristics to meet the requirements for the composition of the three-dimensional mesh model, the corresponding point cloud processing of the initial Gaussian point cloud model can be implemented accordingly to obtain the required pending point cloud model.

[0066] Among them, the above transparency can refer to the opacity information of each Gaussian ellipsoid, represented by the symbol α, which can be used to describe the degree of blocking of the Gaussian ellipsoid to light, that is, the shading ability, that is, the visibility or transparency of the Gaussian ellipsoid, which is usually a value between 0 and 1. The greater the transparency, the lower the visibility of the Gaussian ellipsoid. Therefore, a transparency of 1 means that the Gaussian ellipsoid is completely opaque, that is, completely opaque; and a transparency of 0 means that the Gaussian ellipsoid is completely transparent, that is, it has no blocking effect on light. It can be seen that the transparency directly affects the display effect of the Gaussian ellipsoid in the rendered image. In order to ensure the quality of the rendered image of the object to be modeled, in the point cloud processing process of step S12, it is necessary to reasonably adjust the transparency of the initial Gaussian ellipsoid at different positions to ensure that the transparency of the initial Gaussian ellipsoid corresponding to each surface area of ​​the object to be modeled can be adjusted to 1 or close to 1. The implementation process of this application embodiment will not be described in detail.

[0067] It should be understood that the point cloud processing content of step S12 includes but is not limited to the processing of the transparency of the initial Gaussian point cloud. According to the requirements of the three-dimensional mesh model composition, it is often necessary to process the expression form, color, etc. of the initial Gaussian ellipsoid. It can be determined based on the actual comparison results of the above comparison, and this application and this embodiment will not be described in detail here.

[0068] Step S13, based on the Gaussian splash algorithm and the point cloud model to be determined, three-dimensionally reconstruct the object to be modeled to obtain a target Gaussian point cloud model that satisfies a transparency constraint condition; the transparency constraint condition indicates that the transparency of target Gaussian ellipsoids in the target Gaussian point cloud model that correspond one-to-one to each surface area of ​​the object to be modeled and are independent of each other tends to a first value;

[0069] The pending point cloud model obtained through the above point cloud processing can have the characteristics of being three-dimensionally meshed. However, after the above operation, the consistency between the pending point cloud model and the various surface areas of the object to be modeled often decreases, that is, the pending point cloud model is no longer completely aligned with the surface of the object to be modeled, affecting the rendering effect.

[0070] In order to further improve the above-mentioned problems, the present application will further utilize the Gaussian splashing algorithm to perform three-dimensional reconstruction on the basis of the undetermined point cloud model, and in the three-dimensional reconstruction process, in combination with the transparency requirements of the Gaussian ellipsoids corresponding to different position areas (such as surface areas and internal areas) of the object to be modeled included in the three-dimensional mesh model composition requirements, configure corresponding transparency constraints, so that in the process of optimizing the transparency of each Gaussian ellipsoid according to the transparency constraints, the transparency of the target Gaussian ellipsoids corresponding to each surface area of ​​the object to be modeled (that is, each surface area is occupied by a target Gaussian ellipsoid) tends to be optimized in the direction of a first value (such as 1 or a value close to 1, etc.), ensuring that each target Gaussian ellipsoid in the target Gaussian point cloud model is independent of each other, and can have a strong light-shielding ability, or even be completely opaque, reliably and accurately represent the surface distribution of the object to be modeled and align with its surface, ensure the integrity of the structure in the subsequent three-dimensional meshing process, and the three-dimensional mesh facets accurately represent the characteristics of the corresponding surface area of ​​the object to be modeled, and avoid the problem of the surface being too smooth or rough, that is, lay a reliable foundation for the subsequent three-dimensional mesh model extraction step.

[0071] Regarding the implementation process of how to use the Gaussian splash algorithm to reconstruct the point cloud model to be determined, the implementation steps of the Gaussian splash algorithm to construct the Gaussian point cloud model can be combined. In the iterative optimization process of the Gaussian point cloud model, combined with the transparency constraint conditions proposed in this application, the loss calculation of each iterative optimization is realized, so as to continuously optimize the parameters of the Gaussian point cloud model by minimizing the total loss until the transparency constraint conditions are met, etc., and the Gaussian point cloud model finally optimized is determined as the target Gaussian point cloud model of the object to be modeled. The iterative optimization implementation process of this application embodiment will not be described in detail here.

[0072] Step S14, extracting three-dimensional mesh facets from the target Gaussian point cloud model, and using the extracted multiple three-dimensional mesh facets to construct a three-dimensional mesh model of the object to be modeled.

[0073] Combined with the above analysis, the target Gaussian point cloud model obtained in the present application can be better distributed on the surface of the object to be modeled and aligned with the surface of the object to be modeled compared to the initial Gaussian point cloud model. That is to say, each surface area of ​​the object to be modeled corresponds to only one target Gaussian ellipsoid whose transparency reaches the first value, and different target Gaussian ellipsoids do not overlap, so as to avoid the color value of the corresponding surface area being affected by the overlap of the Gaussian ellipsoids. In this way, in the process of three-dimensional meshing of the target Gaussian point cloud model, it is ensured that each three-dimensional mesh face (3D mesh face, which is the basic unit of the three-dimensional mesh model, refers to a geometric structure composed of multiple vertices (vertices), edges (edges) and faces (faces) in three-dimensional space, usually a triangle) extracted therefrom is aligned with each surface area of ​​the object to be modeled, and accurately represents the corresponding surface features.

[0074] Afterwards, the corresponding 3D mesh faces extracted can be spliced ​​according to the positional relationship of each surface area of ​​the object to be modeled to obtain the required 3D mesh model of the object to be modeled, ensuring that the quality of the 3D mesh model is equivalent to that of the initial Gaussian point cloud model, and efficiently realizing the 3D reconstruction of the real object to be modeled without human intervention, thereby improving the efficiency of obtaining the 3D mesh model, reducing labor costs, and ensuring that the obtained 3D mesh model provides more efficient and high-quality content support for downstream application tasks that require high-quality 3D models, thereby reliably ensuring the user's immersion and experience in downstream applications.

[0075] Reference Figure 2 , is a flow chart of the three-dimensional mesh model acquisition method proposed in the second embodiment of the present application. This embodiment can describe an optional implementation method of how to obtain a target Gaussian point cloud model with three-dimensional meshable features from an initial Gaussian point cloud model in the three-dimensional mesh model acquisition method proposed above, such as Figure 2 As shown, the three-dimensional mesh model acquisition method proposed in this embodiment may include:

[0076] Step S21, obtaining an initial Gaussian point cloud model for the object to be modeled; the initial Gaussian point cloud model is obtained by modeling the collected image of the object to be modeled using a Gaussian splash algorithm;

[0077] Step S22, determining initial attribute features of each initial Gaussian ellipsoid constituting the initial Gaussian point cloud model; the initial attribute features include spatial information, color value and transparency of the corresponding initial Gaussian ellipsoid in three-dimensional space;

[0078] Step S23, based on the spatial information corresponding to each initial Gaussian ellipsoid, the overlapping areas between different initial Gaussian ellipsoids are separated to obtain a plurality of independent Gaussian ellipsoids to be processed;

[0079] In combination with the description of the corresponding parts of the above embodiments, such as the description of the corresponding parts of step S11, after the initial Gaussian point cloud model is constructed based on the traditional Gaussian splashing technology, due to the characteristics of the initial Gaussian ellipsoid (three-dimensional ellipsoid) that constitutes the initial Gaussian point cloud model, the quality of the three-dimensional mesh model directly extracted therefrom is very poor. The present application proposes the above-mentioned technical concept to optimize the Gaussian splashing processing flow to generate a target Gaussian point cloud model that has the characteristics of easy extraction of three-dimensional meshes.

[0080] Since the essence of the 3D meshing process is to find 3D mesh patches (surfaces that are not related to vision) to represent the object to be modeled, that is, to represent the object through a large number of interconnected and non-overlapping 3D mesh patches, the surface formed by these 3D mesh patches is the surface of the object to be modeled. It can be seen that in the 3D meshing process, it is necessary to ensure that each surface area of ​​the object to be modeled in 3D space (that is, the surface of the object to be modeled in 3D space is divided into different areas) can only be occupied by one Gaussian ellipsoid.

[0081] However, when rendering the initial Gaussian point cloud model, multiple Gaussian ellipsoids are superimposed using a transparency blending method. The multiple Gaussian ellipsoids superimposed in space jointly determine the color of a certain point in the three-dimensional space. For example, each pixel in the rendered image is superimposed from the top surface to the bottom layer according to color and transparency, and the superimposed color value is the color of each pixel. It can be seen that the multiple initial Gaussian ellipsoids that constitute the initial Gaussian point cloud model often overlap in the three-dimensional space, that is, there is an overlapping area between any at least two initial Gaussian ellipsoids (that is, the partial area of ​​the initial Gaussian ellipsoid corresponding to the same surface area in the three-dimensional space), which obviously does not meet the requirements of the three-dimensional mesh model.

[0082] In order to improve the above-mentioned problem, the present application can accurately determine which initial Gaussian ellipsoids have overlapping areas in space based on the spatial information of each initial Gaussian ellipsoid constituting the initial Gaussian point cloud model, such as size, position, scale, rotation, and direction. For example, by traversing all initial Gaussian ellipsoids one by one in three-dimensional space to determine whether they overlap with another initial Gaussian ellipsoid in space, the overlapping area between the two initial Gaussian ellipsoids can be determined. After that, the overlapping area is separated from the two initial Gaussian ellipsoids, and a new Gaussian ellipsoid to be processed is constituted by the overlapping area. At the same time, the other non-overlapping areas after the overlapping area is separated from the two initial Gaussian ellipsoids (that is, there is no spatial overlap with any initial Gaussian ellipsoid) constitute the corresponding Gaussian ellipsoid to be processed.

[0083] It should be noted that in practical applications, for any initial Gaussian ellipsoid (ellipsoid 1 for the convenience of description), there may be one or more other initial Gaussian ellipsoids (ellipsoid 2 for the convenience of description) that overlap with it in space. In this case, each overlapping area in space between the ellipsoid 1 and each ellipsoid 2 is determined as a separate Gaussian ellipsoid to be processed, and the remaining area after the ellipsoid 1 is separated from each overlapping area is determined as a separate Gaussian ellipsoid to be processed. Of course, there may be the same overlapping area in space between three or more initial Gaussian ellipsoids. In this case, the overlapping area can still be determined as a separate Gaussian ellipsoid to be processed according to the above-mentioned separation processing method to ensure that each Gaussian ellipsoid to be processed is independent of each other. This application does not give detailed examples of the separation processing implementation process of step S23.

[0084] Among them, the attribute characteristics of a Gaussian ellipsoid to be processed determined by the separated overlapping area can be determined by the attribute characteristics of multiple initial Gaussian ellipsoids with the overlapping area in the overlapping area. Usually, the spatial information of the overlapping area remains unchanged, and the color value and transparency can be respectively determined by the color values ​​and transparency of multiple initial Gaussian ellipsoids with the overlapping area in the overlapping area, such as achieved by a transparency mixing method.

[0085] In a possible implementation, the present application may adopt an integral calculation method to obtain the color value and transparency of each overlapping area, that is, to integrate the color values ​​of different initial Gaussian ellipsoids containing the same overlapping area in the overlapping area to obtain the color value of the Gaussian ellipsoid to be processed composed of the separated overlapping area; to integrate the transparency of different initial Gaussian ellipsoids containing the same overlapping area in the overlapping area to obtain the transparency of the Gaussian ellipsoid to be processed composed of the separated overlapping area. The spatial information such as position, rotation and scale of a separate Gaussian ellipsoid to be processed composed of the separated overlapping area is kept consistent with the spatial information of the overlapping area in the corresponding initial Gaussian ellipsoid before separation. Of course, the method for obtaining the attribute characteristics of each of the above-mentioned Gaussian ellipsoids to be processed includes but is not limited to this implementation method.

[0086] Step S24, two-dimensionalizing the plurality of Gaussian ellipsoids to be processed to obtain a plurality of Gaussian ellipses; the color value and transparency of the Gaussian ellipses are obtained based on the conversion of the color value and transparency of the corresponding Gaussian ellipsoids to be processed;

[0087] Combined with the above analysis of three-dimensional meshing, this application hopes that the final model can well characterize a plane, but the initial Gaussian ellipsoid or the Gaussian ellipsoid to be processed after separation is a three-dimensional ellipsoid. This application proposes that each Gaussian ellipsoid to be processed can be processed as much as possible into a two-dimensional patch on the surface of the final required model, so that it has the characteristics of extracting three-dimensional mesh patches. Therefore, this application proposes to project or convert each three-dimensional Gaussian ellipsoid to be processed onto a two-dimensional plane in a certain direction to obtain a corresponding two-dimensional ellipse. This application does not limit the two-dimensional processing method of the Gaussian ellipsoid, which can be determined according to the actual application scenario and processing requirements.

[0088] In a possible implementation, based on the spatial scales of each Gaussian ellipsoid to be processed in different directions, the two-dimensional plane perpendicular to the direction with the smallest spatial scale (equivalent to the direction in which the three-dimensional ellipsoid is thinnest or approximately thinnest) (i.e., the plane with the largest or nearly largest overlapping area with the Gaussian ellipsoid to be processed in space) can be determined as the projection plane of the Gaussian ellipsoid to be processed. After that, each Gaussian ellipsoid to be processed can be projected onto the corresponding projection plane to complete the two-dimensional processing and obtain the Gaussian ellipse (also called two-dimensional ellipsoid) corresponding to each Gaussian ellipsoid to be processed; the center point of the Gaussian ellipse is consistent with the center point of the corresponding Gaussian ellipsoid to be processed.

[0089] It should be noted that the two-dimensional processing method of the Gaussian ellipsoid during the Gaussian splashing process includes but is not limited to the method described in this embodiment. It can be determined in combination with the actual application scenario and the required accuracy requirements. It can also be based on one or more combinations such as least squares fitting, coordinate transformation, ellipsoid coordinate system, parameterization of the ellipsoid, and implicit expression of the ellipsoid equation to achieve two-dimensional processing of each Gaussian ellipsoid to be processed. The implementation process will not be described in detail one by one in this application.

[0090] Step S25, based on the transparency of each of the multiple Gaussian ellipses, the multiple Gaussian ellipses are subjected to transparency elimination to obtain a pending point cloud model; the transparency of each Gaussian ellipse constituting the pending point cloud model is a first value;

[0091] In the Gaussian point cloud model, a large number of Gaussian ellipsoids will be generated regardless of the surface, interior or exterior of the object, and finally rendered at a specific viewing angle through the transparency blending method. However, combined with the above description of the requirements for the composition of the three-dimensional mesh model and the transparency of the Gaussian ellipsoid, in the three-dimensional meshing process, only the Gaussian ellipsoids to be processed that represent the surface of the object to be modeled need to be processed, that is, the Gaussian ellipsoids to be processed that correspond to each surface area of ​​the object to be modeled need to be processed. The Gaussian ellipsoids to be processed that represent the interior and exterior of the object to be modeled are not needed in the three-dimensional meshing process.

[0092] Therefore, in order to ensure that the three-dimensional mesh patches representing various surface areas of the object to be modeled can be accurately and completely extracted when the three-dimensional mesh model extraction step is subsequently executed, the present application can set a reasonable transparency threshold, thereby implementing a transparency culling operation on each Gaussian ellipsoid to be processed, that is, removing those ellipsoids to be processed whose transparency is lower than the transparency threshold, and updating the transparency of the ellipsoids to be processed whose transparency is higher than the transparency threshold to a first value, such as 1 or a value approximately equal to 1. Through this transparency culling operation, the rendering efficiency and image quality can be improved while maintaining the expressiveness of the model.

[0093] In practical applications, in order to avoid configuring the transparency threshold too large, resulting in a small number of ellipsoids to be processed being eliminated, it is easy to cause the object to be modeled to have background information or other noise data, etc., resulting in the surface of the three-dimensional model not being smooth enough, and there may be content that does not belong to the three-dimensional model itself required by the object to be modeled, which reduces the quality of the model. At the same time, avoid configuring the transparency threshold too small, resulting in too many ellipsoids to be processed being eliminated, so that the three-dimensional model obtained by the final modeling may have problems such as structural defects or holes, which will also affect the quality of the model. The embodiment of the present application can determine a reasonable transparency threshold through multiple tests, and the present application does not limit its numerical value.

[0094] After the above-mentioned point cloud processing, the pending point cloud model obtained in the present application has the characteristics of being able to be three-dimensionally meshed, but after the above-mentioned operations, such as transparency culling operations, it often leads to a decrease in the consistency between the surface distribution represented by the pending point cloud model and the surface of the object to be modeled. It is necessary to further perform point cloud optimization processing on the pending point cloud model to obtain a target Gaussian point cloud model that can accurately represent the surface distribution of the object to be constructed. The present application does not impose any restrictions on the point cloud optimization processing method.

[0095] Step S26, based on the Gaussian splash algorithm and the point cloud model to be determined, converting the captured image of the object to be modeled into a rendered image under the same camera extrinsic parameters;

[0096] Step S27, obtaining a rendering loss between the rendered image and the corresponding captured image;

[0097] As described in the corresponding part of the above embodiment, the present application can use the Gaussian splash algorithm to reconstruct the target point cloud model based on the undetermined point cloud model to obtain the target point cloud model that itself supports the extraction of a high-quality three-dimensional mesh model. The reconstruction process includes but is not limited to the reconstruction steps described in steps S26 to S210 of this embodiment. This application is only illustrated by taking this as an example.

[0098] In the embodiment of the present application, combined with the three-dimensional reconstruction and rendering capabilities of the Gaussian splatting algorithm itself, during the iterative optimization of the point cloud model to be determined, the Splatting process can be directly performed based on the point cloud model to be determined, that is, the process of rendering the Gaussian distribution represented by the point cloud model to be determined into an image, and obtaining the rendered images of the object to be modeled under different surface perspectives. Here, according to the camera extrinsic parameters (Extrinsic Parameters) of each acquired image of the object to be modeled obtained in the process of constructing the above-mentioned initial Gaussian point cloud model, such as the position and posture in the world coordinate system, the three-dimensional point cloud can be accurately projected onto the two-dimensional image plane, and an image similar to the acquired image under the corresponding camera extrinsic parameters can be rendered, that is, each acquired image is converted into a rendered image under the corresponding camera extrinsic parameters. The rendering process is not described in detail in the present application.

[0099] Afterwards, the rendered image and the captured image of the same camera extrinsic parameter can be compared, and the corresponding loss can be calculated from the difference between the two images, which is recorded as rendering loss. The present application does not limit the type of loss function used in the rendering loss calculation process, and can be configured or adjusted according to actual needs, such as using the L1 loss function to calculate the absolute difference between the rendered image and the captured image, etc. The present application does not limit the implementation method of step S27.

[0100] Exemplarily, the rendering loss L can be calculated using the following formula: image :

[0101] (1)

[0102] Wherein, H may represent the height of the captured image / rendered image, and W may represent the width of the captured image / rendered image, and H and W are fixed; R i,j It can represent the color value of the rendered image at the pixel (i, j) under the camera extrinsic parameters of the Gaussian point cloud model, such as the RGB value; G i,j It can represent the color value of the collected image at the pixel (i, j) input by the Gaussian point cloud model; abs() can represent the absolute value function. It can be seen that formula (1) can calculate the absolute difference between the color values ​​of the pixel at the same position of the rendered image and the collected image under the same camera external parameters, that is, abs(R i,j -G i,j ), sum the absolute differences of all pixels of the image, and then divide it by the area of ​​the image to obtain the average value of the absolute differences of each pixel of the image, which is determined as the rendering loss L between the rendered image and the captured image image .

[0103] Step S28, obtaining the transparency loss based on the transparency of each Gaussian ellipse of the to-be-determined point cloud model and the total number of Gaussian ellipses;

[0104] In combination with the steps of obtaining the pending point cloud model described above, this application uses the Gaussian splash algorithm to reconstruct the pending point cloud model. For each Gaussian ellipse, its spatial information (such as position, scale, and variance) can be kept unchanged, and only its transparency and color values ​​are optimized to optimize the model performance and improve the quality of the rendered image.

[0105] Among them, in the transparency optimization process, combined with the above description of the requirements for the composition of the three-dimensional mesh model, in order to ensure that the transparency of the Gaussian ellipsoid representing the surface of the object to be modeled is greater than the transparency threshold, it is expected to be close to a first value, such as 1; the transparency of the Gaussian ellipsoid representing the inside or outside of the object to be modeled is less than the transparency threshold, and it is expected to be close to a second value, such as 0; the present application can add transparency constraints in the transparency optimization process to achieve the expected result, so that the transparency of each Gaussian ellipsoid of the iteratively optimized model approaches 0 or 1, that is, the transparency of the Gaussian ellipse corresponding to each surface area of ​​the object to be modeled is optimized in the direction of increasing to approach the first value, and the transparency of the Gaussian ellipse that does not correspond to any surface area of ​​the object to be modeled is optimized in the direction of decreasing to approach the second value.

[0106] Therefore, the present application can determine the difference between the transparency optimization target value (such as the first value) and the transparency of each Gaussian ellipse of the undetermined point cloud model to be optimized this time, and perform a mean operation on the product of the transparency of each Gaussian ellipse and the difference to obtain the transparency loss, which is calculated as shown in the following formula:

[0107] (2)

[0108] In formula (2), α g It can represent the transparency of the g-th Gaussian ellipse, N represents the total number of Gaussian ellipses, L alpha It can represent the transparency loss of the model after this iteration optimization.

[0109] Step S29, performing weighted summation of rendering loss and transparency loss to obtain target loss;

[0110] Step S210, optimizing the color value and transparency of each Gaussian ellipse of the to-be-determined point cloud model by minimizing the target loss, and obtaining a target Gaussian point cloud model that satisfies the transparency constraint condition;

[0111] After the above method, the corresponding rendering loss L is obtained after each iteration of the model optimization. image and transparency loss L alpha After that, we can perform a weighted summation on the two to obtain the target loss L for this iterative optimization, that is, L=L image +λL alpha , λ can represent the weight value, that is, the influence of transparency loss on the total loss / rendering quality, which can be adjusted according to actual data, and this application does not limit its value.

[0112] Afterwards, the parameters of this iterative optimization model, even the parameters and weights of the above loss function, can be adjusted based on the target loss. By continuously optimizing the color value and transparency of each Gaussian ellipse, the target loss can be reduced. After several iterative optimizations, the target Gaussian point cloud model that meets the transparency constraint is obtained. At this time, the transparency of the target Gaussian ellipsoid corresponding to each surface area of ​​the object to be modeled (such as the Gaussian ellipsoid obtained by the above Gaussian ellipse reconstruction optimization) constituting the target Gaussian point cloud model is close to 1, and the transparency of the Gaussian ellipsoid corresponding to the inside or outside of the object to be modeled is close to 0.

[0113] Step S211 , extracting three-dimensional mesh facets from the target Gaussian point cloud model, and using the extracted multiple three-dimensional mesh facets to construct a three-dimensional mesh model of the object to be modeled.

[0114] Following the above processing, the target Gaussian point cloud model obtained in the embodiment of the present application can better characterize the surface distribution of the object to be modeled, align with the surface of the object to be modeled, and support accurate processing of three-dimensional meshing compared to the initial Gaussian point cloud model. This application does not limit the implementation method of how to extract the three-dimensional mesh model from the target Gaussian point cloud model.

[0115] In summary, the present application adopts a standard Gaussian splash algorithm, and uses the collected image of the object to be modeled to construct an initial Gaussian point cloud model of the object to be modeled. The initial Gaussian ellipsoids superimposed in space can be separated and processed to obtain independent Gaussian ellipsoids to be processed. After two-dimensional processing and transparency elimination operations, a pending point cloud model with three-dimensional grid features is obtained. In order to obtain a Gaussian point cloud model aligned with the label of the object to be modeled, a Gaussian splash algorithm can be used to reconstruct the pending point cloud model, iteratively optimize the reconstructed model, and after each iterative optimization, calculate the rendering distance between the corresponding rendered image and the collected image. The target loss after this optimization is obtained by weighted summation, and the transparency loss of each current Gaussian ellipse is continuously optimized by minimizing the target loss to obtain the target Gaussian point cloud loss that meets the transparency constraint. Then, a three-dimensional mesh model extraction operation is performed on it to obtain a high-quality three-dimensional mesh model with a complete structure and each three-dimensional mesh facet accurately representing the corresponding surface area characteristics of the object to be modeled. This model can be directly used in downstream applications without human intervention, and can meet the needs of downstream applications for building high-quality three-dimensional models, thereby improving the efficiency and accuracy of obtaining three-dimensional mesh models and reducing labor costs.

[0116] Reference Figure 3 , is a flow chart of the three-dimensional mesh model acquisition method proposed in the third embodiment of the present application. This embodiment can provide a detailed description of an optional extraction method for extracting a three-dimensional mesh model from a target Gaussian point cloud model in the three-dimensional mesh model acquisition method proposed above. For the acquisition steps of the target Gaussian point cloud model, please refer to the description of the corresponding part of the above embodiment, which will not be repeated in this embodiment. Based on this, Figure 3 As shown, the 3D mesh model extraction process may include:

[0117] Step S31, based on the octree algorithm, the three-dimensional space where the target Gaussian point cloud model is located is divided into multiple cubes; the number of target Gaussian ellipsoids in each cube is less than the ellipsoid threshold;

[0118] For the above-mentioned reconstructed target Gaussian point cloud model, the three-dimensional space occupied by the target Gaussian point cloud model can be segmented according to the distribution density of the Gaussian ellipsoid to obtain multiple cubes. Optionally, the segmentation of the three-dimensional space can be based on but not limited to the octree algorithm. The octree algorithm is a tree data structure for managing three-dimensional space data. It recursively divides the three-dimensional space where the target Gaussian point cloud model is located into smaller and smaller cubes (i.e., subspaces, which can be called octants) to ensure that the number of Gaussian ellipsoids in each cube is within a certain threshold (referred to as the ellipsoid threshold, and this application does not limit its value). This application does not elaborate on the implementation process of the three-dimensional space segmentation of step S31.

[0119] Step S32, performing local surface fitting on all target Gaussian ellipsoids of each cube to obtain a fitting surface of the corresponding cube;

[0120] For all the target Gaussian ellipsoids in each cube obtained by the above segmentation, the moving least square method can be used to perform local surface fitting processing, that is, the position information of all the target Gaussian ellipsoids is subjected to surface fitting to obtain the fitting surface of the cube. Of course, the present application can also use other fitting algorithms to implement step S32, but since the more complex the basis function for determining a fitting surface is, the more complex the fitting surface obtained is, and the more resources are consumed, the embodiment of the present application can flexibly select the fitting algorithm based on the actual situation, and the present application only uses the fitting algorithm of the moving least square method as an example for explanation.

[0121] In the process of obtaining the fitting surface, the difference between the distribution of each Gaussian ellipsoid in the cube and the fitting surface can be minimized by optimizing the unknown parameters in the basis function (such as the moving least squares method) to accurately obtain the fitting surface of each cube.

[0122] Step S33, based on the fitting surfaces of the cubes, meshing the corresponding cubes to obtain a plurality of three-dimensional mesh facets;

[0123] Combined with the above-mentioned method of obtaining the cube, each fitting surface can be considered as a local surface of the object to be modeled. In this way, by judging the relationship between each vertex of the cube and the fitting surface, the inside or outside of the object to be modeled can be determined for each vertex, so as to accurately and comprehensively extract each three-dimensional mesh face. For example, various situations in which each of the eight vertices of the cube is located inside or outside the target Gaussian point cloud model are determined, and 15 possible modes are determined after analysis. The connection method of each triangle (i.e., the shape of the three-dimensional mesh face) is fixed, thereby executing various possible vertex connections to achieve the segmentation of the corresponding cube and obtain each three-dimensional mesh face of the three-dimensional mesh model.

[0124] Based on this, in a possible implementation, the intersection points of the fitting surfaces of each cube and the edges of the corresponding cube can be obtained, and the intersection points can be determined as the vertices of the three-dimensional mesh patches of the corresponding cube. Based on the fitting surfaces of each cube, the distribution results of the vertices of the corresponding cube inside and outside the target Gaussian point cloud model are determined, and then based on the distribution results, the vertices of the three-dimensional mesh patches of the corresponding cube are connected into one or more triangles, and the triangles are determined as three-dimensional mesh patches.

[0125] Step S34, based on the fitting surfaces of each cube, multiple three-dimensional mesh facets are spliced, and the spliced ​​three-dimensional meshes are smoothed to obtain a three-dimensional mesh model of the object to be modeled.

[0126] According to the method described above, the three-dimensional mesh patches of each surface area of ​​the object to be modeled can be accurately and completely extracted, and each three-dimensional mesh patch represents the characteristics of the corresponding surface area. In this way, the three-dimensional mesh patches extracted from each fitting surface are spliced ​​according to the positional relationship between each fitting surface, and then the spliced ​​three-dimensional mesh is smoothed to obtain a high-quality three-dimensional mesh model of the object to be modeled, ensuring the structural integrity of the three-dimensional mesh model and being able to accurately represent the surface of the object to be modeled, providing more efficient and high-quality content support for downstream applications that require three-dimensional models, without the need for human intervention, thereby improving the efficiency and accuracy of obtaining three-dimensional mesh models.

[0127] A three-dimensional mesh model method provided in an embodiment of the present application is introduced above, and a device for executing the above-mentioned three-dimensional mesh model method will be introduced below.

[0128] Reference Figure 4 , is a schematic diagram of the structure of a three-dimensional grid model device proposed in an embodiment of the present application, such as Figure 4 As shown, the three-dimensional grid model device may include:

[0129] An initial Gaussian point cloud model acquisition module 41 is used to acquire an initial Gaussian point cloud model for the object to be modeled; the initial Gaussian point cloud model is obtained by modeling the collected image of the object to be modeled using a Gaussian splash algorithm;

[0130] The point cloud processing module 42 is used to perform point cloud processing on the initial Gaussian point cloud model based on the three-dimensional grid model composition requirements and the initial attribute characteristics of each initial Gaussian ellipsoid constituting the initial Gaussian point cloud model to obtain a pending point cloud model;

[0131] A three-dimensional reconstruction module 43 is used to perform three-dimensional reconstruction on the object to be modeled based on the Gaussian splash algorithm and the undetermined point cloud model to obtain a target Gaussian point cloud model that satisfies a transparency constraint condition; the transparency constraint condition indicates that the transparency of target Gaussian ellipsoids in the target Gaussian point cloud model that correspond one-to-one to each surface area of ​​the object to be modeled and are independent of each other tends to a first value;

[0132] The three-dimensional mesh extraction module 44 is used to extract three-dimensional mesh facets from the target Gaussian point cloud model, and use the extracted multiple three-dimensional mesh facets to construct a three-dimensional mesh model of the object to be modeled.

[0133] In a possible implementation, the point cloud processing module 42 may include:

[0134] An initial attribute feature determination unit, used to determine initial attribute features of each initial Gaussian ellipsoid constituting the initial Gaussian point cloud model; the initial attribute features include spatial information, color value and transparency of the corresponding initial Gaussian ellipsoid in three-dimensional space;

[0135] A separation processing unit, configured to separate overlapping areas between different initial Gaussian ellipsoids based on the spatial information corresponding to each initial Gaussian ellipsoid, to obtain a plurality of independent Gaussian ellipsoids to be processed;

[0136] A two-dimensional processing unit, used for performing two-dimensional processing on the plurality of Gaussian ellipsoids to be processed, and obtaining a plurality of Gaussian ellipses correspondingly; the color value and transparency of the Gaussian ellipses are obtained based on the conversion processing of the color value and transparency corresponding to the Gaussian ellipsoids to be processed;

[0137] A transparency elimination unit is used to perform transparency elimination on the multiple Gaussian ellipses based on the transparency of each of the multiple Gaussian ellipses to obtain a pending point cloud model; wherein the transparency of each of the Gaussian ellipses constituting the pending point cloud model is the first value.

[0138] In a possible implementation, the 3D reconstruction module 43 may include:

[0139] A rendering image obtaining unit, used for converting the captured image of the object to be modeled into a rendering image under the same camera extrinsic parameters based on the Gaussian splash algorithm and the point cloud model to be determined;

[0140] A rendering loss acquisition unit, used to acquire a rendering loss between the rendered image and the corresponding acquired image;

[0141] A transparency loss obtaining unit, configured to obtain a transparency loss based on the transparency of each Gaussian ellipse of the to-be-determined point cloud model and the total number of the Gaussian ellipses;

[0142] A target loss obtaining unit, used for performing a weighted summation of the rendering loss and the transparency loss to obtain a target loss;

[0143] The optimization unit is used to optimize the color value and transparency of each Gaussian ellipse of the undetermined point cloud model by minimizing the target loss, so as to obtain a target Gaussian point cloud model that meets the transparency constraint condition.

[0144] In some embodiments, during the optimization process of the above-mentioned optimization unit, the spatial information of each Gaussian ellipse remains unchanged, and the transparency of the Gaussian ellipse corresponding to each surface area of ​​the object to be modeled is optimized in an increasing direction, and the transparency of the Gaussian ellipse not corresponding to any surface area of ​​the object to be modeled is optimized in a decreasing direction.

[0145] In a possible implementation, the transparency loss obtaining unit may include:

[0146] A difference determination unit, used to determine the difference between the transparency optimization target value and the transparency of each Gaussian ellipse of the undetermined point cloud model to be optimized this time;

[0147] The transparency loss obtaining unit is used to perform a mean operation on the product of the transparency of each Gaussian ellipse and the difference to obtain the transparency loss.

[0148] In some embodiments, the three-dimensional mesh extraction module 44 may include:

[0149] A segmentation unit is used to divide the three-dimensional space where the target Gaussian point cloud model is located into a plurality of cubes based on an octree algorithm; the number of target Gaussian ellipsoids in each of the cubes is less than an ellipsoid threshold;

[0150] A fitting surface obtaining unit is used to perform local surface fitting on all the target Gaussian ellipsoids of each cube to obtain a fitting surface of the corresponding cube;

[0151] A meshing unit, used for meshing the corresponding cubes based on the fitted surfaces of the cubes to obtain a plurality of three-dimensional mesh facets;

[0152] The splicing processing unit is used to splice the multiple three-dimensional mesh facets based on the fitting surfaces of each of the cubes, and to smooth the spliced ​​three-dimensional meshes to obtain a three-dimensional mesh model of the object to be modeled.

[0153] In one possible implementation, the gridding unit may include:

[0154] An intersection acquisition unit, used for acquiring the intersection points between the fitting surface of each cube and each edge of the corresponding cube, and determining the intersection points as vertices of the three-dimensional mesh patch of the corresponding cube;

[0155] A distribution result determination unit, used for determining the distribution result of each vertex of the corresponding cube inside and outside the target Gaussian point cloud model based on the fitted surfaces of each cube;

[0156] The three-dimensional mesh face determination unit is used to connect the three-dimensional mesh face vertices of the corresponding cube into one or more triangles based on the distribution result, and determine the triangles as three-dimensional mesh faces.

[0157] In a possible implementation, the separation processing unit may include:

[0158] A Gaussian ellipsoid determination unit to be processed is used to separate the overlapping areas between different initial Gaussian ellipsoids based on the spatial information corresponding to each initial Gaussian ellipsoid, and determine the initial Gaussian ellipsoid after separating the overlapping area as the Gaussian ellipsoid to be processed;

[0159] Based on this, the two-dimensional processing unit may include:

[0160] A color value obtaining unit is used to perform an integral operation on the color values ​​of the overlapping regions of the different initial Gaussian ellipsoids containing the same overlapping region, so as to obtain the color value of the Gaussian ellipsoid to be processed formed by the separated overlapping regions;

[0161] A transparency forming unit, used for performing an integral operation on the transparency of each of the different initial Gaussian ellipsoids containing the same overlapping area in the overlapping area, to obtain the transparency of the Gaussian ellipsoid to be processed formed by the separated overlapping area;

[0162] A projection plane determining unit, configured to determine, based on the spatial scales in different directions of each of the Gaussian ellipsoids to be processed, a two-dimensional plane perpendicular to the direction with the smallest spatial scale as the projection plane of the Gaussian ellipsoid to be processed;

[0163] The projection unit is used to project each of the Gaussian ellipsoids to be processed onto the corresponding projection plane to obtain a corresponding Gaussian ellipse; the center point of the Gaussian ellipse is consistent with the center point of the corresponding Gaussian ellipsoid to be processed.

[0164] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any of the three-dimensional mesh model acquisition methods provided in the embodiments of the present application.

[0165] Wherein, when the computer program instructions are loaded and executed on a computer (electronic device), the above-mentioned process or function according to the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, training equipment, or data center to another website, computer, training equipment, or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0166] Reference Figure 5 , is a schematic diagram of the structure of an electronic device proposed in an embodiment of the present application. The electronic device may be a terminal device or a server. The electronic device is a server as an example for explanation. The server may be one or more physical servers, or a cloud server supporting cloud computing, such as Figure 5 As shown, the electronic device may include: at least one communication element 510, at least one memory 520 and at least one processor 530, wherein:

[0167] At least one communication element 510, at least one memory 520 and at least one processor 530 can communicate with each other via a bus. The bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0168] The communication element 510 can be used to obtain a captured image of the object to be modeled, which can be obtained by an independent image acquisition device acquiring an image of the real object to be modeled, such as a laser scanner, a camera, or a terminal device with an image acquisition function, etc. These captured images are used as original images for constructing a Gaussian point cloud model, input into an electronic device, and the three-dimensional mesh model acquisition method proposed in this application is executed to obtain a three-dimensional mesh model of the object to be modeled. The three-dimensional mesh model can also be sent to an execution device that executes downstream application tasks.

[0169] Based on this, the above-mentioned communication element 510 may include but is not limited to: supporting one or more communication elements corresponding to wireless communication methods such as wifi, 5G / 6G mobile communication, Bluetooth and / or near-field channel, GSM (Global System of Mobile communication) communication, GPRS (General Packet Radio Service) communication, etc., so that the electronic device can realize data transmission with other devices (such as the above-mentioned image acquisition device, execution device, database or other data platform, etc.) through the communication element.

[0170] It should be understood that in order to transmit data between the various components inside the electronic device, the communication element 510 may also include one or more interfaces that support wired communication, such as a general-purpose input / output (GPIO) interface, a USB interface, a universal asynchronous receiver / transmitter (UART) interface, etc. The present application does not limit the type of communication element 510 contained in the electronic device and its communication transmission mechanism, which can be determined according to the circumstances.

[0171] The memory 520 can be used to store computer instructions and data, and may include an instruction storage area and a data storage area. The data storage area can store various data, such as captured images of the object to be modeled and intermediate data and results generated during the three-dimensional mesh model acquisition process; the instruction storage area can store software units such as an operating system, an application, and computer instructions required for at least one function (such as a display function, a modeling function, etc.). In an embodiment of the present application, the memory 520 can provide the processor 530 with multiple computer instructions for implementing the three-dimensional mesh model acquisition method of the embodiment of the present application, as well as data or other software and hardware resources required during the execution of the three-dimensional mesh model acquisition method.

[0172] The processor 530, as the control center of the electronic device, can use various interfaces and lines to connect various parts of the entire electronic device, and implement the various steps of the three-dimensional mesh model acquisition method proposed in the embodiment of the present application by loading and executing multiple computer instructions stored in the memory 520. Other components of the electronic device (such as one or more input components and output components, etc.) can also be called to implement corresponding functions.

[0173] In the embodiment of the present application, the memory 520 may include a volatile memory, such as a random access memory (RAM), and may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a mechanical hard disk (HDD) or a solid state drive (SSD) and one or more forms of storage media. The processor 130 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), a digital signal processor (DSP), an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). The present application does not limit the types of the memory 520 and the processor 530 and their working principles, and can be flexibly configured according to actual needs.

[0174] It should be understood that Figure 5 The structure of the electronic device shown does not constitute a limitation on the electronic device in the embodiment of the present application. In practical applications, the electronic device may include Figure 5 More or less components as shown, or a combination of some components, especially when the electronic device is any one or more terminal devices such as a smart phone, a tablet computer, a wearable device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC) or a robot, the electronic device may also include at least one input component such as a touch sensing unit for sensing touch events on a touch display panel, a keyboard, a mouse, a camera, a microphone, etc.; at least one output component such as a display, a speaker, a vibration mechanism, an indicator light, etc.; an antenna, a radio frequency unit; a variety of sensors; a power module, etc. The hardware structure can be determined according to the type of electronic device and its functional requirements, and this application does not list them one by one here.

[0175] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any three-dimensional mesh model acquisition method provided in an embodiment of the present application.

[0176] The computer-readable storage medium may be any available medium that can be stored by a computer or a data storage device such as a training device or a data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0177] In addition, it should be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0178] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by dedicated hardware including dedicated integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. That is to say, in the above embodiments, all or part of the implementation can be achieved by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product.

Claims

1. A three-dimensional mesh model acquisition method, characterized in that: The method comprises: Acquire an initial Gaussian point cloud model for the object to be modeled; the initial Gaussian point cloud model is obtained by modeling the collected image of the object to be modeled using a Gaussian splash algorithm; Based on the requirements for the composition of the three-dimensional grid model and the initial attribute characteristics of each initial Gaussian ellipsoid constituting the initial Gaussian point cloud model, performing point cloud processing on the initial Gaussian point cloud model to obtain a pending point cloud model; Based on the Gaussian splash algorithm and the undetermined point cloud model, the object to be modeled is three-dimensionally reconstructed to obtain a target Gaussian point cloud model that satisfies a transparency constraint condition; the transparency constraint condition indicates that the transparency of target Gaussian ellipsoids in the target Gaussian point cloud model that correspond one-to-one to each surface area of ​​the object to be modeled and are independent of each other tends to a first value; A three-dimensional mesh surface is extracted from the target Gaussian point cloud model, and a three-dimensional mesh model of the object to be modeled is constructed using the extracted multiple three-dimensional mesh surface patches.

2. The method according to claim 1, characterized in that The method of performing point cloud processing on the initial Gaussian point cloud model based on the three-dimensional grid model composition requirements and the initial attribute characteristics of each initial Gaussian ellipsoid constituting the initial Gaussian point cloud model to obtain a pending point cloud model includes: Determine initial attribute features of each initial Gaussian ellipsoid constituting the initial Gaussian point cloud model; the initial attribute features include spatial information, color value and transparency of the corresponding initial Gaussian ellipsoid in three-dimensional space; Based on the spatial information corresponding to each of the initial Gaussian ellipsoids, the overlapping areas between different initial Gaussian ellipsoids are separated to obtain a plurality of independent Gaussian ellipsoids to be processed; Performing two-dimensional processing on the multiple Gaussian ellipsoids to be processed to obtain multiple Gaussian ellipses; the color value and transparency of the Gaussian ellipses are obtained based on the conversion processing of the color value and transparency of the Gaussian ellipsoids to be processed; Based on the respective transparency of the multiple Gaussian ellipses, the multiple Gaussian ellipses are subjected to transparency elimination to obtain a pending point cloud model; wherein the transparency of each of the Gaussian ellipses constituting the pending point cloud model is the first value.

3. The method according to claim 1 or 2, characterized in that: The method of performing three-dimensional reconstruction on the object to be modeled based on the Gaussian splash algorithm and the point cloud model to be determined to obtain a target Gaussian point cloud model that satisfies the transparency constraint condition includes: Based on the Gaussian splash algorithm and the to-be-determined point cloud model, converting the captured image of the to-be-modeled object into a rendered image under the same camera extrinsic parameters; Obtaining a rendering loss between the rendered image and the corresponding acquired image; Obtaining transparency loss based on the transparency of each Gaussian ellipse of the to-be-determined point cloud model and the total number of the Gaussian ellipses; Performing a weighted summation on the rendering loss and the transparency loss to obtain a target loss; By minimizing the target loss, the color value and transparency of each Gaussian ellipse of the undetermined point cloud model are optimized to obtain a target Gaussian point cloud model that satisfies the transparency constraint condition.

4. The method according to claim 3, characterized in that In the process of optimizing the color value and transparency of each Gaussian ellipse of the undetermined point cloud model: The spatial information of each Gaussian ellipse remains unchanged; The transparency of the Gaussian ellipse corresponding to each surface area of ​​the object to be modeled is optimized in an increasing direction, and the transparency of the Gaussian ellipse not corresponding to any surface area of ​​the object to be modeled is optimized in a decreasing direction.

5. The method according to claim 3, characterized in that: The obtaining of the transparency loss based on the transparency of each Gaussian ellipse of the to-be-determined point cloud model and the total number of the Gaussian ellipses comprises: Determine the difference between the transparency optimization target value and the transparency of each Gaussian ellipse of the undetermined point cloud model to be optimized; A mean operation is performed on the product of the transparency of each Gaussian ellipse and the difference to obtain the transparency loss.

6. The method according to claim 3, characterized in that The step of extracting three-dimensional mesh facets from the target Gaussian point cloud model and using the extracted multiple three-dimensional mesh facets to construct a three-dimensional mesh model of the object to be modeled comprises: Based on the octree algorithm, the three-dimensional space where the target Gaussian point cloud model is located is divided into a plurality of cubes; the number of target Gaussian ellipsoids in each of the cubes is less than the ellipsoid threshold; Performing local surface fitting on all the target Gaussian ellipsoids of each cube to obtain a fitting surface of the corresponding cube; Based on the fitted surfaces of the cubes, meshing the corresponding cubes to obtain a plurality of three-dimensional mesh facets; Based on the fitting surfaces of each of the cubes, the multiple three-dimensional mesh facets are spliced, and the spliced ​​three-dimensional meshes are smoothed to obtain a three-dimensional mesh model of the object to be modeled.

7. The method according to claim 6, characterized in that The step of meshing the corresponding cubes based on the fitting surfaces of the cubes to obtain a plurality of three-dimensional mesh facets includes: Obtaining the intersection points of the fitting surface of each cube and each edge of the corresponding cube, and determining the intersection points as vertices of the three-dimensional mesh patch of the corresponding cube; Based on the fitted surfaces of the cubes, determining the distribution results of the vertices of the corresponding cubes inside and outside the target Gaussian point cloud model; Based on the distribution result, the vertices of the three-dimensional mesh surface of the corresponding cube are connected into one or more triangles, and the triangles are determined as three-dimensional mesh surfaces.

8. The method according to claim 2, characterized in that: Based on the spatial information corresponding to each of the initial Gaussian ellipsoids, the overlapping areas between the different initial Gaussian ellipsoids are separated to obtain a plurality of independent Gaussian ellipsoids to be processed, and the plurality of Gaussian ellipsoids to be processed are two-dimensionalized to obtain a plurality of Gaussian ellipses, including: Based on the spatial information corresponding to each of the initial Gaussian ellipsoids, separation processing is performed on the overlapping areas between different initial Gaussian ellipsoids, and the initial Gaussian ellipsoid after the overlapping area is separated is determined as the Gaussian ellipsoid to be processed; Performing an integral operation on the color values ​​of the overlapping regions of the different initial Gaussian ellipsoids containing the same overlapping region, to obtain the color value of the Gaussian ellipsoid to be processed formed by the separated overlapping regions; Performing an integral operation on the transparency of each of the different initial Gaussian ellipsoids containing the same overlapping area in the overlapping area, to obtain the transparency of the Gaussian ellipsoid to be processed formed by the separated overlapping area; Based on the spatial scales of each of the Gaussian ellipsoids to be processed in different directions, determining a two-dimensional plane perpendicular to the direction with the smallest spatial scale as the projection plane of the Gaussian ellipsoid to be processed; Each of the Gaussian ellipsoids to be processed is projected onto the corresponding projection plane to obtain a corresponding Gaussian ellipse; the center point of the Gaussian ellipse is consistent with the center point of the corresponding Gaussian ellipsoid to be processed.

9. A three-dimensional mesh model acquisition device, characterized in that: The device comprises: An initial Gaussian point cloud model acquisition module is used to acquire an initial Gaussian point cloud model for the object to be modeled; the initial Gaussian point cloud model is obtained by modeling the collected image of the object to be modeled using a Gaussian splash algorithm; A point cloud processing module, configured to perform point cloud processing on the initial Gaussian point cloud model based on the three-dimensional grid model composition requirements and the initial attribute characteristics of each initial Gaussian ellipsoid constituting the initial Gaussian point cloud model, to obtain a pending point cloud model; A three-dimensional reconstruction module, configured to perform three-dimensional reconstruction of the object to be modeled based on the Gaussian splash algorithm and the point cloud model to be determined, and obtain a target Gaussian point cloud model that satisfies a transparency constraint condition; the transparency constraint condition indicates that the transparency of target Gaussian ellipsoids in the target Gaussian point cloud model that correspond one-to-one to each surface area of ​​the object to be modeled and are independent of each other tends to a first value; The three-dimensional mesh extraction module is used to extract three-dimensional mesh facets from the target Gaussian point cloud model, and use the extracted multiple three-dimensional mesh facets to construct a three-dimensional mesh model of the object to be modeled.

10. An electronic device, characterized in that: The electronic device comprises: at least one communication element, at least one memory and at least one processor; The communication element is used to obtain a captured image of the object to be modeled; The processor is used to load and execute multiple computer instructions stored in the memory to implement the following steps: Acquire an initial Gaussian point cloud model for the object to be modeled; the initial Gaussian point cloud model is obtained by modeling the collected image of the object to be modeled using a Gaussian splash algorithm; Based on the requirements for the composition of the three-dimensional grid model and the initial attribute characteristics of each initial Gaussian ellipsoid constituting the initial Gaussian point cloud model, performing point cloud processing on the initial Gaussian point cloud model to obtain a pending point cloud model; Based on the Gaussian splash algorithm and the undetermined point cloud model, the object to be modeled is three-dimensionally reconstructed to obtain a target Gaussian point cloud model that satisfies a transparency constraint condition; the transparency constraint condition indicates that the transparency of target Gaussian ellipsoids in the target Gaussian point cloud model that correspond one-to-one to each surface area of ​​the object to be modeled and are independent of each other tends to a first value; A three-dimensional mesh surface is extracted from the target Gaussian point cloud model, and a three-dimensional mesh model of the object to be modeled is constructed using the extracted multiple three-dimensional mesh surface patches.

Images