Methods, apparatuses, devices, media, and products for generating a three-dimensional mesh

Abstract

Embodiments of the present disclosure relate to methods, apparatuses, devices, media and products for generating a three-dimensional mesh. The method comprises obtaining a first point cloud corresponding to an original three-dimensional mesh of a target object by sampling the original three-dimensional mesh using a first sampling method. The method further comprises obtaining a second point cloud corresponding to the original three-dimensional mesh by sampling the original three-dimensional mesh using a second sampling method. The method further comprises determining a first key quantized representation and a first value quantized representation corresponding to the first point cloud. The method further comprises determining a second key quantized representation and a second value quantized representation corresponding to the second point cloud. The method further comprises generating a target three-dimensional mesh for the target object based on the first key quantized representation, the first value quantized representation, the second key quantized representation and the second value quantized representation. Through the method, the quality of the reconstructed three-dimensional mesh is improved, and the user experience is improved.

Description

Technical Field

[0001] The embodiments of this disclosure generally relate to the field of image processing, and more specifically to methods, apparatuses, devices, media, and program products for generating three-dimensional meshes. Background Technology

[0002] Currently, the machine learning industry is developing rapidly, and with this development, machine learning is being applied to various scenarios. Furthermore, with the advancement of machine learning, cross-disciplinary collaborations can be facilitated between different fields.

[0003] With the booming development of industries related to machine learning, machine learning's data processing capabilities in certain fields are becoming increasingly powerful, and its advantages in some areas are becoming more and more apparent. Especially in the field of image processing, machine learning model-related technologies are emerging in large numbers. Nowadays, machine learning models can be used not only to process 2D images but also 3D images. Processing 3D graphics is more complex than processing 2D images, requiring a much larger volume of data. Therefore, there are still many aspects to be studied regarding the application of machine learning models in processing 3D images. Summary of the Invention

[0004] Embodiments of this disclosure provide a method, apparatus, device, medium, and program product for generating three-dimensional meshes.

[0005] According to a first aspect of this disclosure, a method for generating a three-dimensional mesh is provided. The method includes obtaining a first point cloud corresponding to an original three-dimensional mesh of a target object by sampling the original three-dimensional mesh using a first sampling method. The method further includes obtaining a second point cloud corresponding to the original three-dimensional mesh by sampling the original three-dimensional mesh using a second sampling method. The method further includes determining a first key-quantized representation and a first value-quantized representation corresponding to the first point cloud. The method further includes determining a second key-quantized representation and a second value-quantized representation corresponding to the second point cloud. The method further includes generating a target three-dimensional mesh for the target object based on the first key-quantized representation, the first value-quantized representation, the second key-quantized representation, and the second value-quantized representation.

[0006] In a second aspect of this disclosure, an apparatus for generating a three-dimensional mesh is provided. The apparatus includes a first point cloud acquisition module configured to obtain a first point cloud corresponding to an original three-dimensional mesh of a target object by sampling the original three-dimensional mesh using a first sampling method; a second point cloud acquisition module configured to obtain a second point cloud corresponding to the original three-dimensional mesh by sampling the original three-dimensional mesh using a second sampling method; a first key quantization representation and a first value quantization representation determination module configured to determine a first key quantization representation and a first value quantization representation corresponding to the first point cloud; a second key quantization representation and a second value quantization representation determination module configured to determine a second key quantization representation and a second value quantization representation corresponding to the second point cloud; and a target three-dimensional mesh generation module configured to generate a target three-dimensional mesh for a target object based on the first key quantization representation, the first value quantization representation, the second key quantization representation, and the second value quantization representation.

[0007] In a third aspect of this disclosure, an electronic device is provided, including at least one processor; and a storage device for storing at least one program, which, when executed by the at least one processor, causes the at least one processor to implement the method according to the first aspect of this disclosure.

[0008] In a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the method according to a first aspect of this disclosure.

[0009] In a fifth aspect of this disclosure, a computer program product is provided. This computer program product includes a computer program that, when executed by a processor, implements the method according to a first aspect of this disclosure.

[0010] It should be understood that the content described in this section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The above and other objects, features and advantages of this disclosure will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.

[0012] Figure 1 The illustration shows a schematic diagram of an example environment in which some embodiments of the present disclosure may be implemented;

[0013] Figure 2 The illustration shows a schematic diagram of an example method 200 for generating a three-dimensional mesh according to some embodiments of the present disclosure;

[0014] Figure 3 The illustration shows an example of generating a first point cloud and a second point cloud according to some embodiments of the present disclosure;

[0015] Figure 4 The illustration shows a schematic diagram of an example model architecture for generating a three-dimensional mesh according to some embodiments of the present disclosure;

[0016] Figure 5 The illustration shows a schematic diagram of an example of a three-dimensional mesh for evaluating reconstruction according to some embodiments of the present disclosure;

[0017] Figure 6A The illustration is a schematic diagram of an example of a bar chart showing the distribution of multiple datasets according to some embodiments of the present disclosure;

[0018] Figure 6B The illustration shows an example of a pie chart showing the distribution of multiple datasets according to some embodiments of the present disclosure;

[0019] Figure 6C The illustration shows a schematic diagram of examples of multiple objects in multiple datasets according to some embodiments of the present disclosure;

[0020] Figure 7 The illustration shows a schematic block diagram of an apparatus for generating a three-dimensional mesh according to some embodiments of the present disclosure;

[0021] Figure 8 A schematic block diagram of an example device suitable for implementing various embodiments of the present disclosure is illustrated.

[0022] In the various figures, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0023] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0024] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0025] For example, upon receiving a user's proactive request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0026] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0027] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0028] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0029] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0030] In the process of generating 3D meshes, existing 3D generation techniques generally follow this paradigm. First, point clouds are sampled from a 3D shape table using uniform sampling. Then, a variational autoencoder (VAE) is trained to compress the input point cloud into a small-dimensional latent code and decode (reconstruct) it into a 3D shape. Subsequently, after training the VAE, a randomly partitioned test set is used to evaluate the quality of the reconstructed 3D model, that is, to evaluate the ability of the VAE to reconstruct a 3D model.

[0031] The variational autoencoders trained using traditional methods in the aforementioned traditional schemes have certain drawbacks. For example, when training a variational autoencoder, the point cloud obtained using uniform sampling methods is neither efficient nor accurate enough in representing information about the salient regions of an object. This results in insufficient reconstruction capability of the variational autoencoder trained using the obtained point cloud, leading to lower quality reconstructed 3D models. This limits the performance ceiling of diffusion models trained based on traditional schemes and reduces the user experience when using variational autoencoders.

[0032] To address at least the aforementioned and other potential problems, embodiments of this disclosure propose a method for generating a 3D mesh. In this method, a computing device first samples the original 3D mesh of a target object using a first sampling method to obtain a first point cloud corresponding to the original 3D mesh. Then, it samples the original 3D mesh using a second sampling method to obtain a second point cloud corresponding to the original 3D mesh. The second sampling method is used to sample the edges in the original 3D mesh and is different from the first sampling method. Subsequently, the computing device determines a first key quantization representation and a first value quantization representation corresponding to the first point cloud, and a second key quantization representation and a second value quantization representation corresponding to the second point cloud. Finally, the computing device generates a target 3D mesh for the target object based on the first key quantization representation, the first value quantization representation, the second key quantization representation, and the second value quantization representation. This method, by utilizing at least two different sampling methods to sample the target object when generating the target 3D mesh, obtains at least two point clouds, one of which includes edge information. This improves the preservation of fine-grained shape features, retains key geometric details as much as possible, improves the reconstruction quality of the 3D mesh, and enhances the user experience.

[0033] The embodiments of this disclosure will now be described in further detail with reference to the accompanying drawings. Figure 1 The illustrations show example environments in which the devices and / or methods of embodiments of the present disclosure may be implemented. In environment 100, computing device 102 is used to process computational tasks required to generate a three-dimensional mesh. In one example, the disclosed three-dimensional mesh is composed of multiple triangles. In another example, the three-dimensional mesh is composed of geometric shapes of other shapes, such as quadrilaterals. The above examples are merely for describing the present disclosure and are not intended to specifically limit the present disclosure.

[0034] Examples of computing device 102 include, but are not limited to, personal computers, server computers, handheld or laptop devices, mobile devices (such as mobile phones, personal digital assistants (PDAs), media players, etc.), multiprocessor systems, consumer electronics, minicomputers, mainframe computers, and distributed computing environments that include any of the above systems or devices.

[0035] It is understood that this disclosure is only described in conjunction with relevant embodiments and is not intended to limit the scope of protection of this disclosure. Any technical solutions in other disclosures that fall within the scope of protection of this disclosure should be protected.

[0036] like Figure 1 As shown, computing device 102 acquires the original 3D mesh 108 of target object 104. Computing device 102 can obtain a first point cloud 112 corresponding to the original 3D mesh 108 by sampling the original 3D mesh 108 of target object 104 using the first sampling method 106.

[0037] In some embodiments, the computing device 102 may sample the original 3D mesh 108 using an average sampling method, such as using a Poisson uniform sampling method. The resulting first point cloud 112 is used to characterize the approximate, coarse distribution of the point cloud in the original 3D mesh 108.

[0038] The computing device 102 can also obtain a second point cloud 114 corresponding to the original 3D mesh 108 by sampling the original 3D mesh 108 using a second sampling method 110. In some embodiments, the second point cloud 114 sampled by the second sampling method 110 for the original 3D mesh 108 is used to characterize the fine distribution of the point cloud in the original 3D mesh 108. Compared with the first point cloud 112, the distribution of the second point cloud 114 is more accurate and denser. For example, the second sampling method 110 is a sampling method for sharp edges in the original 3D mesh 108 of the target object 104.

[0039] It is understandable that the first sampling method 106 and the second sampling method 110 are different sampling methods, and the first sampling method 106 is a relatively average sampling method, while the second sampling method 110 is a more refined sampling method.

[0040] In some embodiments, the point cloud information of the first point cloud 112 and the second point cloud 114 can be combined, for example, by adding the first point cloud 112 and the second point cloud 114 together, to obtain a set of dense point clouds for the original three-dimensional mesh 108. This set of dense point clouds can more efficiently and more accurately represent the original three-dimensional mesh 108.

[0041] Subsequently, the computing device 102 further obtains a first key quantization representation 116 and a first value quantization representation 118 corresponding to the first point cloud 112. Furthermore, the computing device 102 also obtains a second key quantization representation 120 and a second value quantization representation 122 corresponding to the second point cloud 114.

[0042] After acquiring the first key quantization representation 116, the first value quantization representation 118, the second key quantization representation 120, and the second value quantization representation 122, the computing device 102 uses the acquired first key quantization representation 116, first value quantization representation 118, second key quantization representation 120, and second value quantization representation 122 to generate a target three-dimensional mesh 124 for the target object 104.

[0043] In some embodiments, a quantization representation for a first point cloud 112 is obtained by inputting a first key quantization representation 116 and a first value quantization representation 118 as inputs into a cross-attention module, and a quantization representation for a second point cloud 114 is obtained by inputting a second key quantization representation 120 and a second value quantization representation 122 as inputs into another cross-attention module.

[0044] This method utilizes two different sampling methods to sample the target object during the generation of the target 3D mesh, thereby obtaining salient point clouds and average point clouds to better represent the target object. Furthermore, a dual cross-attention mechanism is employed to enhance the reconstruction effect, thus improving the preservation of fine-grained shape features, retaining key geometric details as much as possible, improving the reconstruction quality of the 3D mesh, and enhancing the user experience.

[0045] The above combination Figure 1 The following is a schematic diagram illustrating an example environment in which some embodiments of this disclosure may be implemented, in conjunction with... Figure 2 A schematic diagram illustrating an example method 200 for generating a three-dimensional mesh according to some embodiments of the present disclosure is shown. This example method can be... Figure 1 The computing device 102 or any suitable computing device in the system shall execute the test.

[0046] like Figure 2 As shown, in example method 200, at box 202, a first point cloud corresponding to the original 3D mesh of the target object is obtained by sampling the original 3D mesh using a first sampling method. For example, computing device 102 uses first sampling method 106 to sample the original 3D network 108 to obtain the first point cloud 112.

[0047] The first sampling method for the first point cloud is a relatively average sampling method, such as the Poisson uniform sampling method. The distribution of the first point cloud obtained after sampling is relatively coarse, and the reconstruction performance for high-fineness meshes is poor, making it impossible to accurately represent the original 3D mesh.

[0048] It is understood that other suitable sampling methods besides Poisson uniform sampling can be used to perform sampling on the original 3D mesh in order to obtain the first point cloud, and this disclosure does not limit this.

[0049] Next, at frame 204, a second point cloud corresponding to the original 3D mesh is obtained by sampling the original 3D mesh using a second sampling method, which is used to sample the edges in the original 3D mesh and is different from the first sampling method. For example, computing device 102 uses the second sampling method 110 to sample the original 3D mesh 108 to obtain the second point cloud 114.

[0050] The second sampling method for the second point cloud 114 differs from the uniform sampling methods mentioned above, such as Poisson uniform sampling. The second sampling method samples the edges in the original 3D mesh. Specifically, the second sampling method samples sharp edges in the original 3D mesh. For ease of description, these sharp edges can also be referred to as target edges, and the dihedral angle of the target edge is less than a threshold angle.

[0051] In some embodiments, the original 3D mesh contains many edges, each formed by the intersection of two faces. For edges formed by the intersection of two faces, there exists a certain angle, generally referred to as a dihedral angle. Edges with dihedral angles less than a threshold angle can be considered sharp edges. For example, when the predetermined threshold angle is 30 degrees, if an edge in the original 3D mesh has a dihedral angle less than 30 degrees, this edge is considered a sharp edge. The second sampling method samples these sharp edges to obtain a second point cloud for the original 3D mesh. For example, the vertices of the sharp edges are sampled to obtain the second point cloud. The accuracy and saliency of the second point cloud obtained using the second sampling method are higher than those of the first point cloud obtained using the first sampling method.

[0052] It should be noted that since the first point cloud is obtained through Poisson uniform sampling, it can effectively characterize the low-frequency region of the target object, while the second point cloud is obtained through sharp edge sampling, and therefore can more effectively characterize the high-frequency region of the target object.

[0053] In some embodiments, a combined point cloud for the original 3D mesh can be obtained by combining the first point cloud and the second point cloud. This combined point cloud includes uniform point clouds and salient point clouds, which can more efficiently characterize the high-frequency and low-frequency regions of the target object.

[0054] In some embodiments, the target object is obtained from multiple objects in multiple datasets. The computing device may also determine multiple edges of each of the multiple objects from the multiple datasets, and determine the number of sharp edges among the multiple edges whose dihedral angle is less than a predetermined threshold (e.g., 30 degrees), and then divide the multiple objects into a predetermined number (e.g., 4) of object sets based on the number of sharp edges.

[0055] Then, at box 206, a first key quantization representation and a first value quantization representation corresponding to the first point cloud are determined. For example, computing device 102 can process the first point cloud to obtain a first key quantization representation 116 and a first value quantization representation 118 for the first point cloud 112.

[0056] After the computing device obtains the first point cloud using the first sampling method, it can further determine the first key (K) quantization representation and the first value (V) quantization representation corresponding to the first point cloud. For example, the key vector and value vector that can be used in the first cross-attention module can be obtained by combining the point cloud data with a predetermined vector matrix.

[0057] At box 208, a second key quantization representation and a second value quantization representation corresponding to the second point cloud are determined. For example, computing device 102 determines a second key quantization representation 120 and a second value quantization representation 122 corresponding to the second point cloud 114.

[0058] Similarly, after the computing device obtains the second point cloud using the second sampling method, it can further determine the second key quantization representation and the second value quantization representation corresponding to the second point cloud. For example, by combining the second point cloud data with a predetermined vector matrix, key vectors and value vectors that can be used in the second cross-attention module can be obtained.

[0059] At box 210, a target 3D mesh for the target object is generated based on the first key quantization representation, the first value quantization representation, the second key quantization representation, and the second value quantization representation. For example, computing device 102 uses the first key quantization representation 116, the first value quantization representation 118, the second key quantization representation 120, and the second value quantization representation 122 to obtain the target 3D mesh 124.

[0060] In some embodiments, after obtaining the first key quantization representation 116, the first value quantization representation 118, the second key quantization representation 120, and the second value quantization representation 122, the computing device 102 applies the obtained first key quantization representation 116 and first value quantization representation 118 to the first cross-attention module, and applies the second key quantization representation 120 and second value quantization representation 122 to the second cross-attention module to generate a first quantization representation for the first point cloud and a second quantization representation for the second point cloud.

[0061] In some embodiments, before generating a first quantized representation of the first point cloud and a second quantized representation of the second point cloud, a downsampling operation may also be performed on the first point cloud and the second point cloud.

[0062] It should be noted that since the first point cloud and the second point cloud obtained by using the first sampling method and the second sampling method have different dimensions, it is necessary to downsample the first point cloud and the second point cloud to make their dimensions consistent, so as to facilitate the calculation of subsequent quantization representation features and the generation of the target 3D mesh.

[0063] For example, the first point cloud has 16,384 points and the second point cloud has 13,684 points. After downsampling, both the first and second point clouds have 512 points. Downsampling reduces the consumption of computing resources and improves computational efficiency.

[0064] After downsampling the first and second point clouds, first data information for the first point cloud and second data information for the second point cloud are determined. The computing device then uses the acquired first and second data information to determine a quantized representation of the query (Q) for the first and second point clouds. For example, the first and second data information obtained from the point clouds are concatenated to form the quantized representation of the query.

[0065] Finally, the computing device uses the acquired query quantization representation, combined with the previously acquired first key quantization representation, first value quantization representation, second key quantization representation, and second value quantization representation, to ultimately generate a target 3D mesh for the target object.

[0066] In some embodiments, the computing device applies a first key quantization representation, a first value quantization representation, and a query quantization representation to a first cross-attention module to obtain a first quantization representation for a first point cloud, and the computing device also applies a second key quantization representation, a second value quantization representation, and a query quantization representation to a second cross-attention module to obtain a second quantization representation for a second point cloud.

[0067] The computing device then combines a first quantized representation of the first point cloud and a second quantized representation of the second point cloud to determine a combined quantized representation of the second point cloud for the first point cloud. For example, the two quantized representations are added together to determine this combined quantized representation. Finally, the combined quantized representation is used to generate a target 3D mesh for the target object.

[0068] In some embodiments, the processes of generating the first key-quantized representation, the first value-quantized representation, the second key-quantized representation, and the second value-quantized representation, and the process of generating the target 3D mesh for the target object, are implemented using a machine learning model. Additionally, the machine learning model used may be a variational autoencoder-based machine learning model.

[0069] Training a machine learning model can be achieved by obtaining the first and second point clouds of the original 3D mesh of the sample object, and then using the first point cloud, the second point cloud, the original 3D mesh, and the generated target 3D mesh to train the machine learning model.

[0070] In some embodiments, after generating the target 3D mesh, it is necessary to evaluate the generated target 3D mesh. The computing device first acquires a first normal map for the original 3D mesh and a second normal map for the target 3D mesh. For example, the first and second normal maps are obtained by performing a rendering operation on the original 3D mesh and the target 3D mesh. Then, the computing device 102 uses the acquired first and second normal maps to evaluate the target 3D mesh.

[0071] In some embodiments, when evaluating the target 3D mesh, the computing device 102 also performs edge detection on the first normal map to determine the target region for the first normal map. For example, it obtains a mask map corresponding to the normal map, where the target region in the mask map corresponds to the edge normal. Next, the points of the target region corresponding to the edge normal are set to 1, and the points corresponding to other non-edge regions are set to 0. Subsequently, the computing device 102 performs a dilation operation on the target region, for example, setting the points adjacent to the points corresponding to the edge normal in the mask map to 1, to determine the mask region for the target region. Then, the computing device 102 uses the mask region to determine the third normal map for the mask region in the first normal map and the fourth normal map for the mask region in the second normal map. Subsequently, by calculating the mean square error between the third normal map and the fourth normal map, the calculated mean square error is used to evaluate the generated target 3D mesh.

[0072] Additionally, machine learning models can be evaluated, for example, by calculating the F-score corresponding to the machine learning model or by calculating the chamfer distance corresponding to the machine learning model.

[0073] In some embodiments, the training machine learning model can also be reconstructed and evaluated by dividing a predetermined number of object sets from multiple datasets.

[0074] This method utilizes two different sampling methods to sample the target object during the generation of the target 3D mesh, thereby obtaining salient point clouds and average point clouds to better represent the target object. Furthermore, a dual cross-attention mechanism is employed to enhance the reconstruction effect of the variational autoencoder, identifying and processing geometrically complex regions during training, improving the preservation of fine-grained shape features, and retaining key geometric details as much as possible. A new metric is also introduced to evaluate the reconstruction quality of the 3D mesh and the trained machine learning model. This improves the reconstruction quality of the 3D mesh and enhances the user experience.

[0075] The above combination Figure 2 Schematic diagrams of example methods for generating three-dimensional meshes according to some embodiments of the present disclosure are described below. Figure 3 A schematic diagram illustrating examples of generating a first point cloud and a second point cloud according to some embodiments of the present disclosure.

[0076] like Figure 3 As shown in Example 300, during the sampling process of the original 3D mesh 302, the first point cloud 304 for the original 3D mesh can be determined using the Poisson uniform sampling method, and the second point cloud 306 for the original 3D mesh 302 can be determined using the sharp edge sampling method. Finally, the determined first and second point clouds are combined, for example, by adding the first and second point clouds together, to determine the dense point cloud 308 for the original 3D mesh.

[0077] By using two different sampling methods to sample the original 3D mesh, the sampling accuracy of the high-frequency region of the target object can be improved, and the high-frequency region of the target object can be efficiently characterized. At the same time, the sampling of the low-frequency region of the target object is also preserved, making the final dense point cloud more accurate and containing more comprehensive point cloud information.

[0078] The above combination Figure 3 Schematic diagrams illustrating examples of generating a first point cloud and a second point cloud according to some embodiments of the present disclosure are described below. Figure 4 A schematic diagram illustrating an example of a model architecture for generating a three-dimensional mesh according to some embodiments of the present disclosure.

[0079] like Figure 4 As shown in Example 400, after sampling the original 3D mesh corresponding to the target object, a first point cloud 402 and a second point cloud 404 are obtained. At the same time, the first key quantization representation and the first value quantization representation for the first point cloud 402, as well as the first key quantization representation and the second value quantization representation for the second point cloud 404, are determined.

[0080] Subsequently, the first point cloud 402 and the second point cloud 404 are downsampled to determine the query quantization representation 406 for the first point cloud and the second point cloud. Then, the query quantization representation 406, the first key quantization representation, and the first value quantization representation are applied to the first cross-attention module 408 to generate the first quantization representation for the first point cloud, and the query quantization representation 406, the second key quantization representation, and the second value quantization representation are applied to the second cross-attention module 410 to generate the second quantization representation for the second point cloud.

[0081] After obtaining the first quantized representation for the first point cloud and the second quantized representation for the second point cloud, the first quantized representation and the second quantized representation are combined, for example, by adding the first quantized representation and the second quantized representation, and then the combined quantized representation is generated.

[0082] The generated combinatorial quantization representation is then applied to the self-attention module 412 to generate a processed combinatorial quantization representation. The processed combinatorial quantization representation is then applied to the self-attention module 414 to further generate processed key quantization and value quantization representations.

[0083] Next, at 416, the query quantization representation corresponding to the spatial point, along with the processed key quantization and value quantization representations, is applied to the cross-attention module 418. Then, at 420, an occupancy space (i.e., occupancy field) for the target object is generated. For example, the values ​​of points inside the target object in the spatial point are set to 0, and the values ​​of points outside the target object in the spatial point are set to 1. Finally, at 422, the target 3D mesh 424 for the target object is drawn using iso-surface 3D implicit functions.

[0084] The above combination Figure 4 Schematic diagrams illustrating examples of model architectures for generating three-dimensional meshes according to some embodiments of the present disclosure are shown below. Figure 5 This illustration depicts an example of evaluating a reconstructed 3D mesh according to some embodiments of the present disclosure. This example can be derived from... Figure 1 The computing device 102 or any suitable device in the system can be used for execution.

[0085] like Figure 5 As shown, after generating the target 3D mesh 516 for the target object, the generated target 3D mesh 516 needs to be evaluated.

[0086] The computing device 102 first acquires a first normal map 502 for the original three-dimensional mesh and a second normal map 520 for the target three-dimensional mesh 516 after rendering 518.

[0087] Next, the computing device 102 performs edge detection 504 on the acquired first normal map 502, for example, using the Canny operator, to obtain a mask map 506 for the first normal map, which includes the target region, the normal of which corresponds to the edge. For example, the value corresponding to the target region is set to 1, and the values ​​corresponding to other regions in the mask map are set to 0. Then, a mask region is generated by dilating the target region, thereby obtaining a mask map 510 that includes the mask region for the target region. For example, the values ​​of points adjacent to the target region are also adjusted to 1 to form the mask region.

[0088] After determining the mask region for the target region, a third normal map 512 for the mask region in the first normal map and a fourth normal map 514 for the mask region in the second normal map are further determined based on the mask region. In one example, the third and fourth normal maps can be determined by multiplying the first normal map and the mask map, as well as the second normal map and the mask map.

[0089] Finally, after determining the third and fourth normal maps, the computing device also calculates the mean square error between the third and fourth normal maps, and uses the calculated mean square error to evaluate the target 3D mesh.

[0090] Additionally, machine learning models can be evaluated. For example, they can be evaluated by calculating the F-score or the chamfer distance corresponding to the machine learning model.

[0091] Additionally, the machine learning model can be evaluated using multiple sets of objects targeting the target object, which are divided by the number of edges of each object in the multiple objects, for example, by the number of sharp edges of multiple edges of each object in the multiple objects whose dihedral angles are less than a predetermined threshold (e.g., 30 degrees).

[0092] The above combination Figure 5 Schematic diagrams illustrating examples of evaluating reconstructed 3D meshes according to some embodiments of the present disclosure are described below. Figure 6A This is a schematic diagram illustrating an example of a histogram showing the distribution of multiple datasets according to some embodiments of the present disclosure. Typically, when evaluating variational autoencoders using randomly partitioned test sets, it is impossible to comprehensively assess the reconstruction quality of the variational autoencoder for different 3D model complexities, especially its reconstruction quality for high-complexity models. This leads to bias in the evaluation of the trained variational autoencoder, affecting the judgment of the model reconstruction quality. The following, in conjunction with... Figures 6A-6C This section introduces a new method for partitioning the dataset.

[0093] like Figure 6A As shown in Example 600A, multiple datasets in dataset 602 are open-source datasets, including dataset 1, dataset 2, dataset 3, and dataset 4.

[0094] Furthermore, multiple objects from multiple datasets were divided into four datasets, ranging from Level 1 to Level 4, with each level containing a corresponding number of objects. Additionally, the Levels were determined by the number of sharp edges in the objects; lower levels indicated fewer sharp edges in the objects.

[0095] The above combination Figure 6A A schematic diagram illustrating an example of a bar chart showing the distribution of multiple datasets according to some embodiments of this disclosure is provided below. Figure 6B A schematic diagram illustrating an example of a pie chart showing the distribution of multiple datasets according to some embodiments of the present disclosure.

[0096] like Figure 6B As shown in Example 600B, combined with the previous Example 600A, the proportion of the four different levels of object sets can be clearly seen from the pie chart.

[0097] The Level 1 object set contains 800 objects, accounting for 22.2% of the total number of multi-objects across all four datasets. The Level 2 object set also contains 800 objects, accounting for 22.2% of the total number of multi-objects across all four datasets. The Level 3 object set contains 1112 objects, accounting for 30.9% of the total number of multi-objects across all four datasets. The Level 4 object set contains 890 objects, accounting for 24.7% of the total number of multi-objects across all four datasets.

[0098] The above combination Figure 6B A schematic diagram illustrating an example of a pie chart showing the distribution of multiple datasets according to some embodiments of this disclosure is provided below. Figure 6C A schematic diagram illustrating examples of multiple objects in multiple datasets according to some embodiments of the present disclosure.

[0099] like Figure 6CAs shown in Example 600C, combining with the previous Examples 600A and 600B, multiple datasets are divided into four object sets. Objects in set 604, corresponding to Level 1, have relatively smooth surfaces with significantly fewer sharp edges. Objects in set 606, corresponding to Level 2, have partially convex surfaces, and the number of sharp edges is increased compared to set 604. Objects in set 608, corresponding to Level 3, have more convex surfaces, and the number of sharp edges is increased compared to set 606. Objects in set 610, corresponding to Level 4, have extremely convex surfaces, and the number of sharp edges is increased compared to set 610. This demonstrates that the number of sharp edges in object sets from Level 1 to Level 4 shows a gradually increasing trend.

[0100] Additionally, objects in Level 1 have a sharp edge count greater than 0 and less than or equal to 5000; objects in Level 2 have a sharp edge count greater than 5000 and less than or equal to 10000; objects in Level 3 have a sharp edge count greater than 10000 and less than or equal to 50000; and objects in Level 4 have a sharp edge count greater than 50000.

[0101] It should be noted that all objects in the four object sets in Example 600C are either authorized objects or objects obtained by utilizing the technical solutions disclosed herein.

[0102] like Figure 7 As shown, the device 700 includes a first point cloud acquisition module 702, configured to obtain a first point cloud corresponding to the original three-dimensional mesh by sampling the original three-dimensional mesh of the target object using a first sampling method; a second point cloud acquisition module 704, configured to obtain a second point cloud corresponding to the original three-dimensional mesh by sampling the original three-dimensional mesh using a second sampling method; a first key quantization representation and value quantization representation determination module 706, configured to determine the first key quantization representation and the first value quantization representation corresponding to the first point cloud; a second key quantization representation and value quantization representation determination module 708, configured to determine the second key quantization representation and the second value quantization representation corresponding to the second point cloud; and a target three-dimensional mesh generation module 710, configured to generate a target three-dimensional mesh for the target object based on the first key quantization representation, the first value quantization representation, the second key quantization representation, and the second value quantization representation.

[0103] In some embodiments, the second point cloud acquisition module 704 includes: a target edge determination module configured to determine a target edge in the original three-dimensional mesh based on the original three-dimensional mesh, wherein the dihedral angle of the target edge is less than a threshold angle; and a second point cloud determination module configured to determine a second point cloud corresponding to the original three-dimensional mesh by sampling the target edge.

[0104] In some embodiments, the first point cloud acquisition module 702 includes a Poisson uniform sampling module, configured to obtain a first point cloud corresponding to the original three-dimensional mesh by sampling the original three-dimensional mesh of the target object using the Poisson uniform sampling method.

[0105] In some embodiments, the target 3D mesh generation module 710 includes: a downsampling module configured to determine first data information and second data information by performing downsampling operations on a first point cloud and a second point cloud; a query quantization representation determination module configured to determine a query quantization representation for the first point cloud and the second point cloud based on the first data information and the second data information; and a target 3D mesh generation module configured to generate a target 3D mesh for a target object based on a first key quantization representation, a first value quantization representation, a second key quantization representation, a second value quantization representation, and a query quantization representation.

[0106] In some embodiments, the target 3D mesh generation module includes: a first quantization representation determination module configured to determine a first quantization representation for a first point cloud by applying a first key quantization representation, a first value quantization representation, and a query quantization representation to a first cross-attention module; a second quantization representation determination module configured to determine a second quantization representation for a second point cloud by applying a second key quantization representation, a second value quantization representation, and a query quantization representation to a second cross-attention module; a combined quantization representation determination module configured to determine a combined quantization representation for the first point cloud and the second point cloud based on the first quantization representation and the second quantization representation; and a target 3D mesh generation module configured to generate a target 3D mesh for a target object based on the combined quantization representation.

[0107] In some embodiments, the apparatus 700 further includes: a first normal map and a second normal map determination module, configured to determine a first normal map for the original three-dimensional mesh and a second normal map for the target three-dimensional mesh; and a target three-dimensional mesh evaluation module, configured to evaluate the target three-dimensional mesh based on the first normal map and the second normal map.

[0108] In some embodiments, the target 3D mesh evaluation module includes: a target region determination module configured to determine a target region for a first normal map by performing edge detection on the first normal map; a mask region determination module configured to determine a mask region for the target region by performing a dilation operation on the target region; a third normal map and a fourth normal map determination module configured to determine, based on the mask region, a third normal map for the mask region in the first normal map and a fourth normal map for the target mask in the second normal map; and a target 3D mesh evaluation module configured to evaluate the target 3D mesh based on the third normal map and the fourth normal map.

[0109] In some embodiments, the target 3D mesh evaluation module includes: a mean square error calculation module configured to calculate the mean square error between a third normal map and a fourth normal map; and a target 3D mesh evaluation module configured to evaluate the target 3D mesh based on the mean square error.

[0110] In some embodiments, the determination of the first key quantization representation and the first value quantization representation, the determination of the second key quantization representation and the second value quantization representation, and the generation of the target 3D mesh are achieved using a machine learning model.

[0111] In some embodiments, the training module for the machine learning model includes: a first sample point cloud and a second sample point cloud acquisition module, configured to acquire the first sample point cloud and the second sample point cloud of the original 3D mesh of the sample object; and a machine learning model training module, configured to train the machine learning model based on the first sample point cloud and the second sample point cloud, the original 3D mesh of the sample, and the 3D mesh of the sample target.

[0112] In some embodiments, the training module for the machine learning model further includes: evaluating the machine learning model by at least one of the following: an F-score calculation module configured to calculate an F-score corresponding to the machine learning model; or a chamfer distance calculation module configured to calculate a chamfer distance corresponding to the machine learning model.

[0113] In some embodiments, the training module for the machine learning model further includes: a plurality of object acquisition module configured to acquire a plurality of objects in a plurality of datasets for evaluating the machine learning model; a plurality of edge determination module configured to determine a plurality of edges for each of the plurality of objects; a plurality of object partitioning module configured to partition the plurality of objects into a predetermined number of object sets based on the number of sharp edges whose dihedral angles are less than a threshold angle among the plurality of edges; and a machine learning model evaluation module configured to evaluate the machine learning model using the predetermined number of object sets.

[0114] Figure 8A schematic block diagram of an example device 800 that can be used to implement embodiments of the present disclosure is shown. Figure 1 The computing device 102 can be implemented using device 800. As shown, device 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 802 or loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 can also store various programs and data required for the operation of device 800. CPU 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0115] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0116] The various processes and handling described above, such as method 200 and examples 300 to 700, can be executed by processing unit 801. For example, in some embodiments, method 200 and examples 300 to 700 can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by CPU 801, one or more actions of example methods 200 and examples 300 to 700 described above can be performed.

[0117] This disclosure can be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0118] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0119] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0120] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0121] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0122] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0123] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0124] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0125] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for generating a three-dimensional mesh, comprising: A first point cloud corresponding to the original three-dimensional mesh is obtained by sampling the original three-dimensional mesh of the target object using a first sampling method; A second point cloud corresponding to the original 3D mesh is obtained by sampling the original 3D mesh using a second sampling method, the second sampling method being used to sample the edges in the original 3D mesh and being different from the first sampling method; Determine the first key quantization representation and the first value quantization representation corresponding to the first point cloud; Determine the second key quantization representation and the second value quantization representation corresponding to the second point cloud; as well as Based on the first key quantization representation, the first value quantization representation, the second key quantization representation, and the second value quantization representation, a target 3D mesh is generated for the target object.

2. The method according to claim 1, wherein obtaining a second point cloud corresponding to the original three-dimensional mesh by sampling the original three-dimensional mesh using a second sampling method comprises: Based on the original 3D mesh, the target edge in the original 3D mesh is determined, and the dihedral angle of the target edge is less than a threshold angle; as well as By sampling the target edge, the second point cloud corresponding to the original three-dimensional mesh is determined.

3. The method according to claim 1, wherein obtaining a first point cloud corresponding to the original three-dimensional mesh of the target object by sampling the original three-dimensional mesh of the target object using a first sampling method comprises: The first point cloud corresponding to the original three-dimensional mesh is obtained by sampling the original three-dimensional mesh of the target object using the Poisson uniform sampling method.

4. The method of claim 1, wherein generating a target 3D mesh for the target object comprises: The first data information and the second data information are determined by downsampling the first point cloud and the second point cloud; Based on the first data information and the second data information, a query quantization representation for the first point cloud and the second point cloud is determined; as well as Based on the first key quantization representation, the first value quantization representation, the second key quantization representation, the second value quantization representation, and the query quantization representation, a target 3D mesh is generated for the target object.

5. The method according to claim 4, wherein generating a target 3D mesh for the target object based on the first key quantization representation, the first value quantization representation, the second key quantization representation, the second value quantization representation, and the query quantization representation comprises: By applying the first key quantization representation, the first value quantization representation, and the query quantization representation to the first cross-attention module, a first quantization representation for the first point cloud is determined. By applying the second key quantization representation, the second value quantization representation, and the query quantization representation to the second cross-attention module, a second quantization representation for the second point cloud is determined. Based on the first quantization representation and the second quantization representation, a combined quantization representation for the first point cloud and the second point cloud is determined; as well as Based on the combined quantization representation, the target 3D mesh is generated for the target object.

6. The method according to claim 1, further comprising: Determine a first normal map for the original 3D mesh and a second normal map for the target 3D mesh; as well as The target 3D mesh is evaluated based on the first normal map and the second normal map.

7. The method of claim 6, wherein evaluating the target 3D mesh based on the first normal map and the second normal map comprises: By performing edge detection on the first normal map, the target region for the first normal map is determined; By performing an expansion operation on the target region, a mask region for the target region is determined; Based on the mask region, a third normal map for the mask region in the first normal map and a fourth normal map for the mask region in the second normal map are determined; as well as The target 3D mesh is evaluated based on the third normal map and the fourth normal map.

8. The method of claim 7, wherein the target 3D mesh is evaluated based on the third normal map and the fourth normal map: Calculate the mean square error between the third normal plot and the fourth normal plot: and The target 3D mesh is evaluated based on the mean square error.

9. The method according to claim 1, wherein the determination of the first key quantization representation and the first value quantization representation, the determination of the second key quantization representation and the second value quantization representation, and the generation of the target three-dimensional mesh are implemented using a machine learning model.

10. The method of claim 9, wherein training the machine learning model comprises: Obtain the first and second sample point clouds of the original 3D mesh for the sample object; as well as The machine learning model is trained based on the first sample point cloud and the second sample point cloud, the original 3D mesh of the sample, and the target 3D mesh of the sample.

11. The method of claim 10, further comprising: The machine learning model is evaluated by at least one of the following: Calculate the F-score corresponding to the machine learning model; or Calculate the chamfer distance corresponding to the machine learning model.

12. The method of claim 10, further comprising: Obtain multiple objects from multiple datasets used to evaluate the machine learning model; Determine multiple edges for each of the plurality of objects; Based on the number of sharp edges among the plurality of edges whose dihedral angles are less than a threshold angle, the plurality of objects are divided into a predetermined number of object sets; and The machine learning model is evaluated using the predetermined number of object sets.

13. An apparatus for generating a three-dimensional mesh, comprising: The first point cloud acquisition module is configured to obtain a first point cloud corresponding to the original three-dimensional mesh by sampling the original three-dimensional mesh of the target object using a first sampling method; The second point cloud acquisition module is configured to obtain a second point cloud corresponding to the original three-dimensional mesh by sampling the original three-dimensional mesh using a second sampling method. The second sampling method is used to sample the edges in the original three-dimensional mesh and is different from the first sampling method. The module for determining the first key quantization representation and the first value quantization representation is configured to determine the first key quantization representation and the first value quantization representation corresponding to the first point cloud. The second key quantization representation and value quantization representation determination module is configured to determine the second key quantization representation and the second value quantization representation corresponding to the second point cloud; as well as The target 3D mesh generation module is configured to generate a target 3D mesh for the target object based on the first key quantization representation, the first value quantization representation, the second key quantization representation, and the second value quantization representation.

14. An electronic device, comprising: At least one processor; as well as A storage device for storing at least one program, which, when executed by the at least one processor, causes the at least one processor to implement the method according to any one of claims 1-12.

15. A computer-readable storage medium having a computer program stored thereon, the computer program implementing the method according to any one of claims 1-12 when executed by a processor.

16. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-12.

Images