Three-dimensional data generation method and electronic equipment

Abstract

The invention discloses a three-dimensional data generation method and electronic equipment, and relates to the technical field of artificial intelligence. The method comprises the following steps: determining geometric feature representation information of a matched three-dimensional model according to demand information of the three-dimensional model; mapping the geometric feature representation information into multi-channel texture distribution information and texture position information corresponding to each texture channel through a feature conversion corresponding relation, and fusing each texture position information based on the multi-channel texture distribution information to generate two-dimensional texture mapping data; removing texture features of the two-dimensional texture mapping graph to obtain a geometric image; and generating three-dimensional model representation data according to the two-dimensional texture mapping graph, the geometric image and the three-dimensional model demand information. According to the invention, the problem of poor three-dimensional data generation quality caused by dependence on a high-quality training sample in the prior art can be solved, and generation of high-quality three-dimensional data can be realized by using large-scale low-quality data.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a three-dimensional data generation method and electronic device. Background Technology

[0002] The task of 3D data generation is to generate data that matches the input text information or at least one image and is used to represent the 3D shape. Related techniques achieve 3D shape representation through compressed encoding or structured representation, which requires high-quality raw 3D data. The limited size of training sample data leads to poor quality of the final generated 3D data. Summary of the Invention

[0003] This invention provides a method and electronic device for generating three-dimensional data, which can generate high-quality three-dimensional data using large-scale low-quality data.

[0004] To solve the above-mentioned technical problems, the present invention provides the following technical solution: This invention provides a method for generating three-dimensional data, comprising: Based on the 3D model requirements, determine the matching geometric feature representation information of the 3D model; map the geometric feature representation information to multi-channel texture distribution information and texture position information corresponding to each texture channel through feature transformation correspondence; and fuse the texture position information based on the multi-channel texture distribution information to generate 2D texture mapping data; remove the texture features from the 2D texture mapping map to obtain the geometric image; and generate 3D model representation data based on the 2D texture mapping map, the geometric image, and the 3D model requirements; wherein, the feature transformation correspondence is the correspondence between the geometric features of the pre-constructed 3D sample model and the 2D texture features.

[0005] The present invention also provides an electronic device, including a memory and a processor, wherein the processor is used to implement the steps of the above-described three-dimensional data generation method when executing a computer program stored in the memory.

[0006] The advantages of the technical solution provided by this invention are as follows: First, by determining the geometric feature representation information of the matching 3D model through the 3D model requirement information, the generation process of 3D data used to represent the 3D model has clear input condition guidance, enabling controllable and directional 3D content generation. Second, the geometric feature representation information is mapped to multi-channel texture distribution information and texture position information corresponding to each texture channel through feature transformation correspondence, and further fused to generate 2D texture mapping data. This process is based on the pre-learned geometric-texture correspondence of the 3D sample model, realizing the decoupled representation and efficient conversion of geometric structure and texture information. In terms of shape representation, it does not rely on compression or structured representation, so there are no high requirements for model quality. It can be learned from massive amounts of Internet data, which not only avoids the empty space caused by structured representation or compressed encoding in related technologies. This invention addresses the issues of prior loss or low encoding efficiency by supporting the learning of general mapping rules from large-scale, diverse, and low-quality 3D samples, thereby improving the model's data utilization and representation robustness. By removing texture features from the 2D texture map to obtain the geometric image, it achieves the separation of texture and geometric information, providing an independent and aligned feature foundation for subsequent multimodal collaborative generation. Finally, by integrating texture, geometry, and high-level semantic requirements, it completes the integrated reconstruction from 2D parametric representation to 3D structure, ensuring that the generated 3D model is highly consistent with the input requirements in terms of visual detail and geometric structure. Simultaneously, the overall process is simple, eliminating the need for complex two-stage decoupling processing, significantly improving the efficiency of 3D content generation and system scalability while maintaining generation quality. This results in an end-to-end, scalable 3D generation method adaptable to large-scale data training. Furthermore, this invention also provides corresponding electronic devices with advantages for the 3D data generation method. Attached Figure Description

[0007] To more clearly illustrate the technical solutions of the present invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0008] Figure 1 This is a schematic diagram of the process for generating a 3D model based on vector set representation.

[0009] Figure 2 This is a schematic diagram of the process for generating a 3D model based on sparse volume representation.

[0010] Figure 3 A schematic diagram of the hardware framework applicable to the three-dimensional data generation method provided by the present invention.

[0011] Figure 4 This is a flowchart illustrating a three-dimensional data generation method provided by the present invention.

[0012] Figure 5 This is a schematic diagram of the deformation network provided by the present invention in an exemplary application scenario.

[0013] Figure 6 This is a schematic diagram of the first inverse mapping network provided by the present invention in an exemplary application scenario.

[0014] Figure 7 This is a schematic diagram of the edge optimization network provided by the present invention in an exemplary application scenario.

[0015] Figure 8 This is a schematic diagram of the allocation network provided by the present invention in an exemplary application scenario.

[0016] Figure 9 This is a schematic diagram of the training process for the feature transformation model provided by the present invention.

[0017] Figure 10 This is a schematic diagram of the structure of the three-dimensional data generation model provided by the present invention in an exemplary application scenario.

[0018] Figure 11 This is a schematic diagram of the structure of a network layer of the three-dimensional data generation model provided by the present invention in an exemplary application scenario.

[0019] Figure 12 This is a schematic diagram of a three-dimensional data editing process in an exemplary editing application scenario provided by the present invention.

[0020] Figure 13 A schematic diagram of the structure of the three-dimensional data editing model provided by the present invention in an exemplary application scenario. Detailed Implementation

[0021] To enable those skilled in the art to better understand the technical solutions of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. In this specification and the aforementioned drawings, the terms "first," "second," "third," "fourth," etc., are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. The term "exemplary" means "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as superior to or better than other embodiments.

[0022] 3D model generation, as a component of AIGC (Artificial Intelligence Generated Content), can directly generate relevant 3D model representation data that characterizes a 3D shape based on input text descriptions or images. It is widely used in creative content production, games, film and television, animation, virtual reality, and advertising.

[0023] Related technologies represent and generate 3D shapes using compressed coding representations (such as vector sets) or structured / sparse volumetric representations. However, these methods suffer from problems such as loss of spatial prior information, low coding efficiency, and reliance on high-quality 3D data. Since these technologies compress or sample the original 3D object, high-quality 3D models are required for training, resulting in relatively small training datasets (typically less than 1 million). This prevents the use of large amounts of data in the pre-training phase of image or video generation, hindering the full utilization of the vast amounts of data available on the internet and limiting the training scale of the network models used to generate 3D models. Furthermore, the Scaling Law (the correspondence between data and model parameters during large-scale model training) based on large-scale model training cannot be extended to larger scales with limited data, making it difficult to follow the Scaling Law for larger-scale model and data expansion. Additionally, these technologies employ a decoupled two-stage process of "shape generation" and "texture generation," leading to complex processing, low training efficiency, and system complexity. For example, ... Figure 1 As shown, vector set representation methods encode 3D shapes as 1D vector sequences. While this approach offers high encoding efficiency, the compressed encoding of the 3D structure introduces spatial prior loss, affecting the geometric fidelity of the generated model. Furthermore, it demands high-quality training data, typically only applicable to high-quality datasets with millions of records, failing to effectively utilize the much larger but inconsistent quality of 3D data available on the internet. Figure 2 As shown, methods using structured representation or sparse volume representation represent 3D shapes by preserving 3D voxels or sparse structure, such as Trellis (a text / image shape generation model). While this method does not lose spatial priors and can maintain good spatial priors, it suffers from low encoding efficiency and high storage and computational overhead. It is also limited by high-quality training data and has a complex multi-stage processing flow, making it difficult to achieve efficient end-to-end generation.

[0024] In view of this, to address the limitations of related technologies, such as reliance on high-quality data leading to a limited scale of trainable data and the inability to significantly improve model capabilities by following the Scaling Law; furthermore, the balance between representation efficiency and fidelity—compressed representations lose geometric details, while structured representations are inefficient; and the increased system complexity and training cost due to decoupled shape and texture generation—this invention addresses the issue of pre-learned geometry-texture correspondences. It maps geometric feature representations to multi-channel texture distribution information and texture position information corresponding to each texture channel, and further fuses these to generate two-dimensional texture mapping data. This achieves decoupled representation and efficient conversion of geometric structure and texture information. Shape representation does not rely on compressed or structured representations, thus requiring less stringent model quality and allowing for learning from massive amounts of internet data. By removing texture features from the 2D texture map to obtain the geometric image, the separation of texture and geometric information is achieved. Finally, by integrating texture, geometry, and high-level semantic requirements, the integrated reconstruction from 2D parametric representation to 3D structure is completed. This makes the generated 3D model highly consistent with the input requirements in terms of visual details and geometric structure. At the same time, the overall process is simple and does not require complex two-stage decoupling processing. This 3D model generation method can effectively utilize large-scale low-quality data, maintain spatial representation capabilities, and has a simple process. It realizes that the 3D generation model can adopt the Scaling Law to train a larger model with more data, just like in the fields of video generation or image generation. At the same time, it can have both spatial prior representation capabilities and simple processing flow.

[0025] The specific application environment architecture or hardware architecture upon which the execution of the 3D data generation method depends is described here, such as... Figure 3 As shown, the hardware architecture applicable to this embodiment can be, for example, a cloud server-client architecture. The cloud server 31 may include multiple computing nodes, equipped with a central processing unit (CPU) with 8 or more cores and a high-speed cloud hard disk as a storage device, and is also configured with a public IP address and elastic bandwidth (≥100Mbps) to ensure stable and efficient data transmission with the client 32. Each computing node is a cloud server instance with a high-performance computing card (such as a graphics processing unit), which is used to receive the 3D model generation request from the client 32 and perform inference tasks on the 3D generated model (approximately 10-20GB of parameters). The CPU can be used to handle tasks such as request scheduling and data preprocessing. The high-speed cloud hard disk (such as a solid-state drive) is used to store pre-trained model files, temporary generated data, and user session information. The client 32 can be a high-performance desktop computer or mobile workstation. The client must be configured with at least an 8-core, 16-thread CPU, a graphics processing unit with at least 12GB of video memory and support for deep learning inference frameworks, and may also include input / output devices such as a display. The following are some possible application scenarios related to the technical solution of this invention, which may include the following: The user inputs a 3D model generation request through the client software 32. This request is a text description containing information about the 3D model generation request, such as "a server with a textured shell." The client software 32 encrypts this text description information via HTTPS (Hypertext Transfer Protocol Secure) and sends it to the API (Application Programming Interface) gateway of the cloud server 31. After receiving the request, the cloud server 31 schedules it to a high-performance computing card on an idle computing node. The computing node loads the pre-trained text into the 3D generation model. This model includes a text encoder that converts the text into semantic vectors, a UV mapping generation model for generating corresponding 2D texture maps and geometric images based on the semantic vectors, and 3D data to represent the 3D model based on the 2D texture maps, geometric images, and 3D model requirements. The computing node then compresses the generated 3D data, such as 3D mesh data, and returns it to the client 32. Client 32 receives the compressed data packet returned from the cloud, decompresses it locally and loads it into memory, and uses the local GPU for graphics rendering: it parses the vertex, normal, and UV coordinates of the 3D mesh, loads the generated texture image, renders the server in real time in the virtual scene, and outputs the rendered image directly to the monitor for display.

[0026] In this scenario, this embodiment can leverage general knowledge learned from massive amounts of publicly available 3D data to quickly respond to abstract textual descriptions and generate high-quality 3D models. It should be noted that the above application scenarios are only shown to facilitate understanding of the ideas and principles of this invention, and the implementation of this invention is not limited in any way. On the contrary, the implementation of this invention can be applied to any applicable scenario. After introducing the technical solution of this invention, various non-limiting embodiments of this invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] Please see first. Figure 4 According to the three-dimensional data generation method provided by the present invention, it can be implemented as a computer program product, installed and running on servers such as AI content generation cloud server clusters, plugins or standalone applications of professional digital content creation software, mobile devices (phones, tablets), augmented reality / virtual reality headsets, in-vehicle entertainment system devices, etc., to respond to user-issued three-dimensional model generation instructions and generate corresponding three-dimensional model representation data. In some embodiments of the method, the method includes the following steps: S401: Determine the geometric feature representation information of the matching 3D model based on the 3D model requirement information.

[0028] The 3D model requirement information refers to the input information used to clarify the target of the 3D model generation. This can be a text description (e.g., "a data center"). If directly input into a neural network model, a text encoder is needed to convert the text description into corresponding text vectors. It can also be one or more reference images (two-dimensional images of the data center), requiring an image encoder. Alternatively, it can be audio or video data, in which case an audio / video data encoder needs to be configured. If the interface model does not support this data format, a network model for converting audio / video data to text data also needs to be configured. Geometric feature representation information is a digital abstract description of the spatial shape and structure of the final generated 3D model. Since it is input to the 3D model requirement information, similar 3D models can be first retrieved from a local database or publicly available data via the network based on the 3D model requirement information. Then, geometric feature representation information is obtained by sampling relevant data from that 3D model, such as collecting 3D point cloud data. For example, geometric feature representation information can be formed by splicing the vertex coordinates and standardized vertex normals of the 3D model along the feature channel dimension, reflecting the spatial position and surface orientation attributes of the model. The 3D model in this step is not a real 3D model, but rather a rough determination of the geometric shape and structure of the 3D model to be generated based on the 3D model requirements information, in order to obtain matching geometric feature representation information.

[0029] S402: The geometric feature representation information is mapped into multi-channel texture distribution information and texture position information corresponding to each texture channel through feature transformation correspondence. Based on the multi-channel texture distribution information, the texture position information is fused to generate two-dimensional texture mapping data.

[0030] Feature transformation correspondence refers to the correspondence between the geometric features of a pre-constructed 3D sample model and the 2D texture features. It establishes a stable mathematical model or mapping rule between 3D geometric features and 2D texture features (i.e., UV mapping), which can be obtained through methods such as UV mapping, spherical mapping and cube mapping, box mapping and parametric mapping, feature matching and free mapping, hardware acceleration, and optimization techniques. The UV mapping method unfolds the surface of the 3D model into a 2D parametric space (UV space) and assigns texture coordinates (u, v) to each vertex. For example, the UV coordinates of a cube can be defined as (0, 0), (1, 0), (1, 1), (0, 1), corresponding to the corner points of the texture image, achieving precise fitting. This method is suitable for planar or developable surface models and is often automatically calculated using UV unfolding algorithms to reduce stretching. Spherical mapping and cube mapping methods: Spherical mapping projects textures onto a spherical or partially spherical surface, suitable for spherical objects (such as the Earth or spherical models), generating texture coordinates through spherical coordinate transformation; cube mapping uses six planar textures to form a cube environment map, often used for reflection effects or panoramic textures, finding the corresponding texture pixels through direction vectors. Box mapping and parametric mapping methods: Box mapping directly maps textures to the surface of an axis-aligned bounding box, suitable for simple geometry (such as a cube), converting 3D coordinates to texture coordinates through linear transformation; parametric mapping maps 3D points to UV space through mathematical functions (such as spherical coordinate transformation). Feature matching and free mapping are for complex, irregular models, using feature detection algorithms (such as scale-invariant feature transformation methods or spatial pyramid algorithms) to match image feature points, and combining marker point calibration techniques to generate mapping rules. Hardware acceleration and optimization techniques refer to managing texture objects through graphics APIs and using GPU parallel processing to improve efficiency; using techniques such as Mipmapping (texture mapping optimization) to generate multi-resolution textures to reduce rendering overhead and ensure real-time performance. This invention does not impose any limitations on this; it only requires obtaining the feature transformation correspondence. These methods do not have high requirements for data quality and can learn the mapping relationship between the two using massive amounts of publicly available data. For example, the feature transformation correspondence can be a mathematical model or mapping rule that can be learned from a large number of publicly available 3D models and can establish a stable association between 3D geometric features and 2D texture features (i.e., UV mapping).

[0031] Multi-channel texture distribution information is based on multi-channel geometric feature data, combined with the assignment probability of vertices and texture channels to determine the importance allocation of each texture channel. This information is used to accurately fuse texture information from each channel, representing the weight of texture allocation across multiple independent regions (channels). Texture location information is the coordinate distribution data of each texture channel in the two-dimensional texture space, defining the coordinates of each texture channel in the two-dimensional texture space (UV space), ensuring accurate correspondence between textures and the surface of the 3D model. Two-dimensional texture mapping data is a two-dimensional texture mapping map (i.e., UV map) generated by fusing multi-channel texture distribution information and texture location information. It is a two-dimensional carrier of the texture features of the 3D model. The UV map is a set of data defining texture mapping relationships; essentially, it is the two-dimensional coordinate system of the model, independent of the model's three-dimensional position. The values ​​of U and V are usually between [0, 1] (sometimes other ranges are also possible). (0, 0) represents the lower left corner of the texture map, (1, 1) represents the upper right corner, and the texture map is a square image from (0, 0) to (1, 1). The process of "flattening" (unfolding) the 3D surface of a model into UV islands on a 2D plane and defining the corresponding position (U, V coordinates) of each vertex in the texture mapping coordinate system is called UV mapping, and the resulting 2D matrix is ​​called the UV map. Here, a UV vertex refers to each vertex of the model being assigned a (U, V) coordinate. The model surface is composed of polygons (usually triangular faces), and the (U, V) coordinates corresponding to the three vertices of each face define which triangular image region should be captured on the texture map. A UV island refers to an independent segment formed by unfolding different parts of the model in UV space; it is a collection of connected faces.

[0032] S403: Remove the texture features from the two-dimensional texture map to obtain a geometric image.

[0033] The geometric image is the geometric structure information of the 3D model. Visual features such as texture color and pattern are removed from the 2D texture mapping data, retaining only the defined geometric mapping relationships (such as the correspondence between vertices and UV coordinates), thus obtaining a geometric image. This image essentially describes the unfolded shape of the 3D surface. For example, when the above steps yield a 2D texture mapping map, the multi-layer UV maps of each data point can be fused using any method to obtain a fused texture map, while simultaneously removing the texture to obtain the corresponding fused geometric map. For the fused texture map, model weights from the image generation domain can be used for initialization and adaptation. Furthermore, since there is a certain degree of mapping between the fused texture map and the fused geometric map, the encoded features in the fused texture map can guide the learning of the features in the fused geometric map.

[0034] S404: Generate 3D model representation data based on 2D texture mapping, geometric images, and 3D model requirements.

[0035] The 3D model representation data refers to the data representing the shape of a 3D object. For example, the 3D model representation data can be a 3D mesh file, which contains the surface geometry and topology of the 3D object. This step synthesizes the 3D model representation data, such as a complete triangular mesh file, based on the 2D texture mapping data (providing texture information), the geometric image (providing geometric structure information), and the initial input 3D model requirement information (providing high-level semantic guidance).

[0036] In the technical solution provided in this embodiment, the geometric feature representation information of the matching 3D model is determined through the 3D model requirement information, so that the generation process of 3D data used to represent the 3D model has clear input condition guidance, enabling controllable and directional 3D content generation. Secondly, the geometric feature representation information is mapped to multi-channel texture distribution information and texture position information corresponding to each texture channel through feature transformation correspondence, and further fused to generate 2D texture mapping data. This process is based on the pre-constructed geometry-texture correspondence, realizing the decoupled representation and efficient conversion of geometric structure and texture information. In terms of shape representation, it does not rely on compression or structured representation, so there is no great requirement for data quality. It can use massive amounts of Internet data for learning, which not only avoids the spatial prior loss caused by structured representation or compressed encoding in related technologies, but also... To address the issue of low encoding efficiency, this method also supports learning general mapping rules from large-scale, diverse, and low-quality 3D samples, thereby improving the model's data utilization and representation robustness. By removing texture features from the 2D texture map to obtain geometric images, it achieves the separation of texture and geometric information, providing an independent and aligned feature foundation for subsequent multimodal collaborative generation. Finally, by integrating texture, geometry, and high-level semantic requirements, it completes the integrated reconstruction from 2D parametric representation to 3D structure, making the generated 3D model highly consistent with the input requirements in terms of visual details and geometric structure. At the same time, the overall process is simple and does not require complex two-stage decoupling processing. While maintaining the generation quality, it significantly improves the efficiency of 3D content generation and system scalability, thus forming an end-to-end, scalable 3D generation method that can adapt to large-scale data training.

[0037] Considering that poor mesh compatibility, uneven texture distribution, and blurred edge details during texture mapping data generation can lead to low texture fit and poor visual effects in the final 3D model, this invention provides an exemplary implementation method for generating two-dimensional texture mapping data through feature transformation correspondence, which may include the following steps: Based on the 3D model requirements, a matching original 3D model is determined, and 3D data points are sampled on the original 3D model. Initialized texture map mesh data is generated based on geometric feature representation information. Adaptive deformation processing is performed on the initial texture map mesh data to generate deformation-optimized data that matches the surface geometry of the original 3D model, while maintaining the consistency of the positional information dimension. The deformation-optimized data is reconstructed into 3D geometric feature representation data with spatial position and surface orientation information. The edge regions of the original 3D model corresponding to the 3D geometric feature representation data are subjected to detail optimization processing and feature dimension expansion to obtain multi-channel geometric feature data. Based on the belonging probability between vertices and texture channels, the multi-channel texture distribution information of the multi-channel geometric feature data is determined, and the multi-channel geometric feature data is inversely mapped to the texture position information of each texture channel.

[0038] The process involves several key components: the original 3D model (matched to the required 3D model information) serves as the initial reference for generating the target 3D data; the 3D data points (sampled from the original 3D model to represent key locations within the model, each containing 3D spatial coordinates, such as 3D point cloud data, sampled using uniform sampling); the initial texture map mesh data (a 2D normalized coordinate mesh constructed from the number of vertices sampled from the original 3D model, presented as a 3D tensor, forming the initial basis for texture mapping); adaptive deformation processing (adjusting the initial texture map mesh data using a multilayer perceptron network to precisely match the surface geometry of the 3D model); deformation optimization data (mesh data obtained after adaptive deformation processing, maintaining the same positional information dimension as the initial mesh and adapting to the 3D model's surface structure); 3D geometric feature representation data (data reconstructed from the deformation optimization data, containing spatial location and surface orientation information, comprehensively reflecting the geometric attributes of the 3D model); and edge region detail optimization processing (refining the model edge regions corresponding to the 3D geometric feature representation data, improving the accuracy of the model edges). Feature dimensionality expansion is the process of increasing the dimensionality of optimized 3D geometric feature data to generate multi-channel geometric feature data. Multi-channel geometric feature data, obtained after feature dimensionality expansion, provides the foundation for generating multi-channel texture distribution and texture location information. Assignment probability is the association probability, or geometric association strength, between a vertex and each texture channel, used to determine the channel assignment of a vertex in texture space.

[0039] In this embodiment, a matching original 3D model file is determined based on the 3D model requirements, and a set of 3D data points are sampled from its surface. Simultaneously, an initial, regularized texture map mesh is generated based on geometric feature representation information. Next, adaptive deformation processing is applied to the initialized texture map mesh data to ensure that this regular mesh conforms to the surface geometry of the 3D model, generating deformation-optimized data (such as optimized mapping coordinates) that matches the surface geometry of the original 3D model. This ensures that the number and dimension of data points are consistent before and after processing, maintaining dimensional consistency of the coordinate set. Then, the deformed 2D mesh data is reconstructed into 3D geometric feature representation data with 3D spatial position and surface normal direction information. In other words, the optimized 2D texture mapping coordinates are reconstructed into 3D geometric feature representations with spatial position and surface orientation information, achieving a reversible conversion from 2D coordinates to 3D geometry while maintaining the integrity of the feature representation. To improve the generation quality, detailed optimization processing is performed on key areas such as the edges and contours of the original 3D model corresponding to the 3D geometric feature representation data. For example, multi-level feature extraction and residual fusion mechanisms can be used to perform local detail optimization on the reconstructed 3D geometric features, focusing on enhancing the geometric accuracy of the model's edge regions while maintaining overall structural stability. The feature dimensions are then expanded to obtain multi-channel geometric feature data containing richer information. Finally, based on the probability that a vertex belongs to different texture channels (i.e., the attribution probability), the multi-channel texture distribution information of the multi-channel geometric feature data is determined. Simultaneously, the multi-channel geometric feature data is inversely mapped back to two-dimensional space to obtain the texture position information of each texture channel, i.e., the UV coordinate fragment corresponding to each channel.

[0040] As shown above, this embodiment initializes the texture map mesh data precisely based on the number of sampled vertices through an unsupervised process including sampling, deformation, reconstruction, optimization, and channel allocation, providing a solid foundation for subsequent texture mapping. Adaptive deformation processing ensures a high degree of matching between the mesh and the surface geometry of the 3D model, improving texture fit. Edge region detail optimization effectively improves the clarity and accuracy of model edges, reducing edge blurring. Multi-channel design makes the texture distribution more uniform and rich, providing high-quality texture data support for the generation of high-quality 3D models. It realizes automatic conversion from 3D point cloud to structured UV mapping, can handle 3D models of various shapes, and generates a reasonable UV map layout with clear island areas, providing a high-quality, standardized intermediate representation for subsequent texture and geometry generation, which is conducive to the effective utilization of large-scale data.

[0041] Furthermore, considering that poor fit between the size and the original 3D model during the initialization of the texture map mesh can lead to subsequent texture mapping distortion and chaotic texture distribution, this embodiment also provides an implementation method for creating a matching, regular 2D parameterized starting point for a 3D point cloud with arbitrary sampling density. The process of generating the initialization texture map mesh data may include the following: Based on the 3D model requirements, a matching original 3D model is determined. The height and width information are determined based on the number of vertices sampled in a single run of the original 3D model. The height information determines the vertical direction, and the width information determines the horizontal direction. In a pre-defined standardized parameter space, the original 3D model is sampled at equal intervals along both the vertical and horizontal directions, and the corresponding 2D texture coordinates of the 3D data points are generated using Cartesian product. The matrix row and column parameters are determined according to the height and width information, and the 2D texture coordinates of each 3D sampling point are used as matrix elements to generate the sampling matrix of the original 3D model. Based on the number of original 3D models processed each time, the sampling matrix of each original 3D model, and the dimensions of the matrix elements, 3D tensor data is generated to serve as the initial texture map mesh data.

[0042] The height information is the vertical dimension parameter of the initialized texture map mesh, determined by the number of vertices sampled in a single run of the original 3D model, and is equal to the width information. The width information is the horizontal dimension parameter of the initialized texture map mesh, which, together with the height information, determines the matrix row and column parameters of the mesh. The normalized parameter space is a parameter space with a fixed range, such as [-1, 1] × [-1, 1], providing a unified reference range for the generation of mesh coordinates. Equal-interval sampling ensures the uniformity of sampling by selecting sampling points at equal intervals in the vertical and horizontal directions of the normalized parameter space. Two-dimensional texture coordinates are coordinates (u, v) used to characterize the position of 3D data points in the two-dimensional texture space, where u is the horizontal coordinate and v is the vertical coordinate. The sampling matrix is ​​a matrix constructed using the two-dimensional texture coordinates of each 3D sampling point as matrix elements, and the row and column parameters are determined based on the height and width information, reflecting the correspondence between 3D data points and two-dimensional texture coordinates. The 3D tensor data is data formed by fusing the number of original 3D models processed each time, the sampling matrices of each model, and the dimensions of the matrix elements; it is the final presentation form of the initialized texture map mesh data.

[0043] In this embodiment, firstly, the square root of the number of vertices N (e.g., 2500) of the original 3D model in a single sampling is calculated and used as the height information H and width information W, respectively, for example, H=W=50. Within a preset standardized 2D parameter space (e.g., coordinate range [-1, 1]×[-1, 1]), samples are taken at equal intervals along the vertical direction (corresponding to the V-axis) and the horizontal direction (corresponding to the U-axis), respectively, according to the height H and width W, to obtain H V coordinates and W U coordinates. Through Cartesian product, all possible (U, V) coordinate combinations are generated, totaling H*W=N points. Then, with the height H as the number of rows and the width W as the number of columns, the 2D texture coordinates of each sampling point are used as matrix elements and arranged to form a sampling matrix. Finally, according to the number B of 3D models processed in each batch, B such sampling matrices are stacked, and considering that each coordinate point is 2D, a 3D tensor data of shape (B, N, 2) is finally generated, which is the initial texture map mesh data G. Where B is the batch size, N is the number of sampling points, and 2 represents the two coordinate values ​​U and V.

[0044] In this embodiment, by constructing a regular two-dimensional mesh that corresponds one-to-one with the three-dimensional sampling points, the consistency of the input format for each processing step is ensured. The mesh size is precisely determined by the number of sampled vertices from the original three-dimensional model, ensuring mesh-model compatibility. Standardized parameter space and equidistant sampling methods guarantee a uniform distribution of two-dimensional texture coordinates, laying the foundation for uniformity in subsequent texture mapping. The three-dimensional tensor data structure facilitates subsequent processing by the network model, improving data processing efficiency. It provides a stable and structured starting point for subsequent neural network learning, enabling the deformation network to focus on learning the mapping relationship from the regular mesh to the unfolded graph of a specific model.

[0045] Furthermore, considering that insufficient alignment between the initial texture map mesh data and the surface geometry of the 3D model can lead to problems such as texture stretching, distortion, and misalignment after texture mapping, this embodiment also provides a method for automatically deforming the initial regular UV mesh to match complex 3D surface shapes, which may include the following: The initial texture map mesh data is input into the first fully connected layer to obtain the first deformation feature, and the first deformation feature is concatenated with the initial texture map mesh data; the deformation concatenation feature is input into the second fully connected layer to obtain the second deformation feature; the second deformation feature is added element-wise with the initial texture map mesh data to obtain the deformation optimization data.

[0046] The first fully connected layer performs preliminary feature extraction and transformation on the initialized texture map mesh data. The first deformation feature, obtained after processing by the first fully connected layer, represents the initial deformation of the initialized texture map mesh data. The deformation splicing feature is a fused feature obtained by channel splicing the first deformation feature with the initialized texture map mesh data, possessing both original mesh information and preliminary deformation features. The second fully connected layer further processes the deformation splicing feature to generate the final deformation feature. The second deformation feature, obtained after processing by the second fully connected layer, represents the final deformation of the mesh data. Element-by-element addition refers to adding the corresponding elements of the second deformation feature with the initial texture map mesh data, achieving the fusion of the deformation feature and the original mesh data.

[0047] In this embodiment, the adaptive deformation processing can be implemented through a deformation network, such as... Figure 5 As shown, it can be represented as The network first inputs the initial texture map mesh data G into the first fully connected layer F1 to extract the first deformation feature. Then, the first deformation feature is concatenated with the original initial texture map mesh data G along the feature dimension. The concatenated feature is then input into the second fully connected layer F2 to obtain the second deformation feature. Finally, using a residual connection, the second deformation feature is added element-wise to the initial texture map mesh data G to obtain the optimized deformation data Gd. The vector dimension remains (B, H*W, 2).

[0048] As shown above, this embodiment achieves precise deformation adjustment of mesh data through progressive processing with two fully connected layers, improving the fit between the mesh and the 3D model surface. The construction of deformation stitching features ensures the effective preservation of the original mesh information, avoiding data loss during deformation. The element-wise addition fusion method allows for a natural transition in deformation effects, effectively avoiding texture stretching and distortion, and improving texture mapping quality. The entire deformation network can learn complex nonlinear deformations, while the residual structure ensures training stability, preventing gradient vanishing or exploding, enabling the network to effectively adjust regular meshes into optimized UV layouts that fit the 3D model surface without overlap.

[0049] Considering that deformation optimization data cannot directly reflect the spatial position and surface orientation information of the 3D model, which can lead to incomplete and inaccurate subsequent geometric feature extraction, this embodiment also provides a process for recovering data with 3D spatial sense and surface orientation information from the optimized 2D UV coordinates, which may include the following: The deformation optimization data is input into the third fully connected layer to obtain the reconstructed features; the reconstructed features and the deformation optimization data are spliced ​​together, and the reconstructed spliced ​​features are input into the fourth fully connected layer; the output of the fourth fully connected layer is used as the three-dimensional geometric feature representation data.

[0050] The third fully connected layer performs preliminary reconstruction processing on the deformation optimization data. The reconstructed features, obtained after processing by the third fully connected layer, serve as an intermediate carrier for the transformation of deformation optimization data into 3D geometric features. The reconstructed stitched features are fused features obtained by channel stitching of the reconstructed features and deformation optimization data, providing comprehensive information for the subsequent generation of 3D geometric features. The fourth fully connected layer performs final processing on the reconstructed stitched features, outputting 3D geometric feature representation data.

[0051] In this embodiment, the reversible transformation from two-dimensional coordinates to three-dimensional geometry can be achieved through an inverse mapping network. For ease of distinction, it can be defined as the first inverse mapping network, such as... Figure 6 As shown, it can be represented as The deformation optimization data Gd is input into the third fully connected layer A1 to obtain the preliminary reconstructed features. Then, this reconstructed feature is concatenated with Gd, and the concatenated feature is input into the fourth fully connected layer A2. The output of the A2 layer is the reconstructed three-dimensional geometric feature representation data Pd, which has the shape (B, N, 6). In the last 6 dimensions, the first three dimensions represent the three-dimensional spatial position of the point, and the last three dimensions represent the surface normal vector at the point.

[0052] As shown above, this embodiment achieves accurate conversion from 2D deformed mesh data to 3D geometric feature data through reconstruction processing using two fully connected layers. The construction of reconstructed stitched features ensures the integrity of the data information, providing a guarantee for the comprehensive extraction of 3D geometric features. The generated 3D geometric feature representation data simultaneously contains spatial location and surface orientation information, providing a comprehensive geometric foundation for subsequent model optimization and texture mapping. By performing inverse conversion from 2D UV space to 3D geometric space through an inverse mapping network, the prediction of UV coordinates to 3D point positions is realized, enabling the system to form a complete process from UV deformation to 3D reconstruction. This provides the possibility for calculating reconstruction loss and driving unsupervised training, ensuring that the learned UV mapping has accurate geometric correspondence.

[0053] Furthermore, considering that if the edge details of the 3D geometric feature data are blurry or the feature dimensions are insufficient, it will lead to rough edges and poor texture representation in the subsequently generated 3D model. This embodiment also provides solutions to the problem that the reconstructed 3D point cloud may not be accurate enough in details such as edges, and how to increase the expressive power of single-point features to support multi-channel representation. These solutions may include the following: The 3D geometric feature representation data is input into the first stacked convolutional layer to obtain the first detail-optimized feature, and then concatenated with the 3D geometric feature representation data. The concatenated detail-optimized feature is input into the second stacked convolutional layer to obtain the second detail-optimized feature. The second detail-optimized feature is then added element-wise with the 3D geometric feature representation data to obtain the 3D geometric feature-optimized data. The 3D geometric feature-optimized data is input into the feature upscaling network to obtain multi-channel geometric feature data. The first and second stacked convolutional layers include multiple feature extraction and fusion layers. Each feature extraction and fusion layer includes a convolutional layer and an activation function layer. The feature upscaling network includes two stacked convolutional layers.

[0054] The first stacked convolutional layer is used to initially extract edge detail features from the 3D geometric feature representation data. The first detail optimization feature, obtained after processing by the first stacked convolutional layer, represents the initial optimization of the 3D model's edges. The detail optimization stitched feature is a fused feature obtained by stitching the first detail optimization feature with the 3D geometric feature representation data, possessing both the original geometric features and the initial edge optimization features. The second stacked convolutional layer has the same structure as the first stacked convolutional layer and is used to further refine the edges of the detail optimization stitched feature. The second detail optimization feature, obtained after processing by the second stacked convolutional layer, represents the depth optimization of the 3D model's edges. The 3D geometric feature optimization data is obtained by adding the second detail optimization feature element-wise with the 3D geometric feature representation data, significantly improving the accuracy of edge details. The feature upscaling network can be a network structure composed of two stacked convolutional layers, used to expand the dimensionality of the 3D geometric feature optimization data. The feature extraction and fusion layer is the basic unit constituting the stacked convolutional layers, including convolutional layers and activation function layers, realizing feature extraction and fusion. Convolutional layers are network layers used to extract local features from data, achieving feature capture through convolutional kernel operations. Activation function layers are network layers used to introduce non-linear transformations, enhancing the network's ability to express features; for example, the ReLU (Linear Rectification Function) activation function can be used.

[0055] In this embodiment, the optimized 3D geometric feature data can be achieved through an edge optimization network, such as... Figure 7As shown, the 3D geometric feature representation data Pd is input into the first stacked convolutional layer M1. This stacked convolutional layer consists of multiple (e.g., three) convolutional layers and ReLU activation function layers stacked sequentially, used to extract local detail features. The extracted detail features are concatenated with the input Pd and then input into the second stacked convolutional layer M2 for further processing. Residual connections are also used, and the output of the second stacked convolutional layer is added to Pd to obtain more detailed 3D geometric feature optimized data Pe. Pe is then input into a feature upscaling network consisting of two stacked convolutional layers. The purpose of this upscaling network is to expand the number of feature channels for each data point, transforming the feature of each point from a low-dimensional representation into a higher-dimensional feature vector, thus obtaining multi-channel geometric feature data Pup. Pup contains richer and more discriminative information, laying the foundation for subsequent channel allocation.

[0056] As shown above, this embodiment achieves deep optimization of the edge details of the 3D model through stacked convolutional layers, significantly improving the clarity and accuracy of the edges. The feature upscaling network effectively expands the feature dimension of the data, and the generated multi-channel geometric feature data provides rich feature support for subsequent multi-channel texture generation. The two concatenation and element-wise addition operations ensure the effective fusion of the original features and optimized features, avoiding feature loss and improving data integrity. It significantly improves the quality of the reconstructed 3D point cloud in terms of contours and details, and through feature upscaling, each point can carry richer information, making it easier for the model to learn to decompose it into different semantically meaningful texture channels.

[0057] Furthermore, considering that unreasonable channel importance allocation and inaccurate texture position mapping during multi-channel texture generation can lead to chaotic texture distribution and poor coordination among textures in different channels, this embodiment also provides a method for reasonably dividing a 3D model surface into multiple texture regions (channels) and generating independent UV coordinates for each region, which may include the following: Multi-channel geometric feature data is input into the fifth fully connected layer, and the output of the fifth fully connected layer is processed using an activation function to obtain the importance weight values ​​of each channel of the multi-channel geometric feature data. Multi-channel geometric feature data is input into the sixth fully connected layer, and the output of the sixth fully connected layer is used as the position information of each channel in the texture space.

[0058] The fifth fully connected layer processes the multi-channel geometric feature data, extracting channel importance-related features. The activation function is used to perform a non-linear transformation on the output of the fifth fully connected layer; for example, the softmax function can be used to ensure the output conforms to a probability distribution. The importance assignment weights are the channel importance scores obtained after processing by the activation function, reflecting the priority of each texture channel in texture generation. The sixth fully connected layer maps the multi-channel geometric feature data to positional information in texture space. Texture space is the space where the two-dimensional texture mapping resides, used to define the coordinate distribution of the texture.

[0059] In this embodiment, the process consists of two steps. The first step is channel allocation: the multi-channel geometric feature data Pup is input into an allocation network consisting of a fifth fully connected layer G1 and a Softmax activation function, such as... Figure 8 As shown, it can be represented as The network calculates a set of weights for each feature point, namely, a weight value Sup representing the importance of each channel. This set of weights indicates the probability that the point belongs to each texture channel, and the sum of the weights for all channels is 1. The second step is channel-level inverse mapping: Pup is input to the sixth fully connected layer, which can be used as the first projection network. The output is the two-dimensional texture coordinates of each point under different channel assumptions, that is, the position information of each channel in texture space. It can be understood that the network generates a complete set of UV coordinates for each possible channel.

[0060] In this embodiment, adaptive model surface partitioning is achieved by introducing channel allocation weights and multi-channel mapping. The activation function ensures that the importance allocation weights conform to the probability distribution, achieving a reasonable allocation of importance for each channel and improving the coordination of texture generation. A fully connected layer enables precise mapping of multi-channel geometric features to texture position information, ensuring the accurate positioning of each channel texture in two-dimensional space and avoiding texture misalignment. The generated multi-channel texture distribution and texture position information provide crucial support for the generation of high-quality two-dimensional texture mapping data. This allows a 3D model to be automatically split into multiple UV islands, each of which can be independently unfolded and optimized, avoiding the severe distortion inevitably caused by a single UV map of a complex model. This results in multi-channel UV mapping with better layout and less stretching, greatly improving the accuracy and visual quality of texture mapping.

[0061] Furthermore, considering that the geometric feature extraction process is incomplete and cannot accurately reflect the geometric properties of the 3D model, this embodiment also provides an implementation process for converting a standard 3D mesh file into a feature-based input that can be processed by a network, which may include the following: Based on the 3D model requirements, a matching original 3D model is determined, and its vertex coordinates and face index information are obtained. Based on the vertex coordinates and face index information, vertex normals are determined and standardized to obtain normal vectors. The normal vectors and vertex coordinates are concatenated along the feature channel dimension, and the concatenated vector serves as the geometric feature representation information corresponding to the 3D model requirements. Vertex coordinates are the coordinate data of the vertices on the surface of the original 3D model in 3D space, defining the spatial position of the vertices. Face index information identifies which vertices constitute each face in the original 3D model, reflecting the model's topology. The vertex normal is a vector perpendicular to the tangent plane of the vertex on the surface of the original 3D model, representing the surface orientation of the vertex. Standardization is a normalization operation on the vertex normals, ensuring the magnitude of the normal vector is 1, guaranteeing the consistency and normalization of the normal direction. The normal vector is the standardized vertex normal, used to concatenate with the vertex coordinates to form the geometric feature representation information. Feature channel concatenation merges the vertex coordinates and normal vectors along the feature channel dimension, forming fused data containing spatial position and surface orientation information. The concatenated vector is the vector data obtained by concatenating the feature channel dimensions, that is, the geometric feature representation information corresponding to the 3D model requirement information.

[0062] In this embodiment, firstly, the original triangular mesh file is matched according to the 3D model requirements. The vertex coordinates (defining the 3D position of each point) and face index information (defining which vertices connect to form triangular patches) of this file are read. Using the vertex coordinates and face indices, the vertex normal of each vertex is determined through geometric calculations (such as averaging the normals of adjacent faces) to describe the orientation of that point. These normals are standardized to become unit vectors, resulting in normal vectors. Finally, in the feature channel dimension, i.e., the last dimension, the normal vector (3D) of each vertex is concatenated with its vertex coordinates (3D) to form a 6D feature vector. The set of these 6D vectors from all vertices constitutes the geometric feature representation information corresponding to the 3D model.

[0063] As shown above, this embodiment provides a standard and efficient geometric information extraction process. The stitching method of feature channel dimensions effectively integrates spatial location and surface orientation information, providing a comprehensive and accurate geometric foundation for subsequent feature transformation and texture generation. By combining the shape (coordinates) and surface properties (normals) of the 3D model, the generated geometric feature representation information can accurately reflect the geometric attributes of the 3D model, providing comprehensive and uniformly formatted geometric input for subsequent processing. This enables the network to simultaneously understand spatial location and local geometric structure, improving the accuracy of subsequent 3D data generation.

[0064] Based on the above embodiments, this embodiment also provides an implementation process for autonomously learning the optimal mapping relationship through large-scale sample data, which can solve the problem of how to enable the network to automatically learn the bidirectional conversion between 3D models and UV mappings without any UV annotations, and may include the following: The feature transformation correspondence is a pre-trained feature transformation model, such as... Figure 9 As shown, the feature transformation model includes a deformation network, a first inverse mapping network (which can be represented as inverse mapping network A), an edge optimization network, a first projection network (which can be represented as projection network B), a feature upscaling network, and an assignment network during the inference phase. During the training phase, it also includes a second projection network (which can be represented as projection network A), a fusion network, and a second inverse mapping network (which can be represented as inverse mapping network B). The training process of the feature transformation model includes: inputting the initial texture map mesh data of the 3D sample model into the feature transformation model, which then passes through the deformation network, the first inverse mapping network, the edge optimization network, and the second projection network sequentially to obtain a 2D texture mapping sample map; and then processing the 3D geometric features output by the edge optimization network. The optimized sample data is input into the feature upscaling network, and the upscaling vectors are input into the assignment network and the first projection network respectively to obtain the importance weight scores of each channel and the multi-channel two-dimensional texture mapping sample map input into the second inverse mapping network. The fusion network merges the importance weight scores of each channel and the output of the second inverse mapping network into a three-dimensional sample optimization model. The loss function is determined based on the three-dimensional geometric feature representation sample data, deformation optimization sample data, two-dimensional texture mapping sample map, multi-channel two-dimensional texture mapping sample map, importance weight scores of each channel, three-dimensional sample optimization model, and three-dimensional point cloud sample data. The feature transformation model is continuously trained based on the loss function until the iteration stopping condition is met.

[0065] The feature transformation model is used to establish the correspondence between the geometric features of the 3D sample model and the 2D texture features. It includes a network structure for inference and training phases. The inference phase is where the model actually generates the feature transformation correspondence, and the training phase is where the model learns the feature transformation rules through sample data. The deformation network is a network that adaptively deforms the initialized texture map mesh data; it is a multilayer perceptron model, and the output vector dimension remains (B, H*W, 2). The inverse mapping network A reconstructs the deformed optimized data into 3D geometric feature representation data. The edge optimization network optimizes the edge details of the 3D geometric feature representation data. The projection network B (which can be a fully connected layer) projects the edge-optimized 3D data into 2D texture mapping data. The feature dimensionality enhancement network expands the feature dimension of the 3D geometric feature optimization data. The assignment network determines the weight values ​​for the importance of each channel of the multi-channel geometric feature data. The projection network A (which can be a fully connected layer) is the network used in the training phase to project the edge-optimized 3D sample data into a 2D texture mapping sample map. The fusion network is used during the training phase to fuse the importance-assigned weights of the output with the output of the second inverse mapping network to form a 3D sample optimization model. To distinguish it from the importance-assigned weights output during the inference phase, the importance-assigned weights of each channel output during the training phase are differentiated by channel importance weight scores. The inverse mapping network B is used during the training phase to map multi-channel 2D texture mapping sample images into 3D data. Initialized texture map mesh data: The initial texture map mesh data of the 3D sample model input to the feature transformation model during the training phase. 3D geometric feature optimization sample data: The geometric feature optimization data of the 3D sample model obtained after processing by the edge optimization network during the training phase. Feature upscaling vector: The multi-channel feature vector obtained after processing by the feature upscaling network during the training phase. 2D texture mapping sample image: The 2D texture mapping sample data obtained after processing by the second projection network during the training phase. Multi-channel 2D texture mapping sample image: The multi-channel 2D texture mapping sample data obtained after processing by the first projection network during the training phase. Importance weight scores for each channel: The importance-assigned weights obtained after processing by the assignment network during the training phase. 3D sample optimization model: The optimized 3D sample model obtained after fusion by the fusion network during the training phase. The loss function is used to measure the difference between the model's prediction results and the real sample data. The iteration stopping condition is the criterion for stopping the iteration during model training, which usually includes the loss function value reaching a preset threshold, the number of iterations reaching a preset upper limit, etc.

[0066] In this embodiment, the training process of the feature transformation model is a self-supervised process of bidirectional cyclic mapping, such as... Figure 9As shown, the initial texture map mesh data of the 3D sample model is input into the feature transformation model, and then successively passes through the deformation network for adaptive deformation processing, through the inverse mapping network A to reconstruct the 3D geometric feature representation sample data, through the edge optimization network to optimize the details of the edge region of the 3D sample model, and through the projection network A to project the 3D geometric feature optimized sample data into a 2D texture mapping sample map; the 3D geometric feature optimized sample data output by the edge optimization network is input into the feature upscaling network, and the feature upscaling vector is input into the assignment network and the projection network B respectively. The projection network B outputs a multi-channel 2D texture mapping sample map to the inverse mapping network A; the fusion network fuses the importance weight scores of each channel output by the assignment network and the multi-channel 3D model output by the second inverse mapping network to obtain the 3D sample optimization model.

[0067] As shown above, this embodiment utilizes an end-to-end unsupervised training method, enabling the model to autonomously learn the optimal mapping relationship between 3D geometric features and 2D texture features. This eliminates the need for manually designed rules and completely removes the reliance on manually labeled UV data. It allows training using a vast amount of unlabeled 3D models obtained from the internet, improving the accuracy and generalization ability of the mapping relationship. The newly added network structure during training enriches the model's learning dimensions, ensuring that the model can fully learn the mapping rules of multi-channel features. The construction of the loss function provides a clear optimization direction for model training, guaranteeing the effectiveness and stability of the training. The trained feature transformation model can efficiently and accurately convert between 3D geometric features and 2D texture features, providing reliable support for subsequent 3D data generation.

[0068] Furthermore, considering that the lack of a comprehensive and reasonable loss measurement standard during model training can lead to biased model learning direction, poor training results, and inability to generate accurate feature transformation correspondences, this invention also provides an exemplary loss function determination process to avoid such problems, which may include the following: Each point in the multi-channel 2D texture mapping sample image is identified as a neighboring point in the 3D sample model. Each group of neighboring points is then converted into neighboring point mapping information in texture space. The inverse mapping loss is determined based on the maximum common-point difference between the multi-channel 2D texture mapping sample image and the neighboring point mapping information. Predicted point cloud data is determined based on the importance weight scores of each channel and the 3D sample optimization model. Projection loss is calculated based on the similarity between the predicted point cloud data and the 3D point cloud sample data, and the similarity between the 3D geometric feature representation sample data and the 3D point cloud sample data. Cyclic consistency loss is determined based on the differences between the predicted point cloud data and the 3D point cloud sample data, the differences between the 3D geometric feature representation sample data and the 3D point cloud sample data, and the differences between the deformation-optimized sample data and the 2D texture mapping sample image. Finally, the loss function is determined based on the inverse mapping loss, the projection loss, and the cycle consistency loss.

[0069] The inverse mapping loss information constrains the minimum distance between each point and its neighbors in texture space. It measures the reasonableness of the mapping between each point and its neighbors in the texture space of the multi-channel 2D texture mapping sample image, preventing excessive clustering of texture coordinates, avoiding texture overlap, ensuring a uniform and reasonable texture distribution after unfolding, and maintaining the local topological relationships of the original 3D model. Neighboring points are the set of points adjacent to the target point in the 3D sample model; their mapping in texture space should maintain a reasonable distance from the target point. Neighboring point mapping information is the coordinate mapping data of neighboring points in texture space. The maximum similarity difference is the maximum value of the difference between the target point and the corresponding neighboring point mapping information in the multi-channel 2D texture mapping sample image. Predicted point cloud data is the point cloud data obtained by combining the importance weight scores of each channel with the 3D sample optimization model; it is the model's prediction result for the 3D point cloud sample data. 3D point cloud sample data is the real point cloud data of the 3D sample model, used to measure the accuracy of the predicted point cloud data. Projection loss information is used to measure the similarity between the predicted point cloud data and the 3D point cloud sample data, and between the 3D geometric feature representation sample data and the 3D point cloud sample data. Cycle consistency loss information is used to measure the loss of data consistency during the iterative transformation of the model, ensuring that the data can revert to its original state after the iterative transformation. The difference is the degree of difference between two data points or datasets, and is used to calculate the cycle consistency loss.

[0070] In this embodiment, avoiding mutual overlap is a requirement in the UV unwrapping process. If the UV coordinates overlap on the 2D plane after unwrapping, the same texture area will be mapped to different parts of the 3D model that should not share textures, resulting in texture errors or visual artifacts. This is a multi-channel 2D texture mapping sample image, which is a UV coordinate set and also a point set, where each point... Each of these is a two-dimensional coordinate, where i represents the i-th point in the point set, S represents the number of channels, and s represents the s-th channel. yes In UV space, there is a set of neighboring points (i.e., neighboring point mapping information). These neighboring points are usually based on the original 3D model. The neighbor relationship is defined, meaning that adjacent points in 3D space should also be adjacent after UV unwrapping. The process of determining the neighbor mapping information can be as follows: During the UV coordinate generation process of the texture channel, for each 2D texture point, the formation of its neighbor set follows the 3D spatial topological relationship: First, find the spatial neighbors of the corresponding vertex in the original 3D model, and determine the nearest neighbor set through a preset upper limit on the number of neighbors; then, convert these 3D neighbor points into corresponding points in the 2D texture space through coordinate mapping; finally, while maintaining the original spatial adjacency relationship, form a fixed number of neighbor points in the current texture channel to ensure the consistency of the local structure of the 3D model during texture unwrapping. Point The number of neighbors, where k represents the k-th neighbor. The maximum difference between points can be determined by... Determine, for example, the inverse mapping loss information It can be calculated using the following formula: ;in, This is the preset minimum safe distance, designed to prevent points from clustering together and overlapping, thereby ensuring the quality of texture mapping.

[0071] In this embodiment, the projection loss is determined by measuring the similarity between two point sets and the surfaces of two shapes. Simply put, it measures the average distance required to move one point set onto another. P is the ground truth 3D point cloud obtained from the 3D sample model. The three-dimensional geometric feature representation data output by the edge optimization network during the training phase is referred to as the three-dimensional geometric feature representation sample data for distinction. Assign importance weights to each channel of the network output during the training phase. To optimize the model based on the 3D samples output during the training phase. These are point clouds obtained through multi-channel combination, and both should be close to the ground truth 3D point cloud P. Note the training phase here. and as well as and The correspondence between them is unknown; for example, calculating the projection loss. Through relational expressions calculate, It means and The similarity between them (i.e.) ), It means and The similarity between them (i.e.) The two can be represented as follows: ; .

[0072] In this embodiment, the training framework is a bidirectional cyclic mapping framework. The goal of the loss is to ensure that after a cyclic transformation, each point can return to its original position or state. Point-to-point means that the loss is calculated for each individual point and then aggregated. Accordingly, this embodiment also calculates the cycle consistency loss. The first two terms of the cycle consistency loss represent the similarity loss of 3D point cycle consistency, and the last term represents the similarity loss of the UV coordinate set. For example, the cycle consistency loss can be calculated using the following relationship: ;in, It is the output of the deformable network during the training phase. It is a 2D UV mapping output by the projection network A during the training phase.

[0073] As shown above, this embodiment, through the unsupervised training process, can replace the original 3D structure with the generated model. This allows massive amounts of data crawled from the internet to be saved using the models trained in this way, resulting in UV maps corresponding to these massive 3D models. Each model, due to its small number of training iterations and simple network (consisting of MLP networks or simple convolutional networks), can run at high speed in parallel on CPUs or even FPGAs. Furthermore, during training, the introduction of inverse mapping loss information effectively avoids texture overlap during texture mapping, ensuring the rationality of texture mapping. Projection loss information accurately measures the similarity of point cloud data through chamfer distance, ensuring that the geometric features generated by the model are consistent with the real samples. Cyclic consistency loss information ensures data consistency during the model's cyclic transformation process, improving model stability and mapping accuracy. The fusion of multi-dimensional losses provides a comprehensive and clear optimization direction for model training, significantly improving model training performance and final feature transformation accuracy.

[0074] Based on the trained feature transformation model described above, this embodiment also provides the inference process of the feature transformation model, which may include the following: Loading a 3D model file to obtain vertex coordinates and face indices; calculating and normalizing vertex normals based on vertex coordinates and face indices; loading model weights obtained from unsupervised pre-training for the 3D model file; concatenating vertices and normals in the last dimension to obtain vector P; performing model inference on vector P to obtain R (texture position information corresponding to each texture channel) and Chart_P (multi-channel texture distribution information). Chart_P has a shape of [1, N, charts_number], representing the probability of each point belonging to each channel; charts_number is the number of channels; N is the number of sampling points; R is the model output corresponding to each channel; traversing the number of channels based on R and Chart_P to obtain the UV coordinates of the corresponding channels; summarizing the UV coordinates and saving them as a UV map.

[0075] As can be seen from the above, the feature transformation model in this embodiment can adaptively learn the mutual transformation between the 3D model and the texture mapping UV map, which improves the representation efficiency and does not require a complicated processing procedure.

[0076] Furthermore, considering that in the process of generating 3D models based on text, if the text information has poor coordination with texture and geometry generation, the generated 3D model will not meet the text requirements, and there will also be problems with inconsistency between texture and geometry. In this embodiment, the required information for the 3D model is text description information. A 3D data generation model is pre-trained. The 3D data generation model includes a text input terminal, a texture image generation network, a feature exchange network, and a geometry image generation network. This embodiment also provides a complete process of how to synthesize a 3D model from a text description using the trained 3D data generation model, which may include the following: Noise is added to both the 2D texture map and the geometric image to obtain noisy texture maps and noisy geometric images. The noisy texture maps are then input into a texture image generation network, and the noisy geometric images are input into a geometry image generation network. 3D model requirements are input into the texture image generation network via a text input terminal. The texture image generation network uses the 3D model requirements to guide the texture generation process, predicting noise from the noisy texture maps and gradually reconstructing a denoised 2D texture image. A feature exchange network connects the texture image generation network and the geometry image generation network, and performs interactive processing on the intermediate process data of the two networks. The geometry image generation network uses the intermediate structural features output by the feature exchange network to guide the geometric feature generation process, predicting noise from the noisy geometric images and gradually reconstructing a denoised geometric image.

[0077] The text description information, presented in natural language, is the core input guiding the generation of the 3D model and represents the requirements of the 3D model. The text input module is used to receive and process the text description information within the 3D data generation model. The texture image generation network generates denoised 2D texture images based on the text description information and a noisy texture map. The feature exchange network facilitates data exchange between the texture image generation network and the geometry image generation network during the intermediate process. The geometry image generation network generates denoised geometric images based on the intermediate structural features output by the feature exchange network and the noisy geometric image. The noisy texture map is the image obtained by adding Gaussian noise to the 2D texture map and serves as the input to the texture image generation network. The noisy geometric image is the image obtained by adding Gaussian noise to the geometric image and serves as the input to the geometry image generation network. The texture generation process refers to the process by which the texture image generation network predicts noise from the noisy texture map and gradually reconstructs the texture image. The intermediate process data includes the feature data and intermediate outputs generated by the texture image generation network and the geometry image generation network during processing. The interactive processing is the process by which the feature exchange network transfers, fuses, and transforms the intermediate process data between the two generation networks. The geometric feature generation process is the process by which a geometric image generation network predicts noise from a noisy geometric image and progressively reconstructs the geometric image. A denoised 2D texture image is a clear 2D texture image obtained after denoising and reconstruction by a texture image generation network. A denoised geometric image is a clear geometric image obtained after denoising and reconstruction by a geometric image generation network.

[0078] The texture image generation network uses a diffusion model to denoise the input, given a three-dimensional representation vector. The diffusion model first generates a series of Markov chain latent vectors. This series of latent vectors is progressively transformed into three-dimensional representation vectors. Add Gaussian noise. The formula for adding Gaussian noise is as follows: .in, Refers to one of the series of latent vectors in a Markov chain. It is a general expression that uses lowercase letters to represent variable indices and uppercase letters to represent the upper limit of the index. This represents the noisy state at time step t. This is the last part of the sequence, representing the final state and pure Gaussian noise. The original image's information has been completely destroyed, becoming purely random noise. This describes how noise is added from the previous time step t-1 to the current time step t. The noise variance schedule is represented by a decimal between 0 and 1, which represents the intensity of the noise added at time step t. The larger the value, the more noise is added at this time step, affecting the original image. The less information is retained, usually These are pre-defined hyperparameters that increase linearly or cosinely with time t. Normal / Gaussian distribution, indicating This variable follows a normal distribution with a mean of 1 / 2. Variance is Let I be the identity matrix, which is a matrix with 1s on the diagonal and 0s on the rest. Based on implicit vectors. The noise reduction process is achieved through relational expressions. To achieve this. Among them, The probability density function refers to the probability distribution of the reverse process, and it has a subscript. This means that the probability distribution is not like the forward process. It's not fixed like that, but rather determined by a neural network (with parameters...) The goal of the probability density function is to learn and predict the actual denoising process. is the mean, and is the center position (i.e., the expected value) of the normal distribution N. This represents the mean predicted by the neural network. In the process of generating a 3D shape, the initial... From Gaussian noise Random sampling is used to gradually reduce noise and obtain a true three-dimensional representation vector. In practice, the bias of the prediction noise, rather than the true 3D representation vector, is typically used as the optimization training function. It can be represented as: ;in, From Gaussian noise Random sampling, It is the input condition c, the input time step t, and the noisy three-dimensional representation vector at time step t. The bias obtained through neural network inference These are the parameters learned by the neural network, c represents various conditional inputs, which in this embodiment refers to text descriptions, L is the optimization objective during model training, E is the mathematical expectation, T is the total number of time steps, and is the maximum number of steps in the diffusion process, which can be a preset constant, such as T=1000.

[0079] In this embodiment, the 3D data generation model combines the aforementioned UV representation with a powerful 2D diffusion model to achieve end-to-end generation from text to a high-quality 3D model. During generation, random noise is first added to the 2D texture map and geometric image, resulting in noisy texture and geometry images. The noisy texture map and text description are then input into the texture image generation network. This network, based on the Stable Diffusion architecture, utilizes its internal cross-attention mechanism to understand the text conditions and performs the reverse denoising process of the diffusion model, progressively predicting and removing noise to ultimately generate a clear, denoised 2D texture image (i.e., a color UV map) that matches the text description. Simultaneously, the noisy geometry image is input into the geometry image generation network. A feature exchange network connects the two networks, dynamically extracting structural and semantic features from the intermediate layers of the texture image generation network during generation and passing them to the corresponding layers of the geometry image generation network. Guided by the feature exchange network, the geometry image generation network performs a denoising process, generating a denoised geometry image structurally aligned with the texture (i.e., a UV map describing the unfolded shape). Finally, the generated texture image and geometric image (i.e., a pair of registered UV maps) are reconstructed through a standard UV inverse mapping process to create the final 3D mesh model.

[0080] As can be seen from the above, the 3D data generation model in this embodiment is a diffusion model for texture-geometry co-generation. It makes full use of the prior knowledge of a powerful text-to-image diffusion model pre-trained on 2D image data. The text description information directly guides the texture generation process, ensuring that the generated texture is highly matched with the text requirements, thus improving the semantic consistency of the 3D model. Through the feature exchange mechanism, the geometry generation follows the semantic guidance of texture generation, realizing real-time data interaction between the texture and geometry generation processes. This enables the texture and geometry to be co-optimized, avoiding the problem of texture and geometry inconsistency. As a result, it can generate a high-quality and consistent 3D model in terms of both visual detail and geometric structure based on the text.

[0081] Furthermore, this embodiment also provides an exemplary network structure for a 3D data generation model, which can avoid problems such as mismatch between the layers of the generation network and asynchronous data interaction, which can lead to data transmission distortion and poor feature fusion effect in the intermediate process, thus affecting the quality of the final 3D model generation.

[0082] like Figure 10 As shown, the texture image generation network, feature exchange network, and geometry image generation network each contain the same total number of layers and are in one-to-one correspondence; for example... Figure 11As shown, the texture image generation network has the same structure for each network layer, which are connected sequentially. Each layer includes a texture convolutional module, a texture self-attention layer, and a cross-attention layer. The 3D model requirement information is input to each cross-attention layer, and the noisy texture map is input to the first texture convolutional module. The geometric image generation network has the same structure for each network layer, which are connected sequentially. Each layer includes a geometric convolutional module and a geometric self-attention layer, and the noisy geometric map is input to the first geometric convolutional module. The feature exchange network has the same structure for each network layer, which includes a connected splicing layer, a multilayer perceptron, and a feature separation layer in the direction of data from input to output. The splicing layer is connected to the texture self-attention layer and geometric self-attention layer of the corresponding layer, and the feature separation layer is connected to the cross-attention layer of the corresponding layer. The separated intermediate structural features are input to the corresponding layer of the geometric image generation network.

[0083] The total number of layers in the network is the same for the texture image generation network, feature exchange network, and geometry image generation network, with each network having the same number of layers and a one-to-one correspondence. The texture convolutional module is used to extract texture features in the texture image generation network. The texture self-attention layer is used to capture the internal dependencies between texture features. The cross-attention layer is used to fuse textual description information and texture features. The geometry convolutional module is used to extract geometric features in the geometry image generation network. The geometric self-attention layer is used to capture the internal dependencies between geometric features. The concatenation layer is used to concatenate the outputs of the texture self-attention layer and the geometric self-attention layer in the feature exchange network. The multilayer perceptron is used to perform nonlinear transformations and feature fusion on the concatenated features in the feature exchange network. The feature separation layer is used to separate the fused features into intermediate structural features. The intermediate structural features are the feature data obtained after processing by the feature exchange network and are used to guide the geometry image generation network. The corresponding layer refers to the network layer corresponding to the position in the three networks, ensuring the synchronization of data interaction.

[0084] In this embodiment, the network architecture on both sides of the 3D data generation model can adopt the Stable Diffusion v2.1 network architecture, and the initial weights are loaded with zero-terminal-SNR (zero terminal signal-to-noise ratio) model weights. The difference is that the network weights of the cross-attention layer are removed from the right-side network. As shown in the diffusion model principle above, the training principle is to predict the added noise from a randomly noisy image, and the text is the textual description of the data, such as "a red poisonous mushroom". The feature exchange network connects with the two end networks and completes the information exchange between the two generation networks. Each layer of the feature exchange network corresponds to each layer of the generation network. Its structure is as follows: stitching layer, multilayer perceptron (MLP), and feature separation layer. In each layer, the stitching layer receives the output features from the texture self-attention layer and the geometry self-attention layer of that layer and stitches them together. The stitched features are then fused by the MLP. The feature separation layer splits the output features of the MLP. One part (which can be regarded as intermediate structural features) is input into the next layer of the geometry image generation network to continue to participate in geometry generation; the other part may be fed back or used for other purposes, which is not limited in this embodiment. During the training of the 3D data generation model, only the feature exchange network, the exchange module, and the geometric image generation network are trained.

[0085] As shown above, this embodiment ensures the synchronization of data interaction by maintaining a consistent total number of layers and a one-to-one correspondence among the three networks, avoiding feature transmission distortion caused by layer misalignment. The cross-attention layers in each layer of the texture image generation network achieve deep fusion of text information and texture features, improving the semantic accuracy of texture generation. The hierarchical design of the feature exchange network enables precise transmission and effective fusion of intermediate process data; intermediate structural features can accurately guide geometric image generation, improving the synergy between texture and geometry. Fine-grained feature interaction at multiple scales is achieved in the texture and geometry generation process. The intermediate features generated by the texture network under text guidance can guide the generation of the geometry network in real time and at multiple scales, ensuring that the final output texture and geometry maps are highly aligned in both overall outline and local details, thereby synthesizing a structurally sound 3D model.

[0086] Based on the above embodiments, considering that users may be dissatisfied with the AI-generated 3D model, this embodiment also supports users to quickly modify the model through simple feedback (such as text editing commands) without having to regenerate it from scratch. The process of interactive editing based on the generated model may include: Upon receiving a text editing instruction, noise is added to both the 2D texture map and the geometric image, resulting in a noisy texture map and a noisy geometric image. The noisy texture map is then concatenated with the 2D texture map, and the concatenated texture image and the text editing instruction are input into the texture image editing network of the trained 3D data editing model. Similarly, the noisy geometric image is concatenated with the geometric image, and the concatenated geometric image is input into the geometric image editing network of the 3D data editing model. The 3D data editing model also includes a feature exchange editing network connecting the geometric image editing network and the texture image editing network. The texture image editing network simultaneously perceives the original state and the editing intent, predicting and generating a 2D texture correction image that matches the editing intent through a diffusion denoising process. The feature exchange editing network interactively processes the intermediate process data of the texture image editing network and the geometric image editing network. Finally, the geometric image editing network, based on the editing semantic perception features output by the feature exchange editing network, corrects the concatenated geometric features into a geometric correction image that matches the edited texture.

[0087] The text editing instructions are the user's modification requests for the generated 2D texture map and / or geometric image, presented in text form, such as "change the mushroom cap to bright red and add white spots." The stitched texture image is the image obtained by channel-wise stitching of the noisy texture map and the original 2D texture map, used by the texture image editing network to perceive the original state and editing intent. The stitched geometric image is the image obtained by channel-wise stitching of the noisy geometric image and the original geometric image, used by the geometric image editing network to perceive the original geometric state. The 3D data editing model is a model used to modify the 3D model according to the text editing instructions, including a texture image editing network, a geometric image editing network, and a feature exchange editing network. The texture image editing network generates a 2D texture correction image that conforms to the editing intent based on the text editing instructions and the stitched texture image. The geometric image editing network generates a geometric correction image based on the editing semantic perception features output by the feature exchange editing network and the stitched geometric image. The feature exchange editing network enables data interaction between the texture image editing network and the geometric image editing network during the intermediate process. The original state refers to the texture and geometry of the 3D model before editing, represented by the original 2D texture map and geometric image. The editing intent is the modification request expressed by the user through text editing instructions. The diffusion denoising process involves the texture image editing network and the geometric image editing network predicting noise from the noisy stitched image, progressively denoising, and generating a corrected image. The 2D texture corrected image is the texture image that matches the editing intent after processing by the texture image editing network. Edit semantic awareness features are feature data containing editing semantics obtained after processing by the feature exchange editing network, used to guide the correction process of the geometric image editing network. The geometric corrected image is the geometric image that matches the 2D texture corrected image after processing by the geometric image editing network.

[0088] After generating an initial 3D model, the editing process begins when the user provides text editing instructions (such as "increase the server height"). Figure 12 As shown, taking the generation of a red mushroom as an example, the text editing instruction is to change the color of the mushroom cap to bright red and add white spots. First, noise is added to the original 2D texture map and geometric image to obtain a noisy version. Then, the noisy texture map is concatenated with the original clear texture map in the channel dimension to form a conditional input containing information about the original state and the state to be edited. This conditional input, along with the text editing instruction, is input into a texture image editing network. Similarly, the noisy geometric image is concatenated with the original geometric image and input into a geometry image editing network. The two networks are connected by a feature exchange editing network. The texture image editing network simultaneously perceives the original texture and the editing intention, and generates a 2D texture correction image that conforms to the text editing instruction through diffusion denoising. The feature exchange editing network extracts features containing editing semantics (e.g., which areas have changed) from the texture editing process and passes them to the geometry image editing network. Guided by these features, the geometry image editing network denoises the concatenated geometric conditions to generate a geometry correction image that matches the edited texture. Finally, the edited 3D model is reconstructed from the corrected texture and geometry.

[0089] As shown above, this embodiment provides an intuitive and efficient AI model post-editing solution. The input of stitched images allows the editing network to simultaneously perceive the original state and noisy data, improving the accuracy and consistency of editing. Text editing commands directly guide the texture correction process, ensuring that the modified texture meets user needs and improving the semantic consistency of the edit. The feature-exchange editing network effectively transmits editing semantics, enabling geometric and texture corrections to proceed collaboratively, ensuring that the modified texture matches the geometric structure. Users do not need complex 3D modeling operations; they can drive the system to simultaneously and collaboratively modify the model's texture and geometry according to their intentions using only natural language commands, greatly improving the efficiency and flexibility of 3D content creation.

[0090] Furthermore, this embodiment also provides an exemplary structure for a three-dimensional data editing model, such as... Figure 13 As shown, this method avoids issues such as hierarchical mismatch and distortion of editing semantics, which can lead to inaccurate responses to text editing commands in geometric corrections and inconsistencies between the texture and geometry of the modified 3D model. This embodiment demonstrates how to construct a network specifically for editing tasks, enabling it to effectively utilize the knowledge of pre-trained generative models and handle additional conditional inputs. This may include the following: The texture image editing network, feature exchange editing network, and geometric image editing network each contain the same total number of layers, and they correspond one-to-one. Specifically, the texture image editing network has identical and sequentially connected layer structures, each including a texture convolutional module, a texture self-attention layer, and a cross-attention layer. Text editing instructions are input to each cross-attention layer, and the stitched texture image is input to the first texture convolutional module. The geometric image editing network also has identical and sequentially connected layer structures, each including a geometric convolutional module and a geometric self-attention layer. The stitched geometric image is input to the first geometric convolutional module. The feature exchange editing network has identical layer structures, and in the direction of data input to output, it includes a connected stitching layer, a multilayer perceptron, and a feature separation layer. The stitching layer is connected to the corresponding texture self-attention layer and geometric self-attention layer, and the feature separation layer is connected to the corresponding cross-attention layer. The feature separation layer extracts features containing editing semantics and inputs these editing semantic-aware features to the corresponding layer of the geometric image editing network.

[0091] Among them, the semantically aware editing features are feature data containing the user's editing intent extracted by the feature exchange editing network from the intermediate process data of the texture image editing network.

[0092] The hierarchical structures of the texture image editing network, geometric image editing network, and feature exchange editing network in this embodiment are basically the same as those of the 3D data generation model network, and they correspond one-to-one. The difference lies in the input conditions: each cross-attention layer of the texture image editing network receives text editing instructions, while its first-layer texture convolutional module receives a stitched texture image (with more channels than during generation). The first-layer geometric convolutional module of the geometric image editing network receives a stitched geometric image. The structure of the feature exchange editing network is consistent with the feature exchange network in the generation model. During the editing process, the feature separation layer actively identifies and extracts those editing semantic-aware features related to the editing action (e.g., the vertical stretching feature corresponding to "increase height") from the features processed by the MLP, and accurately injects these features into the corresponding layers of the geometric image editing network to drive the geometric shape to undergo corresponding and reasonable changes. In other words, the network structure of the 3D data editing model and the 3D data generation model can be the same. The difference lies in the number of input image channels. The 3D data generation model only has a noisy textured image as input, with 4 channels. Due to the compression of the original image by the VAE (Variational Autoencoder), the resolution becomes 1 / 8 of the original, and the number of channels changes from three to four. The 3D data editing model, however, has 8 channels due to image stitching. The model parameters of the 3D data editing model can directly use the parameters of the 3D data generation model as initialization parameters. That is, the 3D data generation model is used as a pre-trained model for the 3D data editing model. Then, through the downstream task of editing, the matching training sample set (e.g., 2000) is used to fine-tune the data before and after editing. Initializing and fine-tuning based on the pre-trained model improves the training efficiency and editing effect of the editing model.

[0093] As shown above, this embodiment achieves editing functionality by expanding the input channels and fine-tuning the network based on the 3D data generation model architecture. It can reuse the powerful priors learned in the generation stage and, through learning from the comparison information before and after editing, quickly master how to transform the user's text editing intent into UV mapping and then specific modifications to the 3D model, achieving an efficient integrated process of generation and editing. Furthermore, the consistency and correspondence of network layers ensure the synchronization and accuracy of editing semantic transmission, avoiding distortion of editing intent caused by layer misalignment. The cross-attention layers of each layer of the texture image editing network achieve deep fusion of text editing instructions and texture features, enabling texture correction to accurately respond to editing needs. The editing semantic awareness features extracted by the feature exchange editing network can accurately guide geometric correction, ensuring a high degree of synergy between the modified texture and geometric structure, improving the editing effect.

[0094] This invention also provides a corresponding apparatus for the three-dimensional data generation method, further enhancing the practicality of the method. The apparatus can be described from both a functional module perspective and a hardware perspective. The three-dimensional data generation apparatus provided by this invention is described below. This apparatus is used to implement the three-dimensional data generation method provided by this invention. In this embodiment, the three-dimensional data generation apparatus may include or be divided into one or more program modules. These one or more program modules are stored in a storage medium and executed by one or more processors to complete the three-dimensional data generation method disclosed in Embodiment 1. The program module referred to in this embodiment is a series of computer program instruction segments capable of performing a specific function, which is more suitable than the program itself for describing the execution process of the three-dimensional data generation apparatus in the storage medium. The following description will specifically introduce the functions of each program module in this embodiment. The three-dimensional data generation apparatus described below can be referred to in correspondence with the three-dimensional data generation method described above.

[0095] From the perspective of functional modules, the three-dimensional data generation device provided in this embodiment may include: The geometric representation module is used to determine the geometric feature representation information of the matching 3D model based on the 3D model requirement information.

[0096] The texture mapping module is used to map geometric feature representation information into multi-channel texture distribution information and texture position information corresponding to each texture channel through feature transformation correspondence. Based on the multi-channel texture distribution information, the texture position information is fused to generate two-dimensional texture mapping data, which serves as a two-dimensional texture mapping map. The feature transformation correspondence is the correspondence between the geometric features of the pre-constructed three-dimensional sample model and the two-dimensional texture features.

[0097] The geometric image acquisition module is used to remove texture features from the two-dimensional texture map to obtain a geometric image.

[0098] The 3D data generation module is used to generate 3D model representation data based on 2D texture mapping, geometric images, and 3D model requirements.

[0099] For example, in some embodiments of this example, the texture mapping module described above can also be used to: determine a matching original 3D model based on the 3D model requirement information, and sample 3D data points on the original 3D model; generate initial texture map mesh data based on geometric feature representation information, perform adaptive deformation processing on the initial texture map mesh data, generate deformation optimization data that matches the surface geometry of the original 3D model, while maintaining the dimensional consistency of the position information; reconstruct the deformation optimization data into 3D geometric feature representation data with spatial position and surface orientation information, perform detail optimization processing on the edge region of the original 3D model corresponding to the 3D geometric feature representation data, and expand the feature dimension to obtain multi-channel geometric feature data; determine the multi-channel texture distribution information of the multi-channel geometric feature data based on the belonging probability between vertices and texture channels, and inversely map the multi-channel geometric feature data into the texture position information of each texture channel.

[0100] As an exemplary implementation of the above embodiments, the texture mapping module can be further used to: determine a matching original 3D model based on the 3D model requirement information; determine height and width information based on the number of vertices sampled in a single run of the original 3D model; determine the vertical direction based on the height information and the horizontal direction based on the width information; sample the original 3D model at equal intervals along the vertical and horizontal directions in a preset standardized parameter space, and generate two-dimensional texture coordinates corresponding to the 3D data points through Cartesian product; determine the matrix row and column parameters according to the height and width information, and use the two-dimensional texture coordinates of each 3D sampling point as matrix elements to generate a sampling matrix of the original 3D model; generate 3D tensor data according to the number of original 3D models processed each time, the sampling matrix of each original 3D model, and the dimension of the matrix elements, to serve as initial texture map mesh data.

[0101] As another exemplary implementation of the above embodiments, the texture mapping module can be further used to: input the initial texture map mesh data into the first fully connected layer to obtain the first deformation feature, and splice the first deformation feature with the initial texture map mesh data; input the deformation splicing feature into the second fully connected layer to obtain the second deformation feature; and add the second deformation feature with the initial texture map mesh data element by element to obtain the deformation optimization data.

[0102] As another exemplary implementation of the above embodiments, the texture mapping module can be further used to: input deformation optimization data into the third fully connected layer to obtain reconstructed features; splice the reconstructed features and deformation optimization data, and input the reconstructed spliced ​​features into the fourth fully connected layer; and use the output of the fourth fully connected layer as three-dimensional geometric feature representation data.

[0103] As another exemplary implementation of the above embodiments, the texture mapping module can be further used to: input three-dimensional geometric feature representation data into a first stacked convolutional layer to obtain a first detail optimization feature, and concatenate the first detail optimization feature with the three-dimensional geometric feature representation data; input the concatenated detail optimization feature into a second stacked convolutional layer to obtain a second detail optimization feature; add the second detail optimization feature to the three-dimensional geometric feature representation data element-wise to obtain three-dimensional geometric feature optimization data; input the three-dimensional geometric feature optimization data into a feature upscaling network to obtain multi-channel geometric feature data; wherein, the first stacked convolutional layer and the second stacked convolutional layer include multiple feature extraction and fusion layers; each feature extraction and fusion layer includes a convolutional layer and an activation function layer; the feature upscaling network includes two stacked convolutional layers.

[0104] As another exemplary implementation of the above embodiments, the texture mapping module can be further used to: input multi-channel geometric feature data into the fifth fully connected layer, and process the output of the fifth fully connected layer using an activation function to obtain the importance allocation weight value of each channel of the multi-channel geometric feature data; input multi-channel geometric feature data into the sixth fully connected layer, and use the output of the sixth fully connected layer as the position information of each channel in the texture space.

[0105] As another exemplary implementation of the above embodiments, the texture mapping module can be further used to: determine the matching original 3D model according to the 3D model requirement information, and obtain the vertex coordinates and face index information of the original 3D model; determine the vertex normal according to the vertex coordinates and face index information, and perform normalization processing on the vertex normal to obtain the normal vector; concatenate the normal vector and the vertex coordinates in the feature channel dimension, and use the concatenated vector as the geometric feature representation information corresponding to the 3D model requirement information.

[0106] For example, in some other embodiments of this example, the texture mapping module can also be used to: input geometric feature representation information into a pre-trained feature conversion model to obtain multi-channel texture distribution information and texture position information corresponding to each texture channel. Correspondingly, the above device may also include a training module, which can be used to: the feature conversion model includes a deformation network, a first inverse mapping network, an edge optimization network, a first projection network, a feature upscaling network, and an allocation network in the inference phase, and further includes a second projection network, a fusion network, and a second inverse mapping network in the training phase; the training process of the feature conversion model includes: inputting the initial texture map mesh data of the three-dimensional sample model into the feature conversion model, sequentially passing it through the deformation network, the first inverse mapping network, the edge optimization network, and the second projection network to obtain a two-dimensional texture mapping sample map; inputting the three-dimensional geometric feature optimization sample data output by the edge optimization network into the feature upscaling network... A three-dimensional network is used to input the feature upscaling vectors into the assignment network and the first projection network, respectively, to obtain the importance weight scores of each channel and the multi-channel two-dimensional texture mapping sample map input to the second inverse mapping network. The fusion network integrates the importance weight scores of each channel and the output of the second inverse mapping network into a three-dimensional sample optimization model. The loss function is determined based on the three-dimensional geometric feature representation sample data, deformation optimization sample data, two-dimensional texture mapping sample map, multi-channel two-dimensional texture mapping sample map, importance weight scores of each channel, three-dimensional sample optimization model, and three-dimensional point cloud sample data. The feature transformation model is continuously trained based on the loss function until the iteration stopping condition is met.

[0107] As an exemplary implementation of the above embodiments, the training module can also be used to: determine that each point of the multi-channel two-dimensional texture mapping sample map corresponds to a neighboring point in the three-dimensional sample model, and convert each group of neighboring points into neighboring point mapping information in the texture space; determine the inverse mapping loss information based on the maximum common point difference between the multi-channel two-dimensional texture mapping sample map and the neighboring point mapping information; determine the predicted point cloud data based on the importance weight score of each channel and the three-dimensional sample optimization model; determine the projection loss information based on the similarity between the predicted point cloud data and the three-dimensional point cloud sample data, and the similarity between the three-dimensional geometric feature representation sample data and the three-dimensional point cloud sample data; determine the cycle consistency loss information based on the difference between the predicted point cloud data and the three-dimensional point cloud sample data, the difference between the three-dimensional geometric feature representation sample data and the three-dimensional point cloud sample data, and the difference between the deformation optimization sample data and the two-dimensional texture mapping sample map; and determine the loss function based on the inverse mapping loss information, the projection loss information, and the cycle consistency loss information.

[0108] For example, in some other embodiments of this embodiment, the above-mentioned three-dimensional data generation module can also be used as follows: the three-dimensional model requirement information is text description information, and the trained three-dimensional data generation model includes a text input terminal, a texture image generation network, a feature exchange network, and a geometric image generation network; noise is added to the two-dimensional texture map and the geometric image respectively to obtain a noisy texture map and a noisy geometric image; the noisy texture map is input to the texture image generation network, the noisy geometric image is input to the geometric image generation network, and the three-dimensional model requirement information is input to the texture image generation network through the text input terminal; the texture image generation network uses the three-dimensional model requirement information to guide the texture generation process, predicts noise from the noisy texture map, and gradually reconstructs a denoised two-dimensional texture image; the feature exchange network connects the texture image generation network and the geometric image generation network, and performs interactive processing on the intermediate process data of the texture image generation network and the geometric image generation network; the geometric image generation network uses the intermediate structural features output by the feature exchange network to guide the geometric feature generation process, predicts noise from the noisy geometric image, and gradually reconstructs a denoised geometric image.

[0109] As an exemplary implementation of the above embodiments, the texture image generation network, feature exchange network, and geometric image generation network each contain the same total number of network layers and correspond one-to-one. Specifically, the texture image generation network has identical and sequentially connected layer structures, each including a texture convolutional module, a texture self-attention layer, and a cross-attention layer. 3D model requirement information is input to each cross-attention layer, and a noisy texture map is input to the first texture convolutional module. Similarly, the geometric image generation network has identical and sequentially connected layer structures, each including a geometric convolutional module and a geometric self-attention layer. A noisy geometric map is input to the first geometric convolutional module. The feature exchange network has identical layer structures, and sequentially includes a concatenating layer, a multilayer perceptron, and a feature separation layer along the input-to-output direction. The concatenating layer is connected to the corresponding texture self-attention layer and geometric self-attention layer, and the feature separation layer is connected to the corresponding cross-attention layer. The separated intermediate structural features are then input to the corresponding layer of the geometric image generation network.

[0110] For example, in some other embodiments of this embodiment, the above-described apparatus may further include an editing module, which may be used to: upon receiving a text editing instruction, add noise to a two-dimensional texture map and a geometric image respectively to obtain a noisy texture map and a noisy geometric image; stitch the noisy texture map with the two-dimensional texture map, and input the stitched texture image and the text editing instruction into the texture image editing network of a trained three-dimensional data editing model; stitch the noisy geometric image with the geometric image, and input the stitched geometric image into the geometric image editing network of the three-dimensional data editing model; the three-dimensional data editing model further includes a feature exchange editing network connecting the geometric image editing network and the texture image editing network; the texture image editing network simultaneously perceives the original state and the editing intention, and predicts and generates a two-dimensional texture correction image that conforms to the editing intention through a diffusion denoising process; the feature exchange editing network interactively processes the intermediate process data of the texture image editing network and the geometric image editing network; the geometric image editing network, based on the editing semantic perception features output by the feature exchange editing network, corrects the stitched geometric features into a geometric correction image that matches the edited texture.

[0111] As an exemplary implementation of the above embodiments, the texture image editing network, feature exchange editing network, and geometric image editing network each contain the same total number of network layers and correspond one-to-one. Specifically, the texture image editing network has identical and sequentially connected layer structures, each including a texture convolutional module, a texture self-attention layer, and a cross-attention layer. Text editing instructions are input to each cross-attention layer, and the stitched texture image is input to the first texture convolutional module. The geometric image editing network has identical and sequentially connected layer structures, each including a geometric convolutional module and a geometric self-attention layer. The stitched geometric image is input to the first geometric convolutional module. The feature exchange editing network has identical layer structures, and sequentially includes a stitching layer, a multilayer perceptron, and a feature separation layer along the input-to-output direction. The stitching layer is connected to the corresponding texture self-attention layer and geometric self-attention layer, and the feature separation layer is connected to the corresponding cross-attention layer. The feature separation layer extracts features containing editing semantics and inputs these editing semantic-aware features to the corresponding layer of the geometric image editing network.

[0112] The three-dimensional data generation device mentioned above is described from the perspective of functional modules. Furthermore, the present invention also provides an electronic device, which is described from the perspective of hardware. The electronic device includes a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above-described three-dimensional data generation method embodiments.

[0113] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described embodiments of the three-dimensional data generation method at runtime.

[0114] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0115] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described embodiments of the three-dimensional data generation method.

[0116] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in any of the above-described embodiments of the three-dimensional data generation method.

[0117] The foregoing has provided a detailed description of a three-dimensional data generation method and electronic device provided by the present invention. The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. Whether the units and algorithm steps of the various examples described in the disclosed embodiments are executed by electronic hardware or computer software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, and such implementations should not be considered beyond the scope of the present invention. Several improvements and modifications can be made to the present invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the present invention.

Claims

1. A method for generating three-dimensional data, characterized in that, include: Determine the geometric feature representation information of the matching 3D model based on the 3D model requirement information; The geometric feature representation information is mapped into multi-channel texture distribution information and texture position information corresponding to each texture channel through feature transformation correspondence. Based on the multi-channel texture distribution information, the texture position information is fused to generate two-dimensional texture mapping data, which serves as a two-dimensional texture mapping map. The texture features of the two-dimensional texture map are removed to obtain a geometric image; Based on the two-dimensional texture mapping map, the geometric image, and the three-dimensional model requirement information, generate three-dimensional model representation data; The feature transformation correspondence is the correspondence between the geometric features of the pre-constructed three-dimensional model and the two-dimensional texture features.

2. The three-dimensional data generation method according to claim 1, characterized in that, The feature transformation correspondence is mapped to multi-channel texture distribution information and texture position information corresponding to each texture channel. Based on the multi-channel texture distribution information, the texture position information is fused to generate two-dimensional texture mapping data, including: Based on the required information of the three-dimensional model, a matching original three-dimensional model is determined, and three-dimensional data points are sampled on the original three-dimensional model; Initialized texture map mesh data is generated based on the geometric feature representation information. Adaptive deformation processing is performed on the initialized texture map mesh data to generate deformation optimization data that matches the surface geometry of the original 3D model, while maintaining the consistency of the dimension of the position information. The deformation optimization data is reconstructed into three-dimensional geometric feature representation data with spatial position and surface orientation information. The edge region of the original three-dimensional model corresponding to the three-dimensional geometric feature representation data is subjected to detail optimization processing and feature dimension expansion to obtain multi-channel geometric feature data. Based on the attribution probability between vertices and texture channels, the multi-channel texture distribution information of the multi-channel geometric feature data is determined, and the multi-channel geometric feature data is inversely mapped to the texture position information of each texture channel.

3. The three-dimensional data generation method according to claim 2, characterized in that, Generate initial texture map mesh data based on the geometric feature representation information, including: Based on the required information of the three-dimensional model, a matching original three-dimensional model is determined. The height and width information are determined based on the number of vertices sampled in a single sampling of the original three-dimensional model. The vertical direction is determined by the height information and the horizontal direction is determined by the width information. In a pre-defined standardized parameter space, the original 3D model is sampled at equal intervals along the vertical and horizontal directions respectively, and the 2D texture coordinates corresponding to the 3D data points are generated by Cartesian product. The sampling matrix of the original 3D model is generated by using the height and width information as matrix row and column parameters and the two-dimensional texture coordinates of each 3D sampling point as matrix elements. Based on the number of original 3D models processed each time, the sampling matrix of the original 3D models, and the dimensions of the matrix elements, 3D tensor data is generated to serve as the initial texture map mesh data.

4. The three-dimensional data generation method according to claim 2, characterized in that, Adaptive deformation processing is performed on the initialized texture map mesh data to reconstruct the deformation-optimized data into three-dimensional geometric feature representation data with spatial position and surface orientation information, including: The initial texture map mesh data is input into the first fully connected layer to obtain the first deformation feature, and the first deformation feature is concatenated with the initial texture map mesh data; The deformation splicing features are input into the second fully connected layer to obtain the second deformation features; The second deformation feature is added element-wise to the initialized texture map mesh data to obtain deformation optimization data; The deformation optimization data is input into the third fully connected layer to obtain the reconstructed features; The reconstructed features are combined with the deformation optimization data, and the reconstructed and combined features are input into the fourth fully connected layer. The output of the fourth fully connected layer is used as three-dimensional geometric feature representation data.

5. The three-dimensional data generation method according to claim 2, characterized in that, The edge regions of the original 3D model corresponding to the 3D geometric feature representation data undergo detail optimization processing and feature dimension expansion, including: The three-dimensional geometric feature representation data is input into the first stacked convolutional layer to obtain the first detail optimization feature, and the first detail optimization feature is concatenated with the three-dimensional geometric feature representation data; The detailed optimization stitched features are input into the second stacked convolutional layer to obtain the second detailed optimization features; The second detailed optimization feature is added element-wise to the three-dimensional geometric feature representation data to obtain the three-dimensional geometric feature optimization data; The optimized three-dimensional geometric feature data is input into a feature upscaling network to obtain multi-channel geometric feature data; The first stacked convolutional layer and the second stacked convolutional layer include multiple feature extraction and fusion layers; each feature extraction and fusion layer includes a convolutional layer and an activation function layer; the feature upscaling network includes two stacked convolutional layers.

6. The three-dimensional data generation method according to claim 2, characterized in that, Based on the attribution probability between vertices and texture channels, the multi-channel texture distribution information of the multi-channel geometric feature data is determined, and the multi-channel geometric feature data is inversely mapped to the texture position information of each texture channel, including: The multi-channel geometric feature data is input into the fifth fully connected layer, and the output of the fifth fully connected layer is processed using an activation function to obtain the importance weight values ​​of each channel of the multi-channel geometric feature data. The multi-channel geometric feature data is input into the sixth fully connected layer, and the output of the sixth fully connected layer is used as the position information of each channel in the texture space.

7. The three-dimensional data generation method according to any one of claims 1 to 6, characterized in that, The feature transformation correspondence is a trained feature transformation model. The feature transformation model includes a deformation network, a first inverse mapping network, an edge optimization network, a first projection network, a feature dimensionality enhancement network, and an allocation network in the inference stage, and also includes a second projection network, a fusion network, and a second inverse mapping network in the training stage. The training process of the feature transformation model includes: The initial texture map mesh data of the 3D sample model is input into the feature transformation model, and then sequentially passed through a deformation network, a first inverse mapping network, an edge optimization network, and a second projection network to obtain a 2D texture mapping sample map. The 3D geometric feature optimization sample data output by the edge optimization network is input into the feature dimensionality increase network, and the feature dimensionality increase vector is input into the allocation network and the first projection network respectively to obtain the importance weight scores of each channel and the multi-channel 2D texture mapping sample map used as input to the second inverse mapping network. The fusion network fuses the importance weight scores of each channel and the output results of the second inverse mapping network into a 3D sample optimization model. The loss function is determined based on the 3D geometric feature representation sample data, deformation optimization sample data, 2D texture mapping sample map, multi-channel 2D texture mapping sample map, importance weight score of each channel, 3D sample optimization model, and 3D point cloud sample data. The feature transformation model is then continuously trained based on the loss function until the iteration stopping condition is met.

8. The three-dimensional data generation method according to claim 7, characterized in that, The process of determining the loss function includes: Each point of the multi-channel two-dimensional texture mapping sample image is determined to correspond to a neighboring point of the three-dimensional sample model, and each group of neighboring points is converted into neighboring point mapping information in the texture space. The inverse mapping loss information is determined based on the maximum common point difference between the multi-channel two-dimensional texture mapping sample image and the neighboring point mapping information. The predicted point cloud data is determined based on the importance weight scores of each channel and the 3D sample optimization model. The projection loss information is calculated based on the similarity between the predicted point cloud data and the 3D point cloud sample data, and the similarity between the 3D geometric feature representation sample data and the 3D point cloud sample data. Based on the difference between the predicted point cloud data and the 3D point cloud sample data, the difference between the 3D geometric feature representation sample data and the 3D point cloud sample data, and the difference between the deformation optimization sample data and the 2D texture mapping sample map, the cycle consistency loss information is determined. The loss function is determined based on the inverse mapping loss information, the calculated projection loss information, and the cycle consistency loss information.

9. The three-dimensional data generation method according to any one of claims 1 to 6, characterized in that, The 3D model requirement information is text description information, and the trained 3D data generation model includes a text input terminal, a texture image generation network, a feature exchange network, and a geometric image generation network. Based on the two-dimensional texture map, the geometric image, and the three-dimensional model requirement information, three-dimensional model representation data is generated, including: Noise is added to the two-dimensional texture map and the geometric image respectively to obtain a noisy texture map and a noisy geometric image; The noisy texture map is input into the texture image generation network, the noisy geometric image is input into the geometric image generation network, and the 3D model requirement information is input into the texture image generation network through the text input terminal; The texture image generation network uses the 3D model requirement information to guide the texture generation process, predicts noise from the noisy texture map, and gradually reconstructs a denoised 2D texture image. The feature exchange network connects the texture image generation network and the geometry image generation network, and performs interactive processing on the intermediate process data of the texture image generation network and the geometry image generation network. The geometry image generation network uses the intermediate structural features output by the feature exchange network to guide the geometric feature generation process, predicts noise from the noisy geometric image, and gradually reconstructs a denoised geometric image.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the three-dimensional data generation method as described in any one of claims 1 to 9.

Images