Method and apparatus for processing mesh models

Abstract

The present disclosure relates to a method for processing a mesh model, a method for training a neural network, an apparatus for processing a mesh model, an electronic device and a computer readable storage medium. The method comprises: obtaining a source two-dimensional image, and determining an embedding feature of the source two-dimensional image based on the source two-dimensional image; obtaining a target two-dimensional image, and determining an embedding feature of the target two-dimensional image based on the target two-dimensional image; determining a semantic mapping feature based on the embedding feature of the source two-dimensional image and the embedding feature of the target two-dimensional image, the semantic mapping feature indicating a correspondence from a pixel in the source two-dimensional image to a pixel in the target two-dimensional image; iteratively updating a deformation model based at least in part on the semantic mapping feature, and deforming and rendering the source mesh model based on the updated deformation model to obtain a target mesh model fused with geometric information of the source mesh model.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence services, and more specifically to a method for processing mesh models, a method for training neural networks, an apparatus for processing mesh models, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Shape correspondence is an important research topic in the field of computer vision and graphics. Its main purpose is to build a point-to-point mapping between the source mesh model and the target mesh model so as to preserve the geometric features of the source mesh model as completely as possible when generating the target mesh model.

[0003] Shape mapping tasks can serve as subtasks within various tasks within computer vision and graphics. For example, in re-topology tasks, shape mapping can be used to determine the correspondence between mesh models with different topologies, transforming complex source topologies into target topologies more suitable for specific application needs, thereby improving the naturalness of 3D modeling. In attribute transfer tasks, once the correspondence between the source and target mesh models is determined, attributes such as color and texture from the source mesh model can be transferred to the target mesh model, achieving efficient attribute assignment. In shape interpolation tasks, interpolation calculations based on the correspondence between multiple mesh models can generate a series of transitional mesh models to support animation production and achieve special deformation effects.

[0004] However, shape mapping tasks face numerous challenges. For example, non-equidistant deformation often occurs during the mapping process from the source mesh model to the target mesh model, meaning that the distance ratio between corresponding points changes during the deformation process, making it difficult to accurately determine the correspondence. Furthermore, different mesh models vary in resolution, noise level, and local detail, all of which reduce the stability of shape mapping tasks and make them difficult to adapt to complex environments. Summary of the Invention

[0005] This disclosure provides a method for processing mesh models, a method for training neural networks, an apparatus for processing mesh models, an electronic device, and a computer-readable storage medium.

[0006] This disclosure provides a method for processing a mesh model, the method comprising: acquiring a source two-dimensional image and determining embedding features of the source two-dimensional image based on the source two-dimensional image; acquiring a target two-dimensional image and determining embedding features of the target two-dimensional image based on the target two-dimensional image; determining semantic mapping features based on the embedding features of the source two-dimensional image and the embedding features of the target two-dimensional image, the semantic mapping features indicating the correspondence between pixels in the source two-dimensional image and pixels in the target two-dimensional image; iteratively updating a deformation model based at least in part on the semantic mapping features; and deforming and rendering the source mesh model based on the updated deformation model to obtain a target mesh model that incorporates the geometric information of the source mesh model.

[0007] This disclosure provides a method for training a neural network. The neural network model includes a mapping module and a deformation module. The mapping module is equipped with a first feature extraction network, a second feature extraction network, a first adaptation network, and a second adaptation network. The method includes: determining preliminary features of a source two-dimensional image sample using the first feature extraction network based on the source two-dimensional image sample; determining embedding features of the source two-dimensional image sample using the first adaptation network based on the preliminary features of the source two-dimensional image sample; determining preliminary features of a target two-dimensional image sample using the second feature extraction network based on the target two-dimensional image sample; determining embedding features of the target two-dimensional image sample using the second adaptation network based on the preliminary features of the target two-dimensional image sample; and training the first adaptation network and the second adaptation network based on the embedding features of the source two-dimensional image and the embedding features of the target two-dimensional image.

[0008] This disclosure provides an apparatus for processing a mesh model, the apparatus comprising: a mapping module configured to: acquire a source two-dimensional image and determine embedding features of the source two-dimensional image based on the source two-dimensional image; acquire a target two-dimensional image and determine embedding features of the target two-dimensional image based on the target two-dimensional image; and determine semantic mapping features based on the embedding features of the source two-dimensional image and the embedding features of the target two-dimensional image, the semantic mapping features indicating the correspondence between pixels in the source two-dimensional image and pixels in the target two-dimensional image; and a deformation module configured to: iteratively update a deformation model at least partially based on the semantic mapping features, and deform and render a source mesh model based on the updated deformation model to obtain a target mesh model that incorporates the geometric information of the source mesh model.

[0009] This disclosure provides an electronic device, including: one or more processors; and one or more memories, wherein the memories store a computer-executable program, and when the processor executes the computer-executable program, the above-described method is performed.

[0010] This disclosure provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the above-described method.

[0011] According to another aspect of this disclosure, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable medium and executes the computer instructions, causing the computer device to perform the methods provided in the foregoing aspects or various alternative implementations of the foregoing aspects.

[0012] Given a source mesh model and a target mesh model, embodiments of this disclosure use a mapping scheme based on two-dimensional images to register the source mesh model to the target mesh model without manual annotation, thereby achieving accurate deformation from the source mesh model to the target mesh model. Attached Figure Description

[0013] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. The accompanying drawings in the following description are merely exemplary embodiments of this disclosure.

[0014] Figure 1 This is an example schematic diagram illustrating a scenario according to an embodiment of the present disclosure.

[0015] Figure 2 A schematic diagram of an example shape corresponding to a task according to an embodiment of the present disclosure is shown.

[0016] Figure 3 A flowchart of a method for processing a mesh model according to an embodiment of the present disclosure is shown.

[0017] Figure 4 A schematic diagram of a neural network model for performing a shape correspondence task according to an embodiment of the present disclosure is shown.

[0018] Figure 5 A schematic diagram of a training method for an improved neural network model according to an embodiment of the present disclosure is shown.

[0019] Figure 6 A schematic diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0020] Figure 7 A schematic diagram of the architecture of an exemplary computing device according to an embodiment of the present disclosure is shown.

[0021] Figure 8 A schematic diagram of a storage medium according to an embodiment of the present disclosure is shown. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this disclosure more apparent, exemplary embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments of this disclosure. It should be understood that this disclosure is not limited to the exemplary embodiments described herein.

[0023] In this specification and accompanying drawings, operations and elements that are substantially the same or similar are indicated by the same or similar reference numerals, and repeated descriptions of these operations and elements are omitted. Furthermore, in the description of this disclosure, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance or order.

[0024] To facilitate the description of this disclosure, the following concepts related to this disclosure are introduced.

[0025] Optionally, the models used in embodiments of this disclosure as described below can all be artificial intelligence models, especially artificial intelligence-based neural network models. Typically, artificial intelligence-based neural network models are implemented as acyclic graphs, where neurons are arranged in different layers. Generally, a neural network model includes an input layer and an output layer, separated by at least one hidden layer. The hidden layer transforms the input received by the input layer into a representation useful for generating the output in the output layer. Nodes are fully connected to nodes in adjacent layers via edges, and there are no edges between nodes within each layer. Data received at nodes in the input layer of the neural network is propagated to nodes in the output layer via any of the hidden layers, activation layers, pooling layers, convolutional layers, etc. The input and output of the neural network model can take various forms, and this disclosure does not limit this.

[0026] The solutions provided in this disclosure involve technologies such as artificial intelligence and / or machine learning, which are specifically illustrated through the following embodiments.

[0027] First, refer to Figure 1 The present disclosure describes application scenarios of methods and apparatuses for processing mesh models according to embodiments of the present disclosure. Figure 1 A schematic diagram of an application scenario 100 according to an embodiment of the present disclosure is shown, wherein a server 110 and a plurality of terminals 120 are schematically illustrated.

[0028] The neural network model of this disclosure can be integrated into various electronic devices, for example. Figure 1The neural network model can be integrated into any electronic device in server 110 and multiple terminals 120. For example, the neural network model can be integrated into terminal 120. Terminal 120 can be a mobile phone, tablet, laptop, desktop computer, personal computer (PC), smart speaker, or smartwatch, but is not limited to these. Alternatively, the neural network model can also be integrated into server 110. Server 110 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminals and servers can be directly or indirectly connected via wired or wireless communication, and this disclosure does not impose any limitations.

[0029] It is understood that the apparatus for inference using the neural network model of the embodiments of this disclosure can be a terminal, a server, or a system composed of a terminal and a server. The method for processing the grid model of the embodiments of this disclosure can be executed on a terminal, on a server, or jointly by a terminal and a server.

[0030] The artificial intelligence model provided in this disclosure can also relate to artificial intelligence cloud services in the field of cloud technology. It is worth noting that both the terminal 110 and server 120 according to the embodiments of this disclosure adhere to data protection principles, respect users' data rights, and ensure user data security and privacy. The terminal 110 and server 120 according to the embodiments of this disclosure will clearly inform users of the purpose, method, and scope of collecting, using, storing, transmitting, and deleting user data, and obtain user consent. The terminal 110 and server 120 according to the embodiments of this disclosure will take reasonable technical and management measures to prevent user data from being leaked, tampered with, damaged, or lost. The providers of the terminal 110 and server 120 according to the embodiments of this disclosure will regularly review and update user data, and promptly delete expired or useless data. Furthermore, cloud service providers using the embodiments of this disclosure respect users' rights to data access, correction, deletion, withdrawal of consent, complaints, and claims, and provide convenient channels and procedures to enable users to effectively exercise these rights.

[0031] Furthermore, the process of data analysis using artificial intelligence technology in terminal 120 or server 110 is conducted based on the principles of legality, rationality, and transparency. The data collected and processed by the artificial intelligence model according to embodiments of this disclosure is relevant, necessary, and appropriate for the predictive purpose, and does not contain any personally identifiable or sensitive information. The neural network model according to embodiments of this disclosure employs appropriate techniques and organizational measures to protect the security and integrity of data, preventing unauthorized access, use, or disclosure.

[0032] The artificial intelligence-based neural network model according to embodiments of this disclosure will comply with relevant data protection regulations and ethical principles. This neural network model is trained on a large amount of anonymized and de-identified data, and does not infringe on the privacy rights of any individual or group. The artificial intelligence model has also undergone rigorous testing and evaluation to ensure that its output results are accurate and reliable, and will not cause any misleading or discriminatory behavior. The artificial intelligence model is designed solely to improve service quality and customer satisfaction and will not be used for any illegal or unethical purposes. Furthermore, the neural network model will be regularly reviewed and updated to adapt to changes in the data environment and legal regulations.

[0033] Shape correspondence is an important research topic in computer vision and graphics. Its main purpose is to construct a point-to-point mapping between a source mesh model and a target mesh model, so as to preserve the geometric features of the source mesh model as completely as possible when generating the target mesh model. Currently, there are two main traditional methods for implementing shape correspondence: functional map methods and registration-for-correspondence methods.

[0034] Function mapping has been widely used in recent years, transforming complex point mapping problems into function mapping problems. In "controlled" scenarios, such as simulations of industrial product design, where the topological differences between mesh models are small, function mapping can quickly and accurately find shape correspondences. However, because function mapping relies on strictly aligned low-rank bases, it performs poorly when dealing with non-equidistant correspondences.

[0035] Registration and mapping methods iteratively deform the source mesh model, gradually aligning it precisely with the target mesh model and generating a dense correspondence. However, these methods often lead to unstable deformation, resulting in distortion artifacts and making it difficult to find an ideal balance between smoothness and deformation accuracy. Furthermore, registration and mapping methods heavily rely on manually added initial alignment or sparse 3D correspondence annotations. When the mesh model undergoes significant non-equidistant deformation, manually adding such annotations is often difficult, leading to a significant degrade in the performance of registration and mapping methods.

[0036] Therefore, both traditional methods for achieving shape correspondence tasks face problems such as unstable deformation and high requirements for manual annotation, which need to be further improved.

[0037] This disclosure uses a registration and correspondence method as the basic paradigm. Given a source mesh model and a target mesh model, this disclosure uses a two-dimensional image-based mapping method to register the source mesh model to the target mesh model without manual annotation, thereby achieving accurate deformation from the source mesh model to the target mesh model.

[0038] In one aspect of the embodiments of this disclosure, the two-dimensional image-based mapping model according to this disclosure can generate stable semantic mapping features. Even in scenarios with significant changes in the topology, shape, and pose of the mesh model, these semantic mapping features exhibit strong robustness, making them more suitable for applications involving differentiable rendering and iterative optimization.

[0039] In one aspect of the embodiments of this disclosure, during the training process of the two-dimensional image-based mapping model according to this disclosure, the two-dimensional image-based mapping model can learn how to construct accurate pixel-to-pixel correspondences in two dimensions through negative loss, avoiding erroneous correspondences caused by the morphological symmetry between the source mesh model and the target network in traditional schemes.

[0040] This disclosure proposes an improved neural network model. The improved neural network model includes at least a mapping model and a deformation model. The mapping model generates semantic mapping features based on a two-dimensional image. The deformation model iteratively updates the semantic mapping features to obtain a target mesh model. However, this disclosure is not limited thereto.

[0041] Therefore, this disclosure utilizes semantic mapping features that indicate the correspondence between pixels in a source 2D image and pixels in a target 2D image to guide the deformation process performed by the deformation model, thereby achieving deformation from the source mesh model to the target mesh model. Through iterative optimization, the deformation model progressively deforms the source mesh model to align with the shape of the target mesh model. Thus, embodiments of this disclosure, through the collaborative work of the mapping model and the deformation model, can achieve accurate deformation from the source mesh model to the target mesh model even under non-equidistant conditions.

[0042] For an improved neural network model, embodiments of this disclosure provide a method for processing a mesh model to perform inference based on such an improved neural network model. The method includes: acquiring a source two-dimensional image and determining embedding features of the source two-dimensional image based on the source two-dimensional image; acquiring a target two-dimensional image and determining embedding features of the target two-dimensional image based on the target two-dimensional image; determining semantic mapping features based on the embedding features of the source two-dimensional image and the embedding features of the target two-dimensional image, the semantic mapping features indicating the correspondence between pixels in the source two-dimensional image and pixels in the target two-dimensional image; iteratively updating a deformation model at least partially based on the semantic mapping features; and deforming and rendering the source mesh model based on the updated deformation model to obtain a target mesh model that incorporates the geometric information of the source mesh model.

[0043] In the process of reasoning using an improved neural network, the embodiments of this disclosure register the source mesh model to the target mesh model without manual annotation through a mapping model based on a two-dimensional image, thereby achieving accurate deformation from the source mesh model to the target mesh model and significantly improving the performance of shape correspondence tasks.

[0044] Similarly, for the improved neural network model, this disclosure also provides a method for training the neural network to train the improved neural network model. The neural network model includes a mapping module and a deformation module. The mapping module is equipped with a first feature extraction network, a second feature extraction network, a first adaptation network, and a second adaptation network. The method includes: determining preliminary features of a source two-dimensional image sample using the first feature extraction network based on the source two-dimensional image sample; determining embedding features of the source two-dimensional image sample using the first adaptation network based on the preliminary features of the source two-dimensional image sample; determining preliminary features of a target two-dimensional image sample using the second feature extraction network based on the target two-dimensional image sample; determining embedding features of the target two-dimensional image sample using the second adaptation network based on the preliminary features of the target two-dimensional image sample; and training the first adaptation network and the second adaptation network based on the embedding features of the source two-dimensional image and the embedding features of the target two-dimensional image.

[0045] During the training phase, this embodiment of the present disclosure uses symmetry loss to learn how to construct accurate pixel-to-pixel correspondences in two dimensions, avoiding erroneous correspondences caused by the morphological symmetry between the source mesh model and the target network in traditional schemes.

[0046] The following reference Figures 2 to 8 Methods for processing mesh models, methods for training neural networks, and apparatus according to embodiments of the present disclosure are described.

[0047] Figure 2 A schematic diagram of an example shape corresponding to a task according to an embodiment of the present disclosure is shown.

[0048] Shape mapping tasks can be widely applied to various scenarios such as topology re-conversion, attribute transfer, and shape interpolation.

[0049] For example, in retopology scenarios, shape correspondence tasks can be used to determine the correspondence between mesh models of different topological structures, transforming complex source topologies into target topologies more suitable for specific application needs, thereby improving the naturalness of 3D modeling. Figure 2 In the retopological scene shown, the input source mesh model is a cartoon bear doll with a regular grid pattern on its surface. The target mesh model is a cartoon fox doll with different body shapes and ear shapes. Through shape mapping, the grid pattern of the source mesh model is accurately mapped to the target mesh model, ensuring that the fox doll's surface displays a regular and natural grid arrangement, adapting to its unique shape characteristics.

[0050] For example, in attribute transfer scenarios, once the correspondence between the source and target mesh models is determined, attributes such as color and texture from the source mesh model can be transferred to the target mesh model, thus achieving rapid attribute assignment. The input source mesh model is a 3D clothing mesh model with a regular checkered pattern, while the target mesh model is a 3D clothing mesh model with a different cut. Through shape mapping, the checkered pattern on the source mesh model can be correctly mapped to the target mesh model, ensuring its regular arrangement and natural deformation on the target clothing.

[0051] For example, in shape interpolation scenarios, interpolation calculations are performed based on the correspondence between multiple mesh models to generate a series of transition mesh models to support animation production and deformation effects. The source mesh model is a model of a smiling animated character, whose facial texture includes details such as beard and freckles. The target mesh model is a model of the same character's crying expression. Through shape correspondence tasks, facial textures are accurately transferred, and freckles and beards can naturally adapt to different expression changes, providing stable facial detail transitions for animation production.

[0052] Figure 3 A flowchart of a method 30 for processing a mesh model according to an embodiment of the present disclosure is shown.

[0053] Method 30 can be used on terminal devices or servers (such as...) Figure 1 The method is executed at the terminal 120 or server 110. Method 30 includes the following operations S301 to S304. Of course, method 30 may also include more or fewer operations, and this disclosure is not limited thereto.

[0054] In operation S301, a source two-dimensional image is acquired, and the embedding features of the source two-dimensional image are determined based on the source two-dimensional image. In operation S302, a target two-dimensional image is acquired, and the embedding features of the target two-dimensional image are determined based on the target two-dimensional image.

[0055] Optionally, both the source 2D image and the target 2D image can be planar images composed of pixels arranged in rows and columns, with each pixel containing color or grayscale information. For example, the source 2D image can be a 2D image obtained by rendering the source mesh model based on the geometric information of the source and target mesh models from a preset camera viewpoint, such as an RGB image rendered from different viewpoints such as the front view, side view, or top view of the source mesh model. The target 2D image refers to a 2D image obtained by rendering the target mesh model based on the geometric information of the target mesh model from the same preset camera viewpoint as the source 2D image. The geometric information of the source and target mesh models differs from that of the target mesh model. For example, if the source 2D image is a 45-degree angle view of the source mesh model, then the corresponding target 2D image should also be a 45-degree angle view of the source mesh model. For example, given a series of fixed camera viewpoints C... i The data is set to i = 1, 2, ..., K, and a 2D image from the fixed camera viewpoint is rendered as the source 2D image. and target two-dimensional image Of course, this disclosure is not limited to this.

[0056] Optionally, for example, a source mesh model refers to a mesh model used to represent a source 3D object in a data-driven form. This polygonal mesh model includes the 3D coordinate information of multiple vertices and the triangular facet information formed by these vertices. For example, a source mesh model of a cube may contain the spatial coordinates of 8 vertices and the triangular facet information formed by these vertices. A target mesh model refers to a mesh model used to represent a target 3D object in a data-driven form. This mesh model also includes the vertex coordinate information and triangular facet information of the target 3D object. For example, a target mesh model of a sphere may contain the spatial coordinates of several vertices and the triangular facet information formed by these vertices. Of course, this disclosure is not limited thereto.

[0057] Optionally, the goal of the shape mapping task is to generate a target mesh model that retains the geometric information of the source mesh model as completely as possible. For example, accurately mapping the geometric information related to color, edges, topology, and texture in the source mesh model of a cube onto the target mesh model of a sphere.

[0058] Optionally, the geometric information of the source and target mesh models may not be directly adapted to each other, and there may be a potential non-equidistant correspondence. Specifically, the geometric information of a mesh model refers to the geometric shape, texture, and structural information used to characterize the 3D object corresponding to that mesh model. Geometric shape information includes the 3D coordinates of multiple vertices of the mesh model and the triangular facet information formed by connecting these vertices. A non-equidistant correspondence refers to a situation where the source and target mesh models cannot achieve a complete match through simple rigid body transformations (such as translation, rotation, or scaling). For example, the source mesh model might represent a smiling face, while the target mesh model represents a neutral expression on the same face, with complex local differences in geometric deformation. Of course, this disclosure is not limited to this.

[0059] Optionally, the embedding features of the source 2D image refer to the high-dimensional feature vectors that can represent the visual content of the image, extracted from the source 2D image through a feature extraction network and an adaptation network. These high-dimensional feature vectors reside in the embedding space and are therefore also called the embedding features of the source 2D image. Each feature point in the embedding features of the source 2D image can indicate information such as local texture features and global semantic features of the corresponding pixel in the source 2D image, or it can indicate low-level features such as edges, corners, textures, and shapes of the corresponding pixel in the source 2D image, or high-level features such as object categories and scene layouts of the corresponding pixel in the source 2D image. The embedding features of the target 2D image refer to the embedding features that can represent the visual content of the image, extracted from the target 2D image through the same or similar feature extraction network and adaptation network. These embedding features have the same dimension and similar semantic meaning as the features of the source 2D image. Of course, this disclosure is not limited to this.

[0060] For example, in operation S301, preliminary features of the source two-dimensional image can be determined using a first feature extraction network based on the source two-dimensional image; and embedding features of the source two-dimensional image can be determined using a first adaptation network based on the preliminary features of the source two-dimensional image. In operation S302, preliminary features of the target two-dimensional image can be determined using a second feature extraction network based on the target two-dimensional image; and embedding features of the target two-dimensional image can be determined using a second adaptation network based on the preliminary features of the target two-dimensional image. The embedding features of the source two-dimensional image and the embedding image of the target two-dimensional image reside in the same embedding space. Of course, this disclosure is not limited thereto.

[0061] For example, in some aspects of this disclosure, the first feature extraction network and the second feature extraction network may be the same feature extraction network, and this feature extraction network may be pre-trained. That is, the pre-trained first and second feature extraction networks can be used to extract preliminary features of the source 2D image and the target 2D image, and then the preliminary features are adapted by an adaptation network to map the preliminary features into the embedding space to obtain the embedding features X of the source 2D image. src Embedding features X of the target 2D image tgt .

[0062] For example, assuming the first and second feature extraction networks are stable diffusion models, the source and target 2D images can be input into the pre-trained stable diffusion model, respectively. Multi-scale features are extracted from multiple intermediate layers of its UNet encoder as preliminary features of the source 2D image. These preliminary features can contain representations of the image at different semantic levels. As another example, assuming the first and second feature extraction networks are DINO visual transformer models, the source and target 2D images can also be input into the pre-trained DINO visual transformer model, respectively. High-level semantic features are extracted from the output of its last transformation block. For example, for a source 2D image containing a cat, the extracted preliminary features can encode multiple levels of visual information such as the cat's fur texture, ear shape, and body posture. Of course, this disclosure is not limited to these limitations.

[0063] Optionally, after obtaining the features of the source 2D image and the preliminary features of the target 2D image, lightweight adaptation networks with fewer parameters (i.e., the first adaptation network and the second adaptation network) can be used to map these features into a common embedding space. The first and second adaptation networks can be neural networks composed of several fully connected layers and nonlinear activation functions, for example, they can contain two fully connected layers with ReLU activation functions. The first and second adaptation networks are trainable, and their training methods will be described in detail later; this disclosure will not elaborate further here. Through the first and second adaptation networks, the features of the source 2D image and the target 2D image are transformed into the same embedding space. Of course, this disclosure is not limited to this.

[0064] In operation S303, semantic mapping features are determined based on the embedding features of the source two-dimensional image and the embedding features of the target two-dimensional image. The semantic mapping features indicate the correspondence between pixels in the source two-dimensional image and pixels in the target two-dimensional image.

[0065] Optionally, in operation S303, a nearest neighbor search can be performed based on the embedding features of the source 2D image and the target 2D image to determine the semantic mapping features. The semantic mapping features can represent the 2D displacement relationship between pixels in the source image and pixels in the target image in the form of a semantic flow map. This semantic flow map can be viewed as a pixel-level correspondence matrix, where each element describes the corresponding position of a pixel in the source 2D image in the target 2D image. For each camera viewpoint C... i This can generate a corresponding semantic stream mapping S. i To reflect the source two-dimensional image and target two-dimensional image The pixel correspondence between them. Of course, this disclosure is not limited to this.

[0066] Optionally, nearest neighbor search can be achieved by calculating the distance between each feature point in the embedded features of the source 2D image and each feature point in the embedded features of the target 2D image. Specifically, for each feature point in the source 2D image, its distance to all feature points in the target 2D image is calculated, and the target feature point with the smallest distance is considered to be the corresponding point of the source feature point.

[0067] Optionally, the correspondence from feature points can further indicate pixel-to-pixel correspondences. Thus, the semantic mapping features can indicate the correspondence from pixels in the source 2D image to pixels in the target 2D image. For example, each feature point corresponds to a pixel block in the 2D image. Through mapping, the correspondence between feature points is extended to all pixels they cover. In this way, a global correspondence from each pixel in the source 2D image to the corresponding pixel in the target 2D image can be generated, thereby achieving finer-grained matching. By operating S301 to S303, non-equidistant pixel-to-pixel correspondences can be determined without manual matching. Of course, this disclosure is not limited thereto.

[0068] In operation S304, the deformable model is iteratively updated based at least in part on the semantic mapping features.

[0069] Optionally, with the registration-correspondence method as the basic paradigm, the deformation model can be used to describe the geometric transformation relationship between the source mesh model and the target mesh model. Operation S304, by iteratively updating the deformation model, progressively obtains a deformation model that integrates the geometric information of the source mesh model into the target mesh model. In one aspect of this disclosure, the deformation model can be represented and learned using neural networks to capture local and global features in non-rigid deformation, for example, by modeling local deformation gradients during deformation using Neural Jacobian Fields (NJFs), or by using neural implicit functions to represent continuous deformation fields. Of course, this disclosure is not limited thereto.

[0070] When using NJF as the deformation model, one or more embodiments of this disclosure can produce smooth deformation results while preserving details and minimizing distortion. Thereafter, although NJF is used as an example deformation model for description, operation S304 can also be implemented using other deformation models. Of course, this disclosure is not limited thereto.

[0071] In operation S305, based on the updated deformation model, the source mesh model is deformed and rendered to obtain a target mesh model that incorporates the geometric information of the source mesh model.

[0072] For example, a target mesh model that incorporates the geometric information of the source mesh model refers to a mesh model processed by a deformation model. This mesh model retains the topological structure of the target mesh model while inheriting some geometric features of the source mesh model. Specifically, this fusion can be reflected in geometric attributes such as vertex positions, normals, and curvature. For instance, in a task involving a human body model, the deformed model can maintain the topological structure of the target human body while incorporating geometric details such as body shape and pose information from the source model. Of course, this disclosure is not limited to this.

[0073] In summary, for reference Figure 3 One aspect of this disclosure describes how, given a source mesh model and a target mesh model, stable semantic mapping features can be generated based on a two-dimensional image, and the source mesh model can be registered to the target mesh model without manual annotation, thereby achieving accurate deformation from the source mesh model to the target mesh model.

[0074] Specifically, in this aspect, stable semantic mapping features are generated based on operations S301 to S303. Even in scenarios with significant changes in the topology, shape, and orientation of the mesh model, these semantic mapping features exhibit strong robustness to adapt to the differentiable rendering and iterative optimization process in operation S304. In operation S304, the semantic mapping features guide the deformation process, and through iterative optimization, the deformation model progressively deforms the source mesh model to align with the shape of the target mesh model. Therefore, even under non-equidistant conditions, this disclosure can achieve accurate deformation from the source mesh model to the target mesh model.

[0075] Next reference Figure 4 The following describes some optional details of the embodiments of this disclosure. Figure 4 A schematic diagram of a neural network model 40 for performing a shape correspondence task according to an embodiment of the present disclosure is shown, which can be used to implement all or part of the operations according to the embodiments of the present disclosure, such as operations S301 to S304. Of course, the present disclosure is not limited thereto.

[0076] Specifically, the neural network model 40 for performing the shape correspondence task includes, but is not limited to, a mapping module 401 and a deformation module 402. The mapping module 401 carries a mapping model for performing operations S301 to S303. The deformation module 402 carries a deformation model for performing operation S304. Of course, the neural network model 40 may also include more or fewer models or modules, and this disclosure does not impose any limitations on this.

[0077] Optionally, the input to the neural network model 40 performing the shape correspondence task includes the source mesh model (which is shown as...). Figure 4 A 3D cartoon image of a red fox-like creature wearing a brown top and a target mesh model (shown as...) Figure 4 (A 3D cartoon character resembling a white rabbit wearing blue pants). The output objective of neural network model 40 is to fuse the geometric information of the source mesh model into the target mesh model as much as possible, in order to generate a target mesh model that incorporates the geometric information of the source mesh model, for example... Figure 4 Target mesh model after rendering Its location Figure 4 In the bottom right corner, there is a colorful rabbit image. Figure 4 The colorful rabbit image shown is for illustrative purposes only and does not represent the actual rendering result.

[0078] Optionally, in operation S301, the mapping module 401 can acquire a single source 2D image by rendering the source mesh model based on the geometric information of the source target mesh model, based on the camera viewpoint, to obtain a source 2D image corresponding to the camera viewpoint. Simultaneously, in operation S302, the mapping module 401 can also acquire a single target 2D image by rendering the target mesh model based on the geometric information of the target mesh model, based on the camera viewpoint, to obtain a target 2D image corresponding to the camera viewpoint. Although this example only describes the acquisition method of a single 2D image, this example can also be extended to scenarios involving the acquisition of multiple 2D images. This disclosure is not limited thereto.

[0079] For example, we can assume there are K camera viewpoints, each with an index of i. In operations S301 and S302, the mapping module 401 obtains K source 2D images respectively. and K target two-dimensional images Source 2D images and target 2D images with the same index correspond to the same camera viewpoint. Then, the mapping module 401 uses these K source 2D images... and K target two-dimensional images To determine K latent features of the source 2D image and K latent features of the target 2D image.

[0080] Specifically, as described above, in operation S301, the mapping module 401 can be based on the source two-dimensional image. The source 2D image is determined using a feature extraction network 4011. Preliminary features, then based on the source two-dimensional image Preliminary features are obtained by using an adaptation network 4012 to determine the source 2D image. Embedding features In operation S302, the mapping module 401 can be based on the target two-dimensional image. The feature extraction network 4013 is used to determine the preliminary features of the target two-dimensional image, and the source two-dimensional image is determined by the adaptation network. Embedding features

[0081] Next, in operation S303, the mapping module 401 can base its work on the embedding features of the source two-dimensional image. and the embedding features of the target two-dimensional image Perform a nearest neighbor search to determine the semantic mapping feature S i The semantic mapping feature represents the two-dimensional displacement relationship between source image pixels and target image pixels in the form of a semantic flow mapping graph.

[0082] In operation S304, the deformation module 402, in each iteration, can use the deformation model (NJF) to estimate the positions of vertices in the source mesh model after deformation to the target mesh model; based on the positions Φ of the vertices in the source mesh model after deformation to the target mesh model... * Determine the displacement corresponding to the vertex; based on the displacement F corresponding to the vertex i Determine the estimated semantic mapping features and the semantic mapping features based on the estimation. With the semantic mapping feature S i Difference L flow Update the parameters in the deformable model.

[0083] The following is a brief explanation of the NJF as a deformation model. Specifically, the per-face Jacobian matrix J of the NJF can be defined by formula (1). i .

[0084]

[0085] Among them, v k v j v i These are the positions of the three original vertices of triangle i in the source 3D mesh model, φ k φ j φ i J represents the positions of three deformable vertices in the target 3D mesh model. i ∈R 3×2 It is a surface Jacobian matrix, B i ∈R 3×2 These are two column vectors that form an orthogonal basis of the tangent space of the surface. This formula also defines a gradient operator, as shown in formula (2).

[0086]

[0087] Where Φ∈R n×3 The position of the vertex after one or more iterations is the position of the deformed vertex. Solving for the position of the deformed vertex involves the Poisson equation. For example, the Poisson equation defines the position Φ of the deformed vertex using formula (3). * .

[0088]

[0089] Among them, |t i | represents the region where triangular facet i is located. This Poisson method can be solved using the least squares method according to formula (4).

[0090]

[0091] Where L is the coche Laplacian operator of the mesh model, A is the mass matrix of the mesh model, and J is the estimated Jacobian matrix J0. i The stack. During registration, the transformation matrix of each face is... For optimization, these matrices are projected onto the tangent space B of each face. i Above, through Obtain the Jacobian matrix J i Finally, formula (4) is used to solve for the final vertex position Φ. * The core idea of ​​formulas (1) to (4) is to define the deformation in a compact, parameterized tangent space, which simplifies the optimization process compared to operating in the environment space. Of course, this disclosure is not limited thereto.

[0092] According to NJF, given the position Φ of the deformed vertex * The vertex can be projected according to the corresponding camera view and the displacement F of that vertex can be calculated. i This process can be described by formula (5).

[0093] F i =Π(Φ * C i )-Π(V src C i (5)

[0094] Where, Φ * It refers to the position of the vertex of the triangular facet in the source mesh model after NJF deformation, while V src It refers to the position of the vertices of the triangular facets in the source mesh model, C. i Let F be the viewpoint of the i-th camera, and Π be the projection operator used to project the coordinates of the vertex onto a 2D coordinate system. Specifically, F... i This could be the displacement of the deformed vertex relative to the vertices of the triangular facet in the source mesh model. F i It is the spatial change of the vertices of the mesh model during the shape deformation process, which describes how the source vertex moves to the target position in the deformed mesh model.

[0095] Then, given the displacement F i and camera perspective C i Then, the source 3D mesh model M can be processed. src Shading is performed to obtain a differentiable semantic flow map. Shading is the process of assigning color to vertices based on certain properties (such as position, normal, texture, or displacement). For example, NVDiff can be used for shading and rendering. NVDiff is a tool that supports differentiable rendering, capable of efficiently rendering differentiable semantic flow maps. It also allows the calculation of the loss gradient during backpropagation. This process can be described by equation (6).

[0096]

[0097] That is, formula (6) describes the use of the normalized two-dimensional displacement as the vertex color, combined with the differentiable rendering function R(·) to render the source mesh model M. src From camera perspective C i The captured 2D image is rendered (colored) to obtain a differentiable semantic flow map. The process.

[0098] Due to differentiable semantic flow map It is NJF based on the source 3D mesh model M src Camera perspective C i and displacement F i The estimated value is closer to the semantic mapping feature S determined in operation S303. i This indicates that the more fully the geometric information of the source mesh model is reflected in the deformed target mesh model, the better. Therefore, an optional objective of the iteration in operation S304 is to map the differentiable semantic flow graph. In operation S303, the semantic mapping feature S is determined. i The difference in the loss converges to a minimum, which is subsequently named the semantic flow loss. Specifically, this process can be illustrated by equation (7).

[0099]

[0100] Therefore, in each iteration, the semantic mapping features obtained in operation S303 can be used to guide the iteration of parameters in the deformation model NJF, thereby enabling the deformation model NJF that finally performs deformation to fuse the geometric information of the source mesh model with the target mesh model, thus obtaining the target mesh model that fuses the aggregated information of the source mesh model in operation S305.

[0101] Alternatively, in addition to using semantic mapping features to guide the iteration of parameters in the deformable model NJF, other information (such as geometric alignment loss, anti-shear loss, etc., detailed below) can be combined with semantic mapping features to guide the iteration of parameters in the deformable model NJF.

[0102] Specifically, in one aspect of this disclosure, in operation S304, in each iteration, the deformation model can be used to estimate the position of the vertices in the source mesh model after deformation to the target mesh model; based on the position of the vertices in the source mesh model after deformation to the target mesh model, the displacement corresponding to the vertex is determined; based on the displacement corresponding to the vertex, the mesh model after deformation by the deformation model is determined; and based on the difference between the deformed mesh model and the target mesh model, the parameters in the deformation model are updated.

[0103] Optionally, the difference between the deformed mesh model and the target mesh model is in Figure 4 The loss L is shown as CD. cd First, based on the vertex displacements described above, the deformed mesh model M can be obtained. deformed Then calculate M. deformed Vertex and target mesh model M target Chamfer distance loss between vertices L cd The Chamfer distance loss measures the geometric difference between the two and can effectively align the mesh model shape. Therefore, one possible direction for iteratively updating the deformable model is to further minimize / converge the Chamfer distance loss. Alternatively, the Chamfer distance loss can be replaced by different distance metrics (such as Euclidean distance or Manhattan distance) to address different types of shape-corresponding tasks.

[0104] Specifically, in one aspect of this disclosure, in operation S304, in each iteration, the deformation model can be used to estimate the position of vertices in the source mesh model after deformation to the target mesh model; based on the position of the vertices in the source mesh model after deformation to the target mesh model, the displacement corresponding to the vertex is determined; based on the displacement corresponding to the vertex, the mesh model after deformation by the deformation model is determined; the deformed mesh model is rendered differently to determine the normal map corresponding to the deformed mesh model; the normal map corresponding to the target mesh model is determined; and the parameters in the deformation model are updated based on the difference between the normal map corresponding to the deformed mesh model and the normal map corresponding to the target mesh model.

[0105] Optionally, the difference between the normal map corresponding to the deformed mesh model and the normal map corresponding to the target mesh model is in Figure 4 The loss L is shown as the normal loss. normal The normal map represents the normal vector of each point on the surface of the mesh model, that is, the vector perpendicular to the surface at that point. Then, these normals are calculated to the target mesh model M. targetThe difference between the normals of corresponding points. Optionally, this difference can be measured using point-to-point angle difference or cosine similarity. In one aspect of this disclosure, differentiable rendering techniques can be used to render M. deformed The normal plot, and minimizing the normal loss L. normal Or cause a loss of normal L normal Convergence is achieved to further optimize the mesh model shape alignment. This is achieved through the normal loss L... normal This allows the deformed mesh model surface to be as close as possible to the target mesh model in the normal direction, thereby improving the matching degree of surface details.

[0106] CD loss L cd and normal loss L normal Both belong to the category of geometry alignment loss, and both aim to correct the deformation of the source mesh model into a new mesh model M. deformed Align the shape with the target mesh model. Of course, other geometric alignment losses can also be used, and this disclosure is not limited thereto.

[0107] Specifically, in one aspect of this disclosure, in operation S304, in each iteration, the Jacobian matrix of each face of the vertices in the source mesh model after deformation to the target mesh model can be estimated using the deformation model; the displacement corresponding to the vertex is determined based on the position of the vertex in the source mesh model after deformation to the target mesh model; and the mesh model M after deformation by the deformation model is determined based on the displacement corresponding to the vertex. deformed Determine the Jacobian matrix of each face in the deformed mesh model. And based on the difference between the Jacobian matrix and the identity matrix I3 of each face in the deformed mesh model, the parameters in the deformed model are updated.

[0108] Specifically, the difference between the Jacobian matrix and the identity matrix I3 of each face in the deformed mesh model is called the identity-preserving loss. This loss is used to constrain the transformation matrix of each face in the mesh model to further increase the stability of the shape during deformation. This identity-preserving loss L... identity The expression is:

[0109]

[0110] in, It is a mesh model M deformedThe Jacobian matrix of the i-th face (or triangular facet). I3 is a 3×3 identity matrix representing the ideal case without deformation. F represents the number of faces in the mesh model. The deformation process applies a transformation to every face in the mesh model, therefore the identity preservation loss L... identity The transformation matrix of each face needs to be summed. By regularizing the transformation matrix of each face, the identity preservation term ensures that the mesh model does not exhibit excessive distortion during deformation, especially during large-scale deformations, effectively avoiding unnatural shape transformations. The optimization goal of this term is to make the transformation matrix of each face after deformation as close as possible to the identity matrix, thereby maintaining the smoothness and naturalness of the deformation.

[0111] However, using only the identity preservation loss is insufficient to solve all over-deformation problems. Applying a large weighting factor limits a wide range of pose deformations, while a small weighting factor may lead to uneven deformation. Therefore, in one aspect of this disclosure, in operation S304, in each iteration, the deformation model can be used to estimate the Jacobian matrix of each face of the source mesh model after the vertices are deformed into the target mesh model; the displacement corresponding to the vertex is determined based on the position of the vertex after the source mesh model is deformed into the target mesh model; the mesh model after deformation by the deformation model is determined based on the displacement corresponding to the vertex; the Jacobian matrix of each face in the deformed mesh model and the rotation component of the Jacobian matrix after polar decomposition are determined; and the parameters in the deformation model are updated based on the difference between the Jacobian matrix of each face in the deformed mesh model and the corresponding rotation component.

[0112] Specifically, the difference between the Jacobian matrix and the corresponding rotation component of each face in the deformed mesh model can be referred to as shear-resistant loss. Shear-resistant loss is an important term used in the regularization deformation process, aiming to suppress excessive shear deformation during deformation and ensure the smoothness and naturalness of the mesh model deformation. Shear deformation refers to unbalanced stretching or compression of the mesh model during deformation, leading to shape distortion, especially when the object undergoes non-uniform deformation or high-frequency detail processing.

[0113] Shear loss resistance L shear By constraining the rotational components of the deformation matrix, it is ensured that deformation does not occur solely through non-rigid shear deformation. Specifically, the shear loss L... shear The format is as follows:

[0114]

[0115] in, It is a mesh model M deformed Jacobian matrix of the i-th face (or triangular facet). yes The rotation component is extracted through polar decomposition. Polar decomposition decomposes an arbitrary matrix into two components: a rotation component and a positive definite transformation component. The rotation component describes only pure rotation, without including shearing or scaling deformation. This part reflects local orientation changes on the surface, but does not include stretching or compression of the shape. Therefore, This value can be used to represent the loss of surface F due to shearing; the smaller the better. Through this regularization method, shear deformation during the deformation process is effectively suppressed, thereby ensuring a smoother and more stable mesh model deformation. Of course, this disclosure is not limited to this.

[0116] In one aspect of this disclosure, the aforementioned losses can be synthesized, with the final loss function serving as the update target for operation S304, to ensure that deformation and geometric alignment in the shape correspondence task can be performed smoothly and naturally. This final loss function is formulated as follows:

[0117] L=λ flow L flow +λ cd L cd +λ normal L normal +λ identity L identity +λ shear L shear (10)

[0118] The various losses have been described above. This loss function balances and optimizes the constraints during the deformation process by minimizing the weighted sum of these losses. The transformation matrix on each surface is then iteratively optimized. After final optimization convergence, a deformed model with optimal parameters can be obtained. Optionally, the weights of each loss term can be adjusted according to task requirements to achieve the best results in different scenarios. Of course, this disclosure is not limited to this.

[0119] In one aspect of this disclosure, the application of the final loss function optimizes the shape deformation process by minimizing the weighted sum of each sub-loss term, particularly in non-equidistant correspondence tasks. Specifically, after optimization convergence, the registration result Φ can be obtained using the aforementioned formula (4). final To extract the 3D correspondence, it is first necessary to examine each deformed vertex. Find the target mesh model M tgt The closest face. Then, Project onto the nearest surface and calculate the centroid coordinates, thus projecting onto the source mesh model M. src and target mesh model M tgt A final mapping relationship is established between them. This process ensures the accurate mapping and position adjustment of each vertex after deformation in non-isometric correspondence tasks. Of course, this disclosure is not limited thereto.

[0120] In one aspect of this disclosure, the design of the final loss function fully considers the non-isotropic deformation problem in shape-corresponding tasks. By combining multiple loss terms, various challenges such as semantic alignment, geometric alignment, normal consistency, and shearing suppression can be addressed while ensuring the smoothness and stability of the deformation process. For example, when performing facial expression transfer, the final loss function not only requires accurate matching of the vertex positions of the mesh model but also needs to ensure the natural deformation of the facial expression and the consistency of the normal direction, avoiding unnatural distortion of the facial expression. By optimizing this loss function, a high degree of consistency in shape and surface details can be ensured in complex deformation tasks. Of course, the loss terms can also be customized according to specific application requirements to handle different types of deformations and corresponding tasks.

[0121] Next, refer to Figure 5 The following describes some optional details of the embodiments of this disclosure. Figure 5 A schematic diagram of a training method 5 for an improved neural network model according to an embodiment of the present disclosure is shown.

[0122] The improved neural network model training method 50 according to embodiments of the present disclosure includes the following operations S501 to S505. Of course, method 50 may also include more or fewer operations, and the present disclosure is not limited thereto.

[0123] In operation S501, based on the source two-dimensional image sample, the preliminary features of the source two-dimensional image sample are determined using a first feature extraction network.

[0124] In operation S502, based on the preliminary features of the source two-dimensional image sample, the embedding features of the source two-dimensional image sample are determined by the first adaptation network.

[0125] In operation S503, based on the target two-dimensional image sample, the preliminary features of the target two-dimensional image sample are determined using a second feature extraction network.

[0126] In operation S504, based on the preliminary features of the target two-dimensional image sample, the embedding features of the target two-dimensional image sample are determined by the second adaptation network.

[0127] In operation S505, the first adaptation network and the second adaptation network are trained based on the embedding features of the source two-dimensional image and the embedding features of the target two-dimensional image.

[0128] As mentioned earlier, the first and second feature extraction networks are pre-trained, and this disclosure does not require further training of them. The training process only involves the first and second adaptation networks. Typically, the first and second adaptation networks are trained jointly.

[0129] Operation S505 may optionally include: sampling matching source 2D points and target 2D points from the embedding features of the source 2D image and the embedding features of the target 2D image; determining the distance between the source 2D points and the target 2D points based on the matching source 2D points and the target 2D points; and adjusting the parameters in the first adaptation network and the second adaptation network based on the distance between the source 2D points and the target 2D points. This disclosure is not limited thereto.

[0130] Specifically, in one aspect of this disclosure, a pair of mesh model samples from a set of 3D mesh model samples are randomly sampled as a source mesh model sample and a target mesh model sample, and the camera pose (C) is randomly sampled based on the prior camera distribution. s C r To render a pair of images (I) src ,I tgt () serves as the source 2D image sample and the target 2D image sample. This random sampling process has a 50% probability that the pose C of the source camera and the target camera will be different. r =C t Of course, other probabilities can be used to make the source camera and target camera poses equal, and this disclosure is not limited thereto. Furthermore, the rendered pair of images (I...) src ,I tgt This can be a normal image or an RGB image. By projecting the true 3D correspondence onto the image plane and utilizing camera pose information, a matching image from the I... src The point and from I tgt The points will be monitored. Of course, this disclosure is not limited to this.

[0131] Specifically, in another aspect of this disclosure, source two-dimensional image samples and target two-dimensional image samples can also be obtained from a two-dimensional image dataset. For example, source two-dimensional images and target two-dimensional image pairs can be uniformly sampled and fed into a first (second) feature extractor and a first (second) adaptation network, respectively, to generate corresponding embedding vectors (X). src ,X tgt Of course, this disclosure is not limited to this.

[0132] Regardless of whether the source and target 2D image samples are obtained by sampling a 3D mesh model sample set or a 2D image dataset, the first and second adaptation networks can be trained using contrastive loss. Contrastive loss encourages the embedding vectors of matching points in the source and target images to be close, thereby achieving accurate point-to-point matching. Specifically, this contrastive loss L... con It can be formalized as follows.

[0133] L con =CL(X) src (P s ),X tgt (P t (11)

[0134] Where CL is a contrast loss similar to CLIP, and P s and P t It is a matching pair of two-dimensional points, X Src (·) and X tgt (·) indicates that from X src and X tgt The process of sampling feature points.

[0135] However, relying solely on contrastive loss may introduce a self-similarity problem, where highly similar regions (such as left and right hands) may be incorrectly mapped to the same features. To address this issue, one aspect of this disclosure also proposes that, in operation S505, a first vertex may be sampled from the source mesh model corresponding to the source 2D image sample; a second vertex may be sampled from the target mesh model corresponding to the target 2D image sample, wherein the second vertex has a pseudo-geodesic distance greater than a predetermined threshold with the first vertex; and the parameters in the first and second adaptation networks may be adjusted using the vertex pair consisting of the first and second vertices as negative samples.

[0136] If the P matched in formula (11) S and P t If a sample is considered a positive sample, then a pseudo-geodetic distance greater than a predetermined threshold can be considered a negative sample. The loss calculated based on this negative sample can be called the geometry-grounded negative loss, which helps to further distinguish geometrically similar but substantially dissimilar parts in the source and target mesh models.

[0137] For example, for each vertex in the source mesh model, the geometry-guided negative loss avoids mismatches by selecting vertices with greater geographic distance in the target mesh model as negative samples. Specifically, the negative loss, by introducing a pseudo-geodesic distance matrix G, enables efficient computation of the geographic distance between two mesh models on a shared mesh model template. A shared mesh model template is a standardized, unified representation of a mesh model, typically used to provide geometric alignment and mapping between different mesh models. As a "reference model," the shared mesh model template usually has a consistent geometry and can be used to transform multiple source or target mesh models to this standard mesh model through parameterization or some form of mapping, facilitating comparison, pairing, and alignment. The negative loss function L... neg This can be expressed as:

[0138] L neg =∑ (p,q),G(p,q)＞th ||X src (Π(p,C s ))·X tgt (Π(q,C t ))||2 (12)

[0139] Where G is the pseudo-geodetic distance matrix pre-computed in the shared mesh model template, th is the threshold distance, (p, q) are vertices randomly sampled from the source and target mesh models, respectively, and Π(·) represents the projection operation. The geometry-guided negative loss alleviates the self-similarity problem, ultimately achieving a more robust shape correspondence. Therefore, the total loss used to train the first and second fitting models is defined as: L 2D =L con +λ neg L neg , where λ neg It is a hyperparameter that controls the weight of the negative loss and is used to balance the relationship between the geometrically guided negative loss and the contrastive loss.

[0140] According to another aspect of this disclosure, an electronic device is also provided for implementing the methods according to embodiments of this disclosure. Figure 6 A schematic diagram of an electronic device 2000 according to an embodiment of the present disclosure is shown.

[0141] like Figure 6 As shown, the electronic device 2000 may include one or more processors 2010 and one or more memories 2020. The memories 2020 store computer-readable code that, when executed by the one or more processors 2010, can perform the methods described above.

[0142] The processor in this disclosure embodiment can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, operations, and logic block diagrams disclosed in this disclosure embodiment. The general-purpose processor can be a microprocessor or any conventional processor, and can be based on an x86 architecture or an ARM architecture.

[0143] In general, the various exemplary embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of embodiments of this disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0144] For example, the method or apparatus according to embodiments of this disclosure can also be used by means of Figure 7 The architecture of the computing device 3000 shown is used for implementation. For example... Figure 7 As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage devices in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and / or communication of the methods provided in this disclosure, as well as program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Figure 7 The architecture shown is merely exemplary and can be omitted as needed when implementing different devices. Figure 7 One or more components in the computing device shown.

[0145] According to another aspect of this disclosure, a computer-readable storage medium is also provided. Figure 8 A schematic diagram of a storage medium 4000 according to the present disclosure is shown.

[0146] like Figure 8As shown, the computer storage medium 4020 stores computer-readable instructions 4010. When the computer-readable instructions 4010 are executed by a processor, the methods according to embodiments of the present disclosure described with reference to the above figures can be performed. The computer-readable storage medium in the embodiments of the present disclosure may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that the memory used in the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0147] This disclosure also provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform a method according to an embodiment of this disclosure.

[0148] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0149] In general, the various exemplary embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of embodiments of this disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0150] The exemplary embodiments of this disclosure described in detail above are merely illustrative and not restrictive. Those skilled in the art will understand that various modifications and combinations of these embodiments or their features can be made without departing from the principles and spirit of this disclosure, and such modifications should fall within the scope of this disclosure. For example, the following provides an overview of some aspects of this disclosure, which can be combined with any other aspects.

[0151] Aspect 1: A method for processing a mesh model is provided, the method comprising: acquiring a source two-dimensional image and determining embedding features of the source two-dimensional image based on the source two-dimensional image; acquiring a target two-dimensional image and determining embedding features of the target two-dimensional image based on the target two-dimensional image; determining semantic mapping features based on the embedding features of the source two-dimensional image and the embedding features of the target two-dimensional image, the semantic mapping features indicating the correspondence between pixels in the source two-dimensional image and pixels in the target two-dimensional image; iteratively updating a deformation model based at least in part on the semantic mapping features; and deforming and rendering the source mesh model based on the updated deformation model to obtain a target mesh model that incorporates the geometric information of the source mesh model.

[0152] Aspect 2: Determining the embedding features of the source two-dimensional image based on the source two-dimensional image includes: determining preliminary features of the source two-dimensional image using a first feature extraction network based on the source two-dimensional image; and determining the embedding features of the source two-dimensional image using a first adaptation network based on the preliminary features of the source two-dimensional image; determining the embedding features of the target two-dimensional image based on the target two-dimensional image includes: determining preliminary features of the target two-dimensional image using a second feature extraction network based on the target two-dimensional image; and determining the embedding features of the target two-dimensional image using a second adaptation network based on the preliminary features of the target two-dimensional image, wherein the embedding features of the source two-dimensional image and the embedding image of the target two-dimensional image are located in the same embedding space.

[0153] Aspect 3: The acquisition of the source 2D image includes: rendering the source mesh model based on the geometric information of the source target mesh model based on the camera viewpoint to obtain a source 2D image corresponding to the camera viewpoint; and the acquisition of the target 2D image includes: rendering the target mesh model based on the geometric information of the target mesh model based on the camera viewpoint to obtain a target 2D image corresponding to the camera viewpoint; wherein the geometric information of the source target mesh model is different from the geometric information of the target mesh model.

[0154] Aspect 4: Determining the semantic mapping feature based on the embedding features of the source two-dimensional image and the embedding features of the target two-dimensional image includes: performing a nearest neighbor search based on the embedding features of the source two-dimensional image and the embedding features of the target two-dimensional image to determine the semantic mapping feature, wherein the semantic mapping feature represents the two-dimensional displacement relationship between source image pixels and target image pixels in the form of a semantic flow mapping graph.

[0155] Aspect 5: The iterative update of the source mesh model based at least in part on the semantic mapping features includes: in each iteration, estimating the position of vertices in the source mesh model after deformation to the target mesh model using the deformation model; determining the displacement corresponding to the vertex based on the position of the vertex in the source mesh model after deformation to the target mesh model; determining the estimated semantic mapping features based on the displacement corresponding to the vertex; and updating the parameters in the deformation model based on the difference between the estimated semantic mapping features and the semantic mapping features.

[0156] Aspect 6: The iterative update of the source mesh model based at least in part on the semantic mapping features further includes: in each iteration, using the deformation model to estimate the position of the vertices in the source mesh model after deformation to the target mesh model; determining the displacement corresponding to the vertex based on the position of the vertex in the source mesh model after deformation to the target mesh model; determining the mesh model after deformation by the deformation model based on the displacement corresponding to the vertex; and updating the parameters in the deformation model based on the difference between the deformed mesh model and the target mesh model.

[0157] Aspect 7: The iterative update of the source mesh model based at least in part on the semantic mapping features further includes: in each iteration, using the deformation model to estimate the position of vertices in the source mesh model after deformation to the target mesh model; determining the displacement corresponding to the vertex based on the position of the vertex in the source mesh model after deformation to the target mesh model; determining the mesh model after deformation by the deformation model based on the displacement corresponding to the vertex; performing differentiable rendering on the deformed mesh model to determine the normal map corresponding to the deformed mesh model; determining the normal map corresponding to the target mesh model; and updating the parameters in the deformation model based on the difference between the normal map corresponding to the deformed mesh model and the normal map corresponding to the target mesh model.

[0158] Aspect 8: The iterative update of the source mesh model based at least in part on the semantic mapping features further includes: in each iteration, using the deformation model to estimate the Jacobian matrix of each face of the vertices in the source mesh model after deformation to the target mesh model; determining the displacement corresponding to the vertex based on the position of the vertex in the source mesh model after deformation to the target mesh model; determining the mesh model after deformation by the deformation model based on the displacement corresponding to the vertex; determining the Jacobian matrix of each face in the deformed mesh model; and updating the parameters in the deformation model based on the difference between the Jacobian matrix of each face in the deformed mesh model and the identity matrix.

[0159] Aspect 9: The iterative update of the source mesh model based at least in part on the semantic mapping features further includes: in each iteration, using the deformation model to estimate the Jacobian matrix of each face of the vertices in the source mesh model after deformation to the target mesh model; determining the displacement corresponding to the vertex based on the position of the vertex in the source mesh model after deformation to the target mesh model; determining the mesh model after deformation by the deformation model based on the displacement corresponding to the vertex; determining the Jacobian matrix of each face in the deformed mesh model and the rotation component of the Jacobian matrix after polar decomposition; and updating the parameters in the deformation model based on the difference between the Jacobian matrix of each face in the deformed mesh model and the corresponding rotation component.

[0160] Aspect 10: A method for training a neural network is provided, the neural network model including a mapping module and a deformation module, the mapping module carrying a first feature extraction network, a second feature extraction network, a first adaptation network, and a second adaptation network, wherein the method includes: determining preliminary features of a source two-dimensional image sample using the first feature extraction network based on the source two-dimensional image sample; determining embedding features of the source two-dimensional image sample using the first adaptation network based on the preliminary features of the source two-dimensional image sample; determining preliminary features of a target two-dimensional image sample using the second feature extraction network based on the target two-dimensional image sample; determining embedding features of the target two-dimensional image sample using the second adaptation network based on the preliminary features of the target two-dimensional image sample; and training the first adaptation network and the second adaptation network based on the embedding features of the source two-dimensional image and the embedding features of the target two-dimensional image.

[0161] Aspect 11: The training of the first adaptation network and the second adaptation network includes: sampling matching source 2D points and target 2D points from the embedding features of the source 2D image and the embedding features of the target 2D image; determining the distance between the source 2D points and the target 2D points based on the matching source 2D points and the target 2D points; and adjusting the parameters in the first adaptation network and the second adaptation network based on the distance between the source 2D points and the target 2D points.

[0162] Aspect 12: The training of the first adaptation network and the second adaptation network further includes: sampling a first vertex from the source mesh model corresponding to the source two-dimensional image sample; sampling a second vertex from the target mesh model corresponding to the target two-dimensional image sample, wherein the second vertex and the first vertex have a pseudo geodesic distance greater than a predetermined threshold; and using the vertex pair formed by the first vertex and the second vertex as negative samples to adjust the parameters in the first adaptation network and the second adaptation network.

[0163] Aspect 13: An apparatus for processing a mesh model, the apparatus comprising: a mapping module configured to: acquire a source two-dimensional image and determine, based on the source two-dimensional image, embedding features of the source two-dimensional image; acquire a target two-dimensional image and determine, based on the target two-dimensional image, embedding features of the target two-dimensional image; and determine, based on the embedding features of the source two-dimensional image and the embedding features of the target two-dimensional image, semantic mapping features indicating a correspondence from pixels in the source two-dimensional image to pixels in the target two-dimensional image; and a deformation module configured to: iteratively update a deformation model, at least in part, based on the semantic mapping features, and deform and render a source mesh model based on the updated deformation model to obtain a target mesh model incorporating geometric information of the source mesh model.

[0164] Aspect 14: An electronic device comprising: one or more processors; and one or more memories, wherein the memories store a computer-executable program that, when executed by the processor, performs the method described in any aspect.

[0165] Aspect 15: A computer-readable storage medium having stored thereon computer instructions which, when executed by a processor, implement the method described in any aspect.

Claims

1. A method for processing a mesh model, the method comprising: Acquire a source two-dimensional image, and determine the embedding features of the source two-dimensional image based on the source two-dimensional image; Acquire a target two-dimensional image, and determine the embedding features of the target two-dimensional image based on the target two-dimensional image; Based on the embedding features of the source 2D image and the embedding features of the target 2D image, semantic mapping features are determined, which indicate the correspondence between pixels in the source 2D image and pixels in the target 2D image. The deformable model is iteratively updated, at least in part, based on the semantic mapping features. Based on the updated deformation model, the source mesh model is deformed and rendered to obtain a target mesh model that incorporates the geometric information of the source mesh model.

2. The method as described in claim 1, wherein, Based on the source two-dimensional image, determining the embedding features of the source two-dimensional image includes: Based on the source two-dimensional image, preliminary features of the source two-dimensional image are determined using a first feature extraction network; and Based on the preliminary features of the source two-dimensional image, the embedding features of the source two-dimensional image are determined through a first adaptation network; Based on the target two-dimensional image, determining the embedding features of the target two-dimensional image includes: Based on the target two-dimensional image, a second feature extraction network is used to determine the preliminary features of the target two-dimensional image; and Based on the preliminary features of the target two-dimensional image, the embedding features of the target two-dimensional image are determined through a second adaptation network. The embedding features of the source 2D image and the embedding image of the target 2D image are located in the same embedding space.

3. The method as described in claim 1, wherein, The acquisition of the source 2D image includes: rendering the source mesh model based on the geometric information of the source target mesh model from the camera's perspective to obtain a source 2D image corresponding to the camera's perspective; and The acquisition of the target two-dimensional image includes: rendering the target mesh model based on the geometric information of the target mesh model based on the camera viewpoint, so as to obtain the target two-dimensional image corresponding to the camera viewpoint; The geometric information of the source and target mesh models differs from that of the target mesh model.

4. The method of claim 2, wherein, The determination of the semantic mapping features based on the embedding features of the source 2D image and the embedding features of the target 2D image includes: Based on the embedding features of the source 2D image and the target 2D image, a nearest neighbor search is performed to determine the semantic mapping features, which represent the 2D displacement relationship between source image pixels and target image pixels in the form of a semantic flow mapping graph.

5. The method of claim 1, wherein, The iterative update of the source mesh model, at least in part based on the semantic mapping features, includes: In each iteration, The deformation model is used to estimate the positions of vertices in the source mesh model after deformation to the target mesh model; Based on the position of the vertex in the source mesh model after deformation to the target mesh model, determine the displacement corresponding to the vertex; Based on the displacement corresponding to the vertex, the estimated semantic mapping features are determined; and The parameters in the deformable model are updated based on the difference between the estimated semantic mapping features and the semantic mapping features.

6. The method of claim 5, wherein, The iterative updating of the source mesh model, at least in part based on the semantic mapping features, further includes: In each iteration, The deformation model is used to estimate the positions of vertices in the source mesh model after deformation to the target mesh model; Based on the position of the vertex in the source mesh model after deformation to the target mesh model, determine the displacement corresponding to the vertex; Based on the displacement corresponding to the vertex, determine the mesh model after deformation by the deformation model; and Based on the differences between the deformed mesh model and the target mesh model, the parameters in the deformed model are updated.

7. The method of claim 5, wherein, The iterative updating of the source mesh model, at least in part based on the semantic mapping features, further includes: In each iteration, The deformation model is used to estimate the positions of vertices in the source mesh model after deformation to the target mesh model; Based on the position of the vertex in the source mesh model after deformation to the target mesh model, determine the displacement corresponding to the vertex; Based on the displacement corresponding to the vertex, determine the mesh model after deformation by the deformation model; The deformed mesh model is rendered differently to determine the normal map corresponding to the deformed mesh model. Determine the normal map corresponding to the target mesh model; and Based on the difference between the normal map corresponding to the deformed mesh model and the normal map corresponding to the target mesh model, the parameters in the deformed model are updated.

8. The method of claim 5, wherein, The iterative updating of the source mesh model, at least in part based on the semantic mapping features, further includes: In each iteration, The Jacobian matrix of each face after the vertices in the source mesh model are deformed into the target mesh model is estimated using the deformation model. Based on the position of the vertex in the source mesh model after deformation to the target mesh model, determine the displacement corresponding to the vertex; Based on the displacement corresponding to the vertex, determine the mesh model after deformation by the deformation model; Determine the Jacobian matrix of each face in the deformed mesh model; and The parameters in the deformed mesh model are updated based on the difference between the Jacobian matrix and the identity matrix of each face in the deformed mesh model.

9. The method of claim 6, wherein, The iterative updating of the source mesh model, at least in part based on the semantic mapping features, further includes: In each iteration, The Jacobian matrix of each face after the vertices in the source mesh model are deformed into the target mesh model is estimated using the deformation model. Based on the position of the vertex in the source mesh model after deformation to the target mesh model, determine the displacement corresponding to the vertex; Based on the displacement corresponding to the vertex, determine the mesh model after deformation by the deformation model; Determine the Jacobian matrix of each face in the deformed mesh model and the rotational components of the Jacobian matrix after polar decomposition; and Based on the difference between the Jacobian matrix and the corresponding rotation component of each face in the deformed mesh model, the parameters in the deformed model are updated.

10. A method for training a neural network, the neural network model comprising a mapping module and a deformation module, the mapping module carrying a first feature extraction network, a second feature extraction network, a first adaptation network, and a second adaptation network, wherein, The method includes: Based on the source two-dimensional image samples, the preliminary features of the source two-dimensional image samples are determined using a first feature extraction network; Based on the preliminary features of the source two-dimensional image samples, the embedding features of the source two-dimensional image samples are determined through a first adaptation network; Based on the target two-dimensional image sample, the preliminary features of the target two-dimensional image sample are determined using a second feature extraction network; Based on the preliminary features of the target two-dimensional image sample, the embedding features of the target two-dimensional image sample are determined through a second adaptation network, and The first adaptation network and the second adaptation network are trained based on the embedding features of the source 2D image and the target 2D image.

11. The method of claim 10, wherein, The training of the first adaptation network and the second adaptation network includes: Sample matching source 2D points and target 2D points from the embedding features of the source 2D image and the embedding features of the target 2D image; Based on the matched source 2D point and target 2D point, determine the distance between the source 2D point and the target 2D point; and Based on the distance between the source 2D point and the target 2D point, the parameters in the first adaptation network and the second adaptation network are adjusted.

12. The method of claim 10, wherein, The training of the first adaptation network and the second adaptation network further includes: The first vertex is sampled from the source mesh model corresponding to the source two-dimensional image sample; A second vertex is sampled from the target mesh model corresponding to the target two-dimensional image sample, wherein the second vertex has a pseudo-geodetic distance greater than a predetermined threshold from the first vertex; and Using the vertex pair formed by the first vertex and the second vertex as negative samples, the parameters in the first adaptation network and the second adaptation network are adjusted.

13. An apparatus for processing a mesh model, the apparatus comprising: The mapping module is configured as follows: Acquire a source two-dimensional image, and determine the embedding features of the source two-dimensional image based on the source two-dimensional image; Acquire a target two-dimensional image, and determine the embedding features of the target two-dimensional image based on the target two-dimensional image; as well as Based on the embedding features of the source 2D image and the embedding features of the target 2D image, semantic mapping features are determined, which indicate the correspondence between pixels in the source 2D image and pixels in the target 2D image. The transformation module is configured as follows: The deformable model is iteratively updated, at least in part, based on the semantic mapping features. Based on the updated deformation model, the source mesh model is deformed and rendered to obtain a target mesh model that incorporates the geometric information of the source mesh model.

14. An electronic device comprising: One or more processors; and One or more memories, wherein the memories store a computer-executable program that, when executed by the processor, performs the method of any one of claims 1-12.

15. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the method as claimed in any one of claims 1-12.

Images