Automatic retopologization of textured 3D meshes

Abstract

Methods and apparatus are described for automating re-topologies of a 3D mesh, including automatic selection and adjustment of corresponding points. Automatic selection of the corresponding points may be performed to refine the positions of the corresponding points using a matching score that is calculated based on a surface normal similarity between a surface corresponding to a candidate corresponding point on the input scan grid and a point on the deformable model of the 3D surface. The match score may also take into account a distance between a candidate corresponding point on the input scan grid and a corresponding point on the deformable model of the 3D surface, and a similarity of surface features, such as a similarity of a surface curvature at the candidate corresponding point on the input scan grid and a corresponding point on the deformable model of the 3D surface.

Description

Background Technology

[0001] Landmark detection is a computer vision task in which salient landmarks are detected and located within an image or video. Facial landmark detection is a computer vision task in which a model predicts keypoints representing landmarks on a face, such as parts of the eyes, nose, and lips. Keypoints can be used, for example, to detect a person's head position and rotation. Facial landmark detection can be used to facilitate head pose estimation, identify gaze direction, and detect facial pose. Facial landmark detection can be challenging due to the variability in image quality and other factors such as pose variability and the presence of occlusion. Summary of the Invention

[0002] Systems and methods are provided for improving the performance and robustness of marker detection systems that incorporate automatic retopology of 3D meshes. Automatic retopology of 3D meshes can include automatic selection and adjustment of corresponding points. Automatic selection of corresponding points can be performed to adjust or refine the positions of corresponding points using a matching score calculated based on the similarity of surface normals between candidate corresponding points on the input scanned mesh and points on a deformable model of the 3D surface. The matching score can also be calculated based on the distance between candidate corresponding points on the input scanned mesh and corresponding points on the deformable model of the 3D surface and / or the similarity of surface features (e.g., the similarity between the surface curvature at the candidate corresponding point on the input scanned mesh and the corresponding point on the deformable model of the 3D surface).

[0003] According to some embodiments, the technical benefits of the systems and methods disclosed herein include improved system robustness and accuracy, reduced costs of computing and storage resources, and improved system performance. Other technical benefits can also be achieved through implementations of the disclosed techniques.

[0004] This summary is provided to present a simplified description of some aspects of the disclosed technology, which are further described in the detailed embodiments below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to the implementation of solutions to any or all the shortcomings mentioned in any part of this disclosure. Attached Figure Description

[0005] Components with similar numbers can refer to common components in different diagrams.

[0006] Figure 1A An embodiment of an input scan mesh and an intermediate mesh is depicted, the intermediate mesh including a deformable model of a 3D surface having a geometry approximating that of the input scan mesh.

[0007] Figure 1BAn example is depicted showing candidate corresponding points on the input scan grid and corresponding key points on the intermediate grid.

[0008] Figure 1C An embodiment is depicted, including the input scan grid, the intermediate grid, and the final refined grid after photometric refinement has been applied to the intermediate grid.

[0009] Figure 1D Various embodiments of images, facial landmarks, and facial reconstruction are depicted.

[0010] Figure 1E Examples of sparse and dense sets of facial landmarks associated with human facial features are depicted.

[0011] Figure 2A An example of a networked computing environment in which the disclosed technologies can be practiced is described.

[0012] Figure 2B An embodiment of various components of a retopology system is described.

[0013] Figure 3A A flowchart is depicted illustrating one embodiment of a process for automating the retopology of 3D meshes.

[0014] Figure 3B A flowchart illustrating another embodiment of a process for automating the retopology of 3D meshes is shown. Detailed Implementation

[0015] The techniques described in this paper provide a mechanism for automating the retopology of 3D meshes, including the automatic selection of correspondence points. Retopology refers to the process of acquiring a source 3D mesh and generating a target 3D mesh that includes a new version of the source 3D mesh with a structured topology. The resulting target 3D mesh can be used to improve rendering, editing, and animation. Correspondence points (or correspondences) can be used to map points within the source 3D mesh to equivalent points in the target 3D mesh. In one example, correspondence points can be used to map the top of an ear in the source 3D mesh to the top of an ear in the target 3D mesh. Retopology is typically a manual process that requires artists to reconstruct the new topology point-to-point. While some tools can be used to help fit a source 3D mesh (or template mesh) to the desired topology in the target 3D mesh, they usually require manual selection of correspondence points for each scan and often also manual cleanup. These manual processes requiring user-annotated correspondence points are time-consuming and prone to human error, inconsistencies, and biases. The technical benefits of automating the selection of corresponding points include faster retopology of 3D meshes, fewer errors and inconsistencies, and more efficient and higher-fidelity reconstruction using a single pass.

[0016] In some cases, automatic retopology of 3D meshes involves three stages. During the first stage, a deformable model of the 3D surface is fitted to the source 3D mesh via landmark detection from a rendered view of the object (e.g., a human head or torso). In one approach, the source 3D mesh (or textured 3D scans) is rendered from multiple viewpoints under uniform lighting conditions to create 2D images, and an intermediate mesh is generated using the source 3D mesh and the 2D images. In another approach, a multi-camera setup can be used to collect 2D images capturing different viewpoints of the object, and the source 3D mesh can be constructed using photogrammetry from these 2D images under uniform lighting conditions. Deep neural networks (DNNs) can be used, and these images can be used to regress keypoints, which are then fitted to the deformable model of the 3D surface. The deformable model of the 3D surface can include a parametric 3D model (e.g., a parametric head model). The source 3D mesh can include an input scan mesh S, and the deformable model of the 3D surface can include an intermediate mesh M having a geometry approximating the input scan mesh S.

[0017] During the second phase, automatic selection and adjustment of corresponding points are performed to refine their positions. Corresponding points can be automatically calculated by sampling a subset of vertices (or points) on the intermediate mesh M. For each sampled point, the position on the input scan mesh S can be identified as a candidate corresponding point. The closer the sampled point on the intermediate mesh M is to the candidate corresponding point on the input scan mesh S, the better the correspondence. In some cases, candidate corresponding points on the input scan mesh S can include points with the best-matching surface normal or surface orientation within a threshold distance of the sampled point on the intermediate mesh M. In one example, the input scan mesh S might correspond to a face, and 20 corresponding points could be determined to supervise surface alignment with the intermediate mesh M.

[0018] For each candidate corresponding point, a matching score can be calculated based on the similarity of the surface normals between the candidate corresponding point on the input scan grid S and the sampled keypoint on the intermediate grid M. For each sampled keypoint on the intermediate grid M... p M It can be determined that it has the characteristics of sampling key points. p M Candidate correspondences of the highest matching score p S Sampling key points p M Corresponding points of candidates p S The matching score between them can be expressed by the following equation (1): Matching score In equation (1), p S It is the point on the input scan grid S. p M These are the key sampling points on the middle grid M. n S It is S in p S The surface normal at that location, and n M It is M in p M The surface normal at the location. The parameter α controls how close the two normals should match. In one example, for a face scan, α is set to ten. The matching score can be calculated with S at... p S The first vector associated with the surface normal at the location and M is... p M The inner product of the second vector associated with the surface normal at a given location. In some cases, the matching score pair S is... p S The surface normal at point M is perpendicular to M. p M The first difference between the surface normals at the point and the point p M and p S The second difference between the positions is used for weighting. Furthermore, the matching score can take into account surface features, such as surface curvature.

[0019] In some embodiments, when M and S are not fully aligned, by... p M exist n M The correspondence is refined by shifting the keypoints directionally by a distance *d* and then finding the point on *S* with the highest matching score. In one example, for a face scan, this refinement could be applied to keypoints behind or on top of the ear, where *d* = 2 cm. For scans containing more than a threshold number of keypoints (e.g., more than 70 keypoints), for each keypoint, the search for the best matching point on *S* that is already close to that keypoint can be limited to a specific number of points (e.g., a maximum of five candidate points).

[0020] In some cases, the distance d is set based on the degree of certainty of each keypoint. For example, if the degree of certainty of a keypoint is greater than a threshold (e.g., 0.9 or 90%), the distance d can be set to 2 cm; however, if the degree of certainty is not greater than the threshold, the distance d can be set to 1 cm or a distance less than the distance d used when the degree of certainty is equal to or greater than the threshold.

[0021] In some cases, the distance d is set based on the degree of uncertainty at each keypoint location. For example, if the degree of uncertainty at a given keypoint location is less than a threshold (e.g., 0.1 or 10%), the distance d can be set to 2 cm; however, if the degree of uncertainty is not less than the threshold, the distance d can be set to 1 cm or a distance less than the distance d used when the degree of uncertainty is less than the threshold.

[0022] Photometric refinement can be performed during the third stage, which automates the retopology of the 3D mesh. After retopology, the intermediate mesh M can be photometrically refined to improve the representation of visual details. This photometric refinement can include optimizations that match the appearance of both S and M under the same lighting and shading model. The rendered appearance of the 3D mesh (which is used to calculate the image spatial loss) can depend on the shape of the surface around the shadow points and therefore can carry information about the surface curvature. The rendering function can utilize differentiable path tracing with a path-tracing diffuse lighting model. For face scanning, photometric refinement after retopology can improve fine-scale surface details, such as wrinkles and eyes, which greatly affects visual similarity and resemblance. In some cases, photometric refinement requires that the fitting and correspondence lookup in the second stage be effective; otherwise, photometric refinement may be infeasible or generate a model with the desired appearance.

[0023] One technical problem with relying on manual intervention to fit or adapt a source 3D mesh with a desired topology to a target 3D mesh (e.g., an intermediate mesh generated before photometric refinement) is that user-annotated correspondences are time-consuming and prone to introducing human error and inconsistencies. The technical benefits of automating correspondence selection using a matching score include reduced time for performing 3D mesh retopology, fewer errors and inconsistencies, and more efficient and higher-fidelity reconstruction. This matching score is calculated based on the similarity of surface normals between candidate correspondences on the source 3D mesh and points on the target 3D mesh, the distance between candidate correspondences on the source 3D mesh and their corresponding points on the target 3D mesh, and / or the similarity of surface curvature at candidate correspondences on the source 3D mesh and their corresponding points on the target 3D mesh. Furthermore, the technical benefits of using automated correspondence selection and adjustment to refine 3D landmark points (e.g., refining correspondences for ears, fingers, and feet) include reduced computational and storage costs and reduced time for generating reconstructions.

[0024] In some implementations, automatic retopology of the 3D mesh includes generating an initial estimate of corresponding points (e.g., 700 corresponding points) using keypoint detection, and then refining the initial corresponding points by moving them to points on the 3D mesh with the most similar normals and / or the most similar surface curvatures based on matching scores. To further improve the accuracy of correspondences associated with complex or high-frequency facial features (e.g., ears and eyes), these points can be shifted along surface normals at their locations (e.g., 2 cm for ears and 1 cm for eyes) before correcting them based on matching scores.

[0025] Figure 1A An embodiment of an input scan mesh 110 and an intermediate mesh 120 is depicted. The input scan mesh 110 may include a source 3D mesh, and the intermediate mesh 120 may include a target 3D mesh, which includes an approximation of the source 3D mesh with a structured topology. Keypoints 122 include points on the base of the ear on the intermediate mesh 120, and candidate corresponding points 112 include corresponding points on the base of the ear on the input scan mesh 110. The input scan mesh 110 can be rendered to create multiple images of a human head from different viewpoints under controlled or uniform lighting conditions. The intermediate mesh 120 may include a deformable model fitted to the 3D surface of the input scan mesh 100 via marker regression. The intermediate mesh 120 may include a parameterized head model, and the keypoints 122 may include one of multiple keypoints as part of a keypoint regression network. These multiple keypoints can be regressed using a DNN and used to generate the intermediate mesh 120.

[0026] Figure 1B An embodiment depicting candidate corresponding points on the input scan mesh 110 and corresponding key points on the intermediate mesh 120 is described. A key point 124 located at the outer center of the ear on the intermediate mesh 120 has a surface normal 128, and a candidate corresponding point 114 located at the outer center of the ear on the input scan mesh has a surface normal 129. The surface normal, or normal vector, can be orthogonal to the surface of the 3D mesh at a specific point. Position adjustments can be made, wherein the key point 124 is deformed or repositioned from its position to the position of the candidate corresponding point 114.

[0027] Figure 1CAn embodiment regarding an input scan mesh 110, an intermediate mesh 120, and a final refined mesh 136 is depicted. The final refined mesh 136 can be automatically generated from images capturing different views of a human head using a three-stage process. During the first stage, the intermediate mesh 120 is fitted to the input scan mesh 100 via landmark regression. The intermediate mesh 120 may include a parametric 3D head model fitted to match the topology of the input scan mesh 110. During the second stage, automatic selection of corresponding points is performed to refine the positions of keypoints on the intermediate mesh 120. Updates to the positions of keypoints can be automatically computed by sampling a subset of vertices (or points) on the intermediate mesh 120. For each sampled point, a position on the input scan mesh 110 can be identified as a candidate corresponding point.

[0028] In some implementations, candidate corresponding points on the input scan grid 110 may include points within a threshold distance of sampling keypoints on the intermediate grid 120, wherein the point has the closest or best-matching surface normal within the threshold distance of the sampling keypoint (e.g., within 2 cm of the sampling keypoint). In some embodiments, candidate corresponding points on the input scan grid 110 may be determined based on a best-match score that considers the similarity of surface normals between candidate corresponding points on the input scan grid 110 and sampling keypoints on the intermediate grid 120. In one instance, the match score between the sampling keypoint and the candidate corresponding point may be calculated using equation (1), where the α value is set to ten. In some cases, a match score may be calculated that takes into account differences in surface orientation, position, and / or surface curvature. During the third phase, the intermediate grid 120 is photometrically refined to better match fine details found in the input scan grid 110 (e.g., to better match the appearance of wrinkles on a face).

[0029] In some cases, landmarks are points in the correspondence between two meshes of a face. For a face, a landmark might correspond to the tip of the nose or the corner of the eye. Landmarks play a role in face-related computer vision, for example, to extract facial regions of interest or to help constrain 3D model fitting. However, many aspects of facial identity or expression cannot be encoded solely by a sparse set of landmarks. For example, without landmarks on the cheeks, we might not be able to tell if someone has high cheeks. Or, without landmarks around the extraocular regions, we might not be able to tell if someone is gently closing their eyes or wrinkling their face. To reconstruct, for example, a face more accurately, landmarks can be combined with additional signals such as depth images or techniques such as differentiable rendering.

[0030] Keypoints in an image are typically located by having a neural network generate a heatmap image, where the heatmap has high pixel values ​​in the portion of the image closer to the keypoint and low pixel values ​​in the portion farther away from the keypoint. Another algorithm (argmax) can be run on the heatmap image to find the maximum value that serves as a peak. The location of this peak is the 2D location of the keypoint. The value of the peak in the heatmap can sometimes be used as a measure of uncertainty.

[0031] Alternatively, each keypoint can be predicted as a 2D random variable, typically following a distribution with location (x, y) and standard deviation sigma. A neural network can be trained to maximize the log-likelihood of the sample from each predicted keypoint as equal to the ground truth. Keypoint uncertainty arises during training because the neural network is penalized for errors and uncertainties regarding keypoint locations. The derivation is provided below (assuming a uniform prior on sigma): The true value consists of a set of key point coordinates. .

[0032] Each keypoint is predicted as a 2D random variable, typically following a standard deviation with location (x', y') and (circular) standard deviation. σ The distribution of the keypoints. For the predicted keypoint random variable, the relative likelihood of the sample from this random variable being equal to the true keypoint location is: Given a complete set of N key points, predict a set of coordinates. We use log-likelihoods because they are more convenient. Since the expectation is to maximize the log-likelihood of samples from each predicted keypoint being equal to the true value, we minimize the loss as the sum of negative log-likelihoods.

[0033] For clarity, this is divided into two parts: Loss σ The network is penalized for excessive uncertainty in key point predictions, and the loss... µ The network is penalized for poor keypoint prediction. Additionally, in some cases, the symmetric Gaussian distribution in the above example can be extended to an asymmetric Gaussian distribution.

[0034] The distribution of uncertainty values ​​can also be influenced during training by introducing appropriate priors. The above formula assumes a uniform prior on the predicted sigma. Regarding the (2D) Gaussian distribution... σThe natural choice for the prior (or accuracy) is the Wishart distribution (a conjugate prior to the Gaussian distribution), although other priors can be used. In the univariate case, this prior is the Gamma distribution. This has the effect of encouraging the network during training to allocate more neural resources to cases where it is currently performing poorly (where sigma is large) and less neural resources to cases where it is already relatively certain (where sigma is small), in order to balance the usefulness of its keypoint predictions for fitting downstream models. See below for the derivation of the assumed Gamma prior.

[0035] Definitions (precision, Gaussian distribution, Gamma distribution, or Wishart distribution) a and b It is the (manually) tuned constant "shape and inverse scale". Log-likelihood derivation for each key point By minimizing -log( p To maximize p Each keypoint loss term of the Gamma prior First item It's a constant, so it has no effect on training (ignore it). Add the following to the loss instead of - log ( Gam ): Same in terms of σ: Additionally, object detection can be achieved through keypoint uncertainty. A sliding window can be applied to the image, and the average keypoint confidence of each window can be measured. If no window with high average keypoint confidence is found, it can be determined that the object is not in the image. Otherwise, the window reporting the highest average keypoint confidence can be considered to contain the object.

[0036] Some use cases implemented using the described techniques include: receiving image input from a conventional color (RGB) camera instead of a depth camera, predicting more landmarks, and combining it with a model fitter that predicts inherent camera parameters (e.g., focal length). This is important for recovering 3D structures from RGB images taken by various cameras to achieve good results. Other use cases include performing 3D reconstructions from multiple views, where uncertainties in each view are considered, and where the extrinsic parameters of each camera are optimized simultaneously. More generally, image input can be received from various types of cameras (e.g., web cameras, depth cameras, cameras on head-mounted displays (HMDs), IR cameras, event cameras, etc.), and the images can be RGB, depth maps, IR, etc., and the camera placement can be outside-in (e.g., the sensor is stationary as in a web camera), or introspective position tracking, where the camera or sensor is located on the device being tracked (e.g., an HMD). In the case of an HMD, dense landmarks observed on facial portions can also be used to locate the HMD itself relative to the face.

[0037] When landmark detection algorithms use keypoints, keypoint confidence, or certainty, can be useful. For example, when fitting a 3D model to 2D keypoints, a keypoint with low confidence can be considered unreliable and discounted during model fitting. This can happen, for example, when a keypoint is occluded. Estimating uncertainty can also be used to train a better landmark estimator. In some cases, a neural network can determine the confidence value for each keypoint.

[0038] refer to Figure 1D Given a single image 110, multiple landmarks (e.g., 70 landmarks) including landmark 172 can be robustly and accurately predicted. Lines are drawn between the landmarks to aid visualization. A 3D deformable face model can be fitted to the predicted landmarks to reconstruct the face in 3D, e.g., face reconstruction 174.

[0039] refer to Figure 1E Compared to sparse facial landmark sets, dense landmark sets can be used to cover the entire head in greater detail, including the ears, eyes, and teeth. These dense landmarks can provide improved detection of facial identity and subtle expressions. Figure 1E As shown, the sparse set of facial landmarks includes landmark 182, and the dense set of facial landmarks includes landmark 184.

[0040] Figure 2AAn embodiment of a networked computing environment 200 in which the disclosed technologies can be implemented is depicted. The networked computing environment 200 includes a retopologically configured system 220, a storage device 259, a server 260, and a computing device 254 that communicate with each other via one or more networks 280. The networked computing environment 200 may include various computing and storage devices interconnected via one or more networks 280. The networked computing environment 200 may correspond to or provide access to cloud computing environments that provide Software as a Service (SaaS) or Infrastructure as a Service (IaaS) services. One or more networks 280 may allow computing devices and / or storage devices to connect to and communicate with other computing devices and / or other storage devices. In some cases, the networked computing environment 200 may include other computing devices and / or other storage devices, not shown. Other computing devices may include, for example, mobile computing devices, non-mobile computing devices, servers, workstations, laptop computers, tablet computers, desktop computers, or information processing systems. Other storage devices may include, for example, storage area network storage devices, networked storage devices, hard disk drives, solid-state drives, data storage systems, or cloud-based data storage systems. One or more networks 280 may include cellular networks, mobile networks, wireless networks, wired networks, secure networks (such as corporate private networks), insecure networks (such as open wireless networks), local area networks (LANs), wide area networks (WANs), the Internet, or combinations of networks.

[0041] In some embodiments, the computing devices within the networked computing environment 200 include physical hardware computing devices or virtual computing devices, such as one or more virtual machines. The storage devices within the networked computing environment 200 may include physical hardware storage devices or virtual storage devices, such as one or more virtual disks. Physical hardware storage devices may include non-volatile and volatile storage devices.

[0042] The retopology system 220 may include a distributed computing system or a system for performing automatic retopology of 3D meshes. For example... Figure 2AAs shown, the retopology system 220 includes a network interface 225, a processor 226, a memory 227, and a disk 228, all of which communicate with each other. The network interface 225, processor 226, memory 227, and disk 228 may include physical or virtualized components. In one example, the network interface 225, processor 226, memory 227, and disk 228 may be provided by virtualization infrastructure or cloud-based infrastructure. The network interface 225 allows the retopology system 220 to connect to one or more networks 280. The network interface 225 may include a wireless network interface and / or a wired network interface. The processor 226 allows the retopology system 220 to execute computer-readable instructions stored in the memory 227 to perform the processes described herein. The processor 226 may include one or more processing units, such as one or more CPUs, one or more GPUs, and / or one or more NPUs. The memory 227 may include one or more types of memory (e.g., RAM, SRAM, DRAM, EEPROM, flash memory). The disk 228 may include a hard disk drive and / or a solid-state drive. The memory 227 and disk 228 may include hardware storage devices.

[0043] The computing device 254 may include a mobile computing device, such as a tablet computer, that allows a user to access a graphical user interface for retopologizing the system 220. The user interface may be provided by the retopologizing system 220 and displayed using the display screen of the computing device 254.

[0044] A server, such as server 260, can allow client devices, such as retopology system 220 or computing device 254, to download information or files (e.g., executable files, text, applications, audio, image, or video files) from the server. Server 260 may include a hardware server. In some cases, a server may act as an application server or a file server. Generally, a server can refer to a hardware device that acts as a host in a client-server relationship, or to a software process that shares resources with one or more clients or performs work for one or more clients.

[0045] Server 260 includes a network interface 265, a processor 266, memory 267, and a disk 268, all of which communicate with each other. Network interface 265 allows server 260 to connect to one or more networks 280. Network interface 265 may include a wireless network interface and / or a wired network interface. Processor 266 allows server 260 to execute computer-readable instructions stored in memory 267 to perform the processes described herein. Processor 266 may include one or more processing units, such as one or more CPUs, one or more GPUs, and / or one or more NPUs. Memory 267 may include one or more types of memory (e.g., RAM, SRAM, DRAM, EEPROM, flash memory). Disk 268 may include a hard disk drive and / or a solid-state drive. In some cases, disk 268 includes a flash-based SSD or a hybrid HDD / SSD drive. Memory 267 and disk 268 may include hardware storage devices.

[0046] Networked computing environment 200 can provide a cloud computing environment for one or more computing devices. In one embodiment, networked computing environment 200 may include virtualization infrastructure that provides software, data processing, and / or data storage services to end users accessing services via the networked computing environment. In one example, networked computing environment 200 may use retopology system 220 to provide cloud-based applications to computing devices such as computing device 254.

[0047] Figure 2B An embodiment of various components of a retopology system 220 is depicted. As depicted, the retopology system 220 includes hardware-level components and software-level components. Hardware-level components may include one or more processors 270, one or more memories 271, and one or more disks 272. Both the one or more memories 271 and the one or more disks 272 may include storage devices. Software-level components may include software applications and computer programs. In some embodiments, the source 3D mesh generator 281, the intermediate mesh generator 282, and the photometric refinement controller 284 are implemented using software or a combination of hardware and software. The one or more processors 270 may include a processing system.

[0048] In some cases, software-level components are run using dedicated hardware servers. In other cases, software-level components can be run using virtual machines or containerized environments running on multiple machines. In various embodiments, software-level components can be run from the cloud (e.g., software-level components can be deployed using cloud-based computing and storage infrastructure).

[0049] like Figure 2BThe software-level components described may also include virtualization layer processes, such as virtual machine 273, hypervisor 274, container engine 275, and host operating system 276. Hypervisor 274 may include a native hypervisor (or a bare-metal hypervisor) or a managed hypervisor (or a type 2 hypervisor). Hypervisor 274 can provide a virtual operating platform for running one or more virtual machines (e.g., virtual machine 273). The hypervisor may include software for creating and running virtual machine instances. Virtual machine 273 may include multiple virtual hardware devices, such as virtual processors, virtual memory, and virtual disks. Virtual machine 273 may include a guest operating system capable of running one or more software applications. Virtual machine 273 may run the host operating system 276 on which container engine 275 can run.

[0050] Container Engine 275 can run on top of Host Operating System 276 to run multiple isolated instances (or containers) on the same operating system kernel of Host Operating System 276. Containers can facilitate operating system-level virtualization and provide a virtualized environment for running applications and their dependencies. Containerized applications can include applications running within isolated runtime environments (or containers). Container Engine 275 can acquire container images and transform them into runtime processes. In some cases, Container Engine 275 can group the containers that make up an application into logical units (or pods). A pod can contain one or more containers, and all containers in a pod can run on the same node in the cluster. Each pod can be used as a deployment unit for the cluster. Each pod can run a single instance of an application.

[0051] In some embodiments, the depicted components of the retopology system 220, including the intermediate mesh generator 282, are implemented in the cloud or a virtualized environment that allows virtual hardware to be created and decoupled from the underlying physical hardware.

[0052] In some embodiments, the intermediate mesh generator 282 can perform automatic selection and adjustment of corresponding points to refine the intermediate 3D mesh (e.g., Figure 1A The position of the corresponding point on the middle grid (120) in the grid.

[0053] The retopology system 220 can store and utilize one or more machine learning models stored in memory such as memory 271. One or more processors, such as processor 270, can be used to train, execute, and / or deploy one or more machine learning models. The one or more machine learning models can include neural networks (e.g., deep neural networks), support vector machine models, decision tree-based models, k-nearest neighbor models, Bayesian networks, or other types of models, such as linear and / or nonlinear models. Linear models can be specified as linear combinations of input features. Neural networks can include feedforward neural networks, recurrent neural networks, or convolutional neural networks. The one or more machine learning models can include one or more generative AI models. The one or more machine learning models can include one or more multimodal models. The one or more machine learning models can include one or more language models, such as security-specific LLMs.

[0054] Large Language Models (LLMs) can refer to language models that include neural networks with a large number of parameters (e.g., millions or billions of parameters or weights). To reduce training time and cost, transfer learning can be utilized, where a pre-trained model is used as a starting point for a specific task and then trained or fine-tuned using a supervised dataset for that specific task. In one example, a large dataset can be used to pre-train an LLM, which is then fine-tuned using a much smaller dataset to customize the LLM for a specific task. Pre-training can refer to the act of training a machine learning model from scratch using a large corpus of data without any prior knowledge. Fine-tuning can refer to the transfer learning process of modifying a pre-trained LLM by training it in a supervised or semi-supervised manner. In some cases, fine-tuning involves adapting a pre-trained LLM to a specific task by fine-tuning it using a task-specific dataset.

[0055] LLMs can include transformer models implemented using transformer-based neural network architectures. A transformer model can include an encoder and / or a decoder. The encoder can extract features from an input sequence, and the decoder can use the features extracted from the encoder to produce an output sequence. In some cases, the encoder includes one or more encoding layers, and the decoder can include one or more decoding layers. Each encoding and decoding layer can include a self-attention mechanism that associates tokens within a token sequence with other tokens within the sequence. In one example, the self-attention mechanism can allow the transformer model to examine words within a sentence and determine the relative importance of other words within the same sentence to the examined word. In some cases, the encoder includes self-attention layers and feedforward neural network layers, and the decoder can include two self-attention layers and feedforward neural network layers. Transformer models (or transformers) can utilize encoder-decoder architectures, encoder-only architectures, or decoder-only architectures.

[0056] An example of a transformer model is the Generative Pre-trained Transformer (GPT) model. GPT models can include a type of LLM that uses deep learning to generate human-like text. A GPT model can be called "generative" because it can generate new content based on given input cues (e.g., text cues); "pre-trained" because it is trained on a large corpus of data before being fine-tuned for a specific task; and "transformer" because it utilizes a transformer-based neural network architecture to process input cues to generate output content (or responses). Generative AI can be used to generate new content such as text, images, audio, and video content.

[0057] In some embodiments, a machine learning model is trained to generate (or complete) a language text response given input text prompts. The input text prompts can provide information to help guide the machine learning model in generating an appropriate text response. Prompt engineering can be used to modify or update the input text prompts so that the machine learning model generates a more relevant text response. In some cases, a text response is generated by using a transformer model (e.g., a GPT language model) to predict the next set of words in the word sequence provided by the input text prompts. The transformer model can be trained using a set of input prompt-response pairs.

[0058] Multimodal learning can refer to a type of machine learning where machine learning models are trained to understand multiple forms of input data (e.g., text, images, video, and audio data) originating from different modalities. Image data can include different types of images, such as color images, depth images, and thermal images. In some cases, machine learning models include multimodal models, language models, or visual models.

[0059] Figure 3A A flowchart illustrating one embodiment of a process for automating the retopology of 3D meshes is provided. In one embodiment, Figure 3A The process can be made by, for example Figure 2B The retopology system 220 in the system performs retopology system operations. In another embodiment, Figure 3A The process can be implemented using cloud-based computing platforms or cloud-based computing services.

[0060] In step 302, a set of images is acquired. This set of images may include images capturing the object from different viewpoints. The images may include RGB images and / or depth images. The object may include a head or face. In step 304, an input scan grid for the object is generated using this set of images. In some cases, marker detection or marker regression can be used to generate the input scan grid. In other cases, the input scan grid can be generated via photogrammetry. The input scan grid can be... Figure 2B It is generated using the source 3D mesh generator 281.

[0061] In step 306, an intermediate mesh with a geometry approximating the input scan mesh is generated for the object. Keypoint regression (or marker regression) can be used to generate the intermediate mesh. Figure 2B An intermediate mesh generator 282 is used to generate an intermediate mesh. In step 308, key points on the intermediate mesh are identified. In some cases, the intermediate mesh may have multiple key points associated with features of an object. In one example, the object may include a face, and the number of key points may include 70 key points associated with different facial features. For each key point, up to a maximum number of candidate corresponding points may be considered (e.g., up to ten candidate corresponding points).

[0062] In step 310, the distance between a keypoint on the intermediate grid and a candidate corresponding point on the input scan grid is determined. This distance may correspond to the difference between a first position associated with the keypoint and a second position associated with the candidate corresponding point on the input scan grid. In step 312, a first surface normal associated with the keypoint and a second surface normal associated with the candidate corresponding point are determined. In step 314, a matching score is calculated. The matching score may be calculated based on the distance between the keypoint and the candidate corresponding point, the first surface normal, and the second surface normal. In one embodiment, equation (1) may be used to calculate the matching score. In another embodiment, the matching score may be calculated by determining the difference between the first and second surface normals and summing a first weighting coefficient multiplied by the difference between the first and second surface normals and a second weighting coefficient multiplied by the distance between the keypoint and the candidate corresponding point. In some cases, the matching score may be calculated based on the first position of the keypoint, the second position of the candidate corresponding point, the first surface normal, the second surface normal, and the similarity between the first surface associated with the keypoint and the second surface associated with the candidate corresponding point.

[0063] In step 316, candidate corresponding points are selected based on matching scores. In one embodiment, a candidate corresponding point may have the highest matching score among a set of possible corresponding points for a keypoint. In step 318, the candidate corresponding points are used to adjust or deform the intermediate mesh. In one embodiment, the intermediate mesh may be deformed based on the candidate corresponding points. In step 320, a 3D mesh for the object is generated by applying photometric refinement to the intermediate mesh. This can be used... Figure 2B The photometric refinement controller 284 in the middle is used to perform photometric refinement.

[0064] Figure 3B A flowchart illustrating another embodiment of a process for automating the retopology of 3D meshes is shown. In one embodiment, Figure 3B The process can be made by, for example Figure 2B The retopology system 220 in the system performs retopology system operations. In another embodiment, Figure 3B The process can be implemented using cloud-based computing platforms or cloud-based computing services.

[0065] In step 342, a source 3D mesh for the object is obtained. In step 344, a target 3D mesh for the object is generated. The target 3D mesh can be generated such that it has a geometry approximating the source 3D mesh or attempting to closely match it. This can be achieved using... Figure 2B An intermediate mesh generator 282 is used to generate the target 3D mesh. In some cases, the target 3D mesh may have significantly fewer surfaces than the source 3D mesh. In step 346, key points on the target 3D mesh are sampled. In step 348, the distance between the key points and candidate corresponding points on the source 3D mesh is determined. In step 350, a first surface normal associated with the key points and a second surface normal associated with the candidate corresponding points are determined.

[0066] In step 352, the surface curvature difference between the first surface associated with the keypoint and the second surface associated with the candidate corresponding point is determined. In step 354, a matching score is calculated or determined based on the first position of the keypoint, the second position of the candidate corresponding point, the normal of the first surface, the normal of the second surface, and / or the surface curvature difference. In one embodiment, equation (1) can be used to calculate the matching score. Figure 2B The circuitry within the intermediate mesh generator 282 or related to Figure 2B The executable code associated with the intermediate grid generator 282 in the middle is used to calculate the matching score.

[0067] In step 356, candidate corresponding points are selected based on the matching score. In one example, a candidate corresponding point can be selected as the best corresponding point for a keypoint on the target 3D mesh because it has the highest matching score among a set of possible corresponding points for that keypoint. In step 358, the target 3D mesh is adjusted or deformed using the candidate corresponding point. In step 360, photometric refinement is applied to the target 3D mesh. [The last sentence appears to be incomplete and possibly refers to a different process.] Figure 2B The photometric refinement controller 284 in the middle is used to perform photometric refinement.

[0068] At least one embodiment of the disclosed technology includes a storage device for storing instructions that, when executed, cause the system to perform operations including: generating an input scan grid associated with an object; generating an intermediate grid associated with the object, the intermediate grid having a geometry approximating the input scan grid; identifying key points on the intermediate grid; determining distances between the key points and candidate corresponding points on the input scan grid; determining a first surface normal associated with the key points and a second surface normal associated with the candidate corresponding points; calculating a matching score based on the distances between the key points and the candidate corresponding points, the first surface normal, and the second surface normal; selecting the candidate corresponding points based on the matching score; and adjusting the intermediate grid using the candidate corresponding points.

[0069] At least one embodiment of the disclosed technology includes: generating an input scan mesh associated with an object; generating an intermediate mesh associated with the object, the intermediate mesh having a geometry approximating the input scan mesh; identifying key points on the intermediate mesh; determining the distance between the key points and candidate corresponding points on the input scan mesh; determining a first surface normal associated with the key points and a second surface normal associated with the candidate corresponding points; calculating a matching score based on the distance between the key points and the candidate corresponding points, the first surface normal, and the second surface normal; selecting candidate corresponding points based on the matching score; and adjusting the intermediate mesh using the candidate corresponding points.

[0070] The disclosed techniques can be described in the context of computer-executable instructions that are executed by a computer or processor. Computer-executable instructions may correspond to computer program code, routines, programs, objects, software components, data structures, or portions of other types of computer-related structures that can be used to perform processes using a computer. The computer program code used to implement various operations or aspects of the disclosed techniques can be developed using one or more programming languages, including object-oriented programming languages ​​such as Java or C++, functional programming languages ​​such as Lisp, procedural programming languages ​​such as "C" or Visual Basic, or dynamic programming languages ​​such as Python or JavaScript. In some cases, the computer program code or machine-level instructions derived from it may be executed entirely on the end user's computer, partially on the end user's computer, partially on the end user's computer and partially on a remote computer, or entirely on a remote computer or server.

[0071] The flowcharts and block diagrams in the accompanying drawings provide illustrations of the implementable architecture, functionality, and operation of systems, methods, and computer program products according to various aspects of the disclosed technology. In this regard, each step in the flowchart may correspond to a portion of program module or computer program code, which may include one or more computer-executable instructions for implementing the specified function. In some embodiments, the functions indicated within the steps may occur in a non-linear order as shown in the drawings. For example, depending on the function involved, two consecutively shown steps may actually be performed substantially simultaneously, or the steps may sometimes be performed in reverse order. In some embodiments, steps may be omitted and other steps added without departing from the spirit and scope of the subject matter. In some embodiments, the functions mentioned within the steps may be implemented using hardware, software, or a combination of hardware and software. As examples, hardware may include microcontrollers, microprocessors, field-programmable gate arrays (FPGAs), and electronic circuitry.

[0072] For the purposes of this document, the term "processor" may refer to a physical hardware processor or a virtual processor, unless otherwise expressly stated. A virtual machine may include one or more virtual hardware devices, such as a virtual processor and virtual memory that communicates with the virtual processor.

[0073] For the purposes of this document, it should be noted that the dimensions of the various features depicted in the accompanying drawings may not necessarily be drawn to scale.

[0074] For the purposes of this document, references to “embodiment,” “one embodiment,” “some embodiments,” “another embodiment,” and other variations thereof in the specification may be used to describe various features, functions, or structures included in at least one or more embodiments, and do not necessarily refer to the same embodiment unless the context clearly indicates otherwise.

[0075] For the purposes of this document, a connection can be a direct connection or an indirect connection (e.g., via another part). In some cases, when a component is referred to as being connected to or coupled to another component, the component can be directly connected to the other component or indirectly connected to the other component via an intermediate component. When a component is referred to as being directly connected to another component, there is no intermediate component between the component and the other component.

[0076] For the purposes of this document, the term "based on" may be understood as "at least partially based on".

[0077] For the purposes of this document, the use of numerical terms such as “first” object, “second” object, and “third” object without additional context may not imply an order of objects, but may instead be used for identification purposes to identify or distinguish individual objects.

[0078] For the purposes of this document, the term "collection" can refer to a "set" of one or more objects.

[0079] For the purposes of this document, the phrases “the first object corresponds to the second object” and “the first object is relative to the second object” may refer to the first object and the second object being equivalent, similar, or related in character or function.

[0080] For the purposes of this document, the term “or” should be interpreted in both combined and separate contexts. Unless otherwise expressly stated, the list of items linked with the conjunction “or” should not be construed as requiring mutual exclusivity between items, but rather as “and / or”. As used herein, the terms “at least one,” “one or more,” and “and / or” are open-ended expressions that are both combined and separate in operation. The phrase “A and / or B” covers embodiments having a single element A, a single element B, or elements A and B together. The phrase “at least one of A, B, and C” covers embodiments having a single element A, a single element B, a single element C, elements A and B together, elements A and C together, elements B and C together, or elements A, B, and C together. Unless otherwise expressly stated, the indefinite articles “a” and “an” used herein should generally be interpreted as meaning “at least one” or “one or more.”

[0081] The various embodiments described above can be combined to provide further embodiments. These and other changes can be made to the embodiments based on the detailed description above. Generally, the terminology used in the following claims should not be construed as limiting the claims to the specific embodiments disclosed in the specification and claims, but should be interpreted to include all possible embodiments and the full scope of equivalents conferred by these claims. Therefore, the claims are not limited to this disclosure.

[0082] The disclosure presented herein covers the topics set forth in the following exemplary clauses.

[0083] Clause 1: A system for automating the retopology of a 3D mesh, comprising: a storage device for storing instructions, which, when executed, cause the system to perform operations including: generating an input scan mesh associated with an object; generating an intermediate mesh associated with the object, the intermediate mesh having a geometry approximating the input scan mesh; identifying key points on the intermediate mesh; determining distances between the key points and candidate corresponding points on the input scan mesh; determining a first surface normal associated with the key points and a second surface normal associated with the candidate corresponding points; calculating a matching score based on the distances between the key points and the candidate corresponding points, the first surface normal, and the second surface normal; selecting the candidate corresponding points based on the matching score; and adjusting the intermediate mesh using the candidate corresponding points.

[0084] Clause 2: The system according to Clause 1 further includes instructions that, when executed, cause the system to perform operations including: adjusting the key point to the candidate corresponding point; and using the candidate corresponding point to deform the intermediate mesh.

[0085] Clause 3: The system according to Clause 1 further includes instructions that, when executed, cause the system to perform operations including: generating a 3D mesh for the object by applying photometric refinement to the intermediate mesh.

[0086] Clause 4: The system according to Clause 1 further includes instructions that, when executed, cause the system to perform operations including: determining a surface curvature associated with a first surface at the key point; and calculating the matching score based on the surface curvature associated with the first surface at the key point.

[0087] Clause 5: The system according to Clause 1 further includes instructions that, when executed, cause the system to perform operations including: determining a vector difference between the first surface normal and the second surface normal; and calculating the matching score based on the vector difference.

[0088] Clause 6: The system according to Clause 1 further includes instructions that, when executed, cause the system to perform operations including: acquiring a set of images capturing different views of the object; and using the set of images to generate the input scan grid.

[0089] Clause 7: The system according to Clause 1 further includes instructions that, when executed, cause the system to perform operations including: restricting the candidate corresponding points to a threshold distance from the key points.

[0090] Clause 8: The system according to Clause 1, wherein: the object comprises a human head; and the intermediate grid has less surface area than the input scan grid.

[0091] Clause 9: The system according to Clause 1, wherein: the intermediate mesh comprises a deformable model fitted to the 3D surface of the input scan mesh using keypoint regression techniques.

[0092] Clause 10: The system according to Clause 1 further includes instructions that, when executed, cause the system to perform operations including: generating the intermediate mesh using a deep neural network via keypoint regression; determining a threshold distance based on the degree of uncertainty associated with the keypoint; shifting the keypoint by the threshold distance in the direction of the first surface normal; and identifying the candidate corresponding point with the highest matching score from the set of corresponding points.

[0093] Clause 11: A method for automating the retopology of a 3D mesh, comprising: generating an input scan mesh associated with an object; generating an intermediate mesh associated with the object, the intermediate mesh having a geometry approximating the input scan mesh; identifying key points on the intermediate mesh; determining distances between the key points and candidate corresponding points on the input scan mesh; determining a first surface normal associated with the key points and a second surface normal associated with the candidate corresponding points; calculating a matching score based on the distances between the key points and the candidate corresponding points, the first surface normal, and the second surface normal; selecting the candidate corresponding points based on the matching score; and adjusting the intermediate mesh using the candidate corresponding points.

[0094] Clause 12: The method according to Clause 11, wherein: adjusting the intermediate grid includes deforming the intermediate grid based on the position of the candidate corresponding point.

[0095] Clause 13: The method described in Clause 11 further includes: generating a 3D mesh for the object by applying a photometric refinement technique to the intermediate mesh.

[0096] Clause 14: The method according to Clause 11 further includes: determining the surface curvature associated with the first surface at the key point; and calculating a matching score based on the surface curvature associated with the first surface at the key point.

[0097] Clause 15: The method according to Clause 11 further includes: determining a vector difference between the first surface normal and the second surface normal; and calculating the matching score based on the vector difference.

[0098] Clause 16: The method according to Clause 11 further includes: acquiring a set of images capturing different views of the object; and using the set of images to generate the input scan grid.

[0099] Clause 17: The method according to Clause 11, wherein: the object comprises a human torso; and the intermediate grid has fewer surfaces than the input scan grid.

[0100] Clause 18: The method according to Clause 11, wherein: the intermediate mesh comprises a deformable model fitted to the 3D surface of the input scan mesh using keypoint regression techniques.

[0101] Clause 19: The method according to Clause 11 further includes: determining a threshold distance based on the degree of certainty associated with the keypoint; shifting the keypoint by the threshold distance in the direction of the first surface normal; and identifying candidate corresponding points with the highest matching score from the set of corresponding points.

[0102] Clause 20: A system for automating the retopology of a 3D mesh, comprising: a storage device configured to store a source 3D mesh; and a processing system communicating with the storage device, the processing system being configured to: generate a target 3D mesh having a geometry approximating the source 3D mesh; identify key points on the target 3D mesh; determine distances between the key points and candidate corresponding points on the source 3D mesh; determine a first surface normal associated with the key points and a second surface normal associated with the candidate corresponding points; determine a surface curvature difference between the first surface associated with the key points and the second surface associated with the candidate corresponding points; calculate a matching score based on the distance between the key points and the candidate corresponding points, the difference between the first surface normal and the second surface normal, and the surface curvature difference; select the candidate corresponding points based on the matching score; deform the target 3D mesh using the candidate corresponding points; and apply photometric refinement to the target 3D mesh after deforming the target 3D mesh.

Claims

1. A system for automating the retopology of 3D meshes, comprising: Storage device (271) for storing instructions that, when executed, cause the system to perform operations, including: Generate an input scan grid associated with the object; Generate an intermediate mesh associated with the object, the intermediate mesh having a geometry that approximates the input scan mesh; Identify key points on the intermediate grid; Determine the distance between the key point and the candidate corresponding point on the input scan grid; Determine the first surface normal associated with the key point and the second surface normal associated with the candidate corresponding point; A matching score is calculated based on the distance between the key point and the candidate corresponding point, the first surface normal, and the second surface normal. The candidate corresponding point is selected based on the matching score; and The intermediate grid is adjusted using the candidate corresponding points.

2. The system according to claim 1, further comprising instructions, which, when executed, cause the system to perform an operation, the operation including: Adjust the key points to the corresponding candidate points; as well as The intermediate mesh is deformed using candidate corresponding points.

3. The system according to any one of claims 1-2, further comprising instructions, which, when executed, cause the system to perform an operation, the operation including: A 3D mesh for the object is generated by applying photometric refinement to the intermediate mesh.

4. The system according to any one of claims 1-3, further comprising instructions, which, when executed, cause the system to perform an operation, the operation including: Determine the surface curvature associated with the first surface at the key point; as well as The matching score is calculated based on the surface curvature associated with the first surface at the key point.

5. The system according to any one of claims 1-3, further comprising instructions, which, when executed, cause the system to perform an operation, the operation including: Determine the vector difference between the first surface normal and the second surface normal; as well as The matching score is calculated based on the vector difference.

6. The system of claim 1, further comprising instructions, which, when executed, cause the system to perform an operation, the operation including: Acquire a set of images capturing different views of the object; as well as The input scan grid is generated using the set of images.

7. The system of claim 1, further comprising instructions, which, when executed, cause the system to perform an operation, the operation including: The candidate corresponding points are restricted to a threshold distance from the key points.

8. The system according to claim 1, wherein: The object includes a human head; and The intermediate grid has fewer surfaces than the input scan grid.

9. The system according to claim 1, wherein: The intermediate mesh comprises a deformable model fitted to the 3D surface of the input scan mesh using keypoint regression technology.

10. The system of claim 1, further comprising instructions, which, when executed, cause the system to perform an operation, the operation including: The intermediate grid is generated using a deep neural network via keypoint regression; The threshold distance is determined based on the degree of uncertainty associated with the key point; The key point is shifted by the threshold distance in the direction of the first surface normal; as well as Identify the candidate corresponding point with the highest matching score from the set of corresponding points.

11. A method for automating the topology of 3D meshes, comprising: Generate (304) the input scan grid associated with the object; Generate (306) an intermediate mesh associated with the object, the intermediate mesh having a geometry that approximates the input scan mesh; Identify key points on the intermediate grid described in (308); Determine the distance between the key point (310) and the candidate corresponding point on the input scan grid; Determine (312) the first surface normal associated with the key point and the second surface normal associated with the candidate corresponding point; A matching score (314) is calculated based on the distance between the key point and the candidate corresponding point, the first surface normal, and the second surface normal. The candidate corresponding point is selected based on the matching score (316); as well as The intermediate grid (318) is adjusted using the candidate corresponding points.

12. The method according to claim 11, wherein: Adjusting the intermediate grid includes deforming the intermediate grid based on the position of the candidate corresponding points.

13. The method according to any one of claims 11-12, further comprising: A 3D mesh for the object is generated by applying photometric refinement techniques to the intermediate mesh.

14. The method according to any one of claims 11-13, further comprising: Determine the surface curvature associated with the first surface at the key point; as well as The matching score is calculated based on the surface curvature associated with the first surface at the key point.

15. The method according to any one of claims 11-13, further comprising: Determine the vector difference between the first surface normal and the second surface normal; as well as The matching score is calculated based on the vector difference.

16. The method of claim 11, further comprising: Acquire a set of images capturing different views of the object; as well as The input scan grid is generated using the set of images.

17. The method of claim 11, wherein: The object includes the human torso; and The intermediate grid has fewer surfaces than the input scan grid.

18. The method according to claim 11, wherein: The intermediate mesh comprises a deformable model fitted to the 3D surface of the input scan mesh using keypoint regression technology.

19. The method of claim 11, further comprising: The threshold distance is determined based on the degree of certainty associated with the key point; The key point is shifted by the threshold distance in the direction of the first surface normal; as well as Identify the candidate corresponding point with the highest matching score from the set of corresponding points.

20. A system for automating the retopology of 3D meshes, comprising: Storage device (271), which is configured as a storage source 3D grid; as well as The processing system (270) communicating with the storage device is configured to: Generate a target 3D mesh with a geometry that approximates the source 3D mesh; Identify key points on the target 3D mesh; Determine the distance between the key point and the candidate corresponding point on the source 3D mesh; Determine the first surface normal associated with the key point and the second surface normal associated with the candidate corresponding point; Determine the surface curvature difference between the first surface associated with the key point and the second surface associated with the candidate corresponding point; The matching score is calculated based on the distance between the key point and the candidate corresponding point, the difference between the first surface normal and the second surface normal, and the surface curvature difference. The candidate corresponding point is selected based on the matching score; The target 3D mesh is deformed using the candidate corresponding points; as well as After deforming the target 3D mesh, photometric refinement is applied to the target 3D mesh.

Images