Non-rigid registration method and system for implementing dynamic 3D meshes

Abstract

A non-rigid alignment method for implementing a dynamic 3D mesh according to one embodiment of the present invention comprises: a step of performing skeleton-based hierarchical alignment between a source mesh and a target mesh based on a skeleton tree structure; a step of generating a UV map for the source mesh and generating a normal map based on the UV map; and a step of performing correction on the target mesh based on the normal map using a deep learning-based correction network.

Description

Technology Field

[0001] The present invention relates to a technology for constructing a 3D model of a dynamic object, and specifically, to a technology that improves the accuracy and efficiency of registration by using Multi-View Stereo (MVS)-based features and skeleton-based hierarchical registration techniques while performing non-rigid registration of mesh sequences extracted from multi-view images. Background Technology

[0003] 3D modeling and registration techniques for dynamic objects are widely used in movies, games, virtual reality, medical simulations, and the like. In particular, non-rigid registration techniques that generate 3D dynamic object models using image data extracted from multiple viewpoints are widely utilized.

[0004] However, conventional non-rigid registration techniques suffer from cumulative errors occurring at the endpoints. Specifically, because existing Iterative Closest Point (ICP)-based methods perform registration based on adjacent vertices, cumulative errors occur at distant endpoints such as hands or feet; this compromises the structural consistency of the model and degrades registration quality.

[0005] Furthermore, conventional non-rigid alignment techniques suffer from the problem of incomplete data recovery. Incomplete data occurs due to occluded areas or limited viewpoints in multi-view setups, and existing methods are unable to accurately recover this missing data. Although alignment is primarily performed using the average deformation of surrounding vertices, it has limitations in that the quality of the alignment is poor due to a lack of detailed recovery.

[0006] Therefore, there is a growing need for technology that overcomes the aforementioned limitations of conventional technology and provides more precise and consistent non-rigid alignment. Prior art literature

[0008] Korean Patent Publication No. 10-2022-0181230 (Publication Date: January 31, 2024) The problem to be solved

[0009] One technical aspect of the present invention aims to solve the problems of the aforementioned prior art by providing a non-rigid alignment technology for dynamic 3D mesh implementation that can provide more accurate dynamic modeling by utilizing more precise and consistent non-rigid alignment for 3D dynamic objects.

[0010] One technical aspect of the present invention aims to provide a non-rigid alignment technology for dynamic 3D mesh implementation that can reduce cumulative errors occurring in the terminal regions and achieve high alignment accuracy even in regions with a lot of dynamic deformation through a hierarchical alignment method utilizing the tree structure of a human skeleton.

[0011] One technical aspect of the present invention aims to provide a non-rigid alignment technique for implementing a dynamic 3D mesh that can provide robust alignment against external factors such as lighting changes, shadows, and occlusion by utilizing multi-viewpoint stereo (MVS) features.

[0012] One technical aspect of the present invention aims to provide a non-rigid alignment technique for dynamic 3D mesh implementation that utilizes UV mapping to restore surface details of a hidden area and enables precise restoration while maintaining structural consistency.

[0013] One technical aspect of the present invention aims to provide a non-rigid alignment technique for dynamic 3D mesh implementation that can ensure structural consistency of the aligned mesh sequence and provide various application possibilities by performing alignment and deformation with the remaining meshes while maintaining the topology of the first mesh.

[0014] The above-mentioned objectives and various advantages of the present invention will become more apparent to those skilled in the art from the preferred embodiments of the present invention. means of solving the problem

[0016] One technical aspect of the present invention proposes a non-rigid alignment method for implementing a dynamic 3D mesh. The non-rigid alignment method for implementing a dynamic 3D mesh comprises: a step of performing skeleton-based hierarchical alignment between a source mesh and a target mesh based on a skeleton tree structure; a step of generating a UV map for the source mesh and generating a normal map based on the UV map; and a step of performing correction on the target mesh based on the normal map using a deep learning-based correction network.

[0017] In one embodiment, the step of performing skeleton-based hierarchical alignment between a source mesh and a target mesh based on the skeleton tree structure may include: identifying major joint points based on the skeleton for the source mesh and the target mesh; starting alignment with respect to a center for the major joint points; and performing alignment while extending from the center to the ends of the major joint points.

[0018] In one embodiment, the step of generating a UV map for the source mesh and generating a normal map based on the UV map may include: a step of unfolding the surface of the 3D model of the source mesh into a 2D space; and a step of assigning 2D UV coordinates to each vertex unfolded into the 2D space.

[0019] In one embodiment, the step of generating a UV map for the source mesh and generating a normal map based on the UV map may further include the step of mapping surface information of the 3D model to the normal map based on assigned 2D UV coordinates.

[0020] In one embodiment, the step of generating a UV map for the source mesh and generating a normal map based on the UV map may further include the step of setting a hidden area of ​​the target mesh by referring to the normal map information of the source mesh.

[0021] In one embodiment, the step of performing correction on the target mesh based on the normal map using the deep learning-based correction network may include: performing initial vertex matching based on the UV maps of the source mesh and the target mesh; calculating an energy item to adjust the position of each vertex; and adjusting the vertex position according to the energy item.

[0022] In one embodiment, the energy item is calculated through the following mathematical formula, and

[0023]

[0024] Here, represents rigid body energy, soft energy, data energy, boundary energy, and MVS feature energy, respectively.

[0025] In one embodiment, the vertex information for the i-th pixel pi of the normal map is a mathematical formula

[0026] Determined according to, and here, v i1 ,v i2 ,v i3 is pixel p in the UV map i These are the vertices forming the triangle surrounding , and α i ,β i ,γ i is p i These are the barycentric coordinates for the three vertices.

[0027] In one embodiment, the center coordinate α i ,β i ,γ i is, mathematical formula

[0028]

[0029] Satisfying, and here, is the area of ​​the triangle formed by vertices v1, v2, and v3, and , and *?* are each p i It is the area of ​​the sub-triangle containing.

[0030] In one embodiment, the p i Normal vector N(p) for i ) is, mathematical formula

[0031] Satisfies.

[0032] Another technical aspect of the present invention proposes a non-rigid alignment system for dynamic 3D mesh implementation. The non-rigid alignment system for dynamic 3D mesh implementation comprises: a joint-based alignment model that performs skeleton-based hierarchical alignment between a source mesh and a target mesh based on a skeleton tree structure; and a normal map-based correction model that generates a UV map for the source mesh, generates a normal map based on the UV map, and performs correction for the target mesh based on the normal map using a deep learning-based correction network.

[0033] In one embodiment, the joint-based alignment model can identify major joint points based on the skeleton for the source mesh and the target mesh, start alignment with respect to the major joint points with respect to the center, and then perform alignment while extending from the center to the ends of the major joint points.

[0034] In one embodiment, the normal map-based correction model can unfold the surface of the 3D model of the source mesh into a 2D space and assign 2D UV coordinates to each vertex unfolded into the 2D space.

[0035] In one embodiment, the normal map-based correction model can map surface information of the 3D model to the normal map based on assigned 2D UV coordinates.

[0036] In one embodiment, the normal map-based correction model can set the occluded area of ​​the target mesh by referring to the normal map information of the source mesh.

[0037] In one embodiment, the normal map-based correction model performs initial vertex matching based on the UV maps of the source mesh and the target mesh, calculates an energy item to adjust the position of each vertex, and then adjusts the vertex position according to the energy item.

[0038] In one embodiment, the energy item is calculated through the following mathematical formula, and

[0039]

[0040] Here, represents rigid body energy, soft energy, data energy, boundary energy, and MVS feature energy, respectively.

[0041] In one embodiment, the vertex information for the i-th pixel pi of the normal map is a mathematical formula

[0042] Determined according to, and here, v i1 ,v i2 ,v i3 is pixel p in the UV map i These are the vertices forming the triangle surrounding , and α i ,β i ,γ i is p i These are the barycentric coordinates for the three vertices.

[0043] In one embodiment, the center coordinate α i ,β i ,γ i is, mathematical formula

[0044]

[0045] Satisfying, and here, is the area of ​​the triangle formed by vertices v1, v2, and v3, and , and are respectively p i It is the area of ​​the sub-triangle containing.

[0046] In one embodiment, the p i Normal vector N(p) for i ) is, mathematical formula

[0047]

[0048] Satisfies. Effects of the invention

[0050] According to one embodiment of the present invention, there is an effect of providing more accurate dynamic modeling by using more precise and consistent non-rigid matching for three-dimensional dynamic objects.

[0051] According to one embodiment of the present invention, a hierarchical alignment method utilizing the tree structure of a human skeleton has the effect of reducing cumulative errors occurring in the distal regions and achieving high alignment accuracy even in regions with significant dynamic deformation.

[0052] According to one embodiment of the present invention, by utilizing multi-viewpoint stereo (MVS) features, it is possible to provide robust alignment against external factors such as lighting changes, shadows, and occlusion phenomena.

[0053] According to one embodiment of the present invention, surface details of a hidden area are restored using UV mapping, and precise restoration is possible while maintaining structural consistency.

[0054] According to one embodiment of the present invention, by performing alignment and deformation with the remaining meshes while maintaining the topology of the first mesh, the structural consistency of the aligned mesh sequence is ensured, and various application possibilities can be provided.

[0055] The effects of the invention described above are not limited to all the various effects of this application, and these various effects can be easily understood from the specific embodiments in the detailed description. Brief explanation of the drawing

[0057] FIG. 1 is a configuration diagram illustrating a non-rigid matching system for implementing a dynamic three-dimensional mesh according to an embodiment of the present invention. FIG. 2 is a flowchart illustrating a non-rigid alignment method for implementing a dynamic three-dimensional mesh according to an embodiment of the present invention. FIG. 3 is a drawing illustrating an example of a tree hierarchy of joints and a 3D human mesh according to an embodiment of the present invention. Figure 4 is a diagram showing the results of a transformation using MVS (Multi-View Stereo) features and a comparison of other methods. FIG. 5 is a flowchart illustrating a non-rigid alignment method for implementing a dynamic three-dimensional mesh according to an embodiment of the present invention. FIG. 6 is a flowchart illustrating a skeleton-based hierarchical alignment method according to one embodiment of the present invention. FIG. 7 is a flowchart illustrating a method for generating a normal map based on a UV map according to an embodiment of the present invention. FIG. 8 is a flowchart illustrating a method for restoring a hidden area of ​​a target mesh according to an embodiment of the present invention. FIG. 9 is a flowchart illustrating a normal map-based correction method based on a deep learning-based correction network according to an embodiment of the present invention. FIG. 10 is a diagram illustrating an exemplary computing operating environment of a system server according to an embodiment of the present invention. Figure 11 is a diagram illustrating examples of experimental results for each method on a data set. Figure 12 is a table illustrating quantitative figures in this dataset. Figure 13 is a diagram comparing the results of applying and not applying hierarchical joint-based alignment to evaluate the effect of the hierarchical structure. Figure 14 shows a table illustrating the advantages of hierarchical joint-based alignment. Figure 15 is a diagram visualizing the matching results depending on whether normal map diffusion is used, in order to emphasize the effect of phase consistency restoration using UV normal map diffusion. Specific details for implementing the invention

[0058] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims.

[0059] The terms used in this specification will be briefly explained, and the invention will be described in detail.

[0060] The terms used in this invention have been selected based on currently widely used general terms, taking into account their functions within the invention; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this invention should be defined not merely by their names, but based on their meanings and the overall content of the invention.

[0061] Throughout the specification, when a part is described as "comprising" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, terms such as "part," "module," and "unit" used in the specification refer to a unit that processes at least one function or operation and may be implemented as software, hardware components such as FPGAs or ASICs, or a combination of software and hardware. However, the terms "part," "module," and "unit" are not limited to software or hardware. "Part," "module," and "unit" may be configured to reside in an addressable storage medium or configured to run one or more processors. Accordingly, as an example, terms such as "part," "module," and "unit" include components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

[0062] Below, embodiments of the present invention are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the invention. Additionally, parts of the drawings that are irrelevant to the description are omitted to clearly explain the invention.

[0063] Terms including ordinal numbers, such as "first," "second," etc., may be used to describe various components, but the components are not limited by the terms. The terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of multiple related items or any one of the multiple related items.

[0065] Non-rigid registration is essential for mesh reconstruction of 3D dynamic objects, but existing methods struggle to achieve registration accuracy in complex and deformable scenes.

[0066] The present invention proposes a non-rigid registration system for dynamic 3D mesh implementation (hereinafter abbreviated as ‘non-rigid registration system’ or ‘system’) based on a new two-stage framework combining Hierarchical Joint-Based Registration and Topology-Consistent Restoration, and a non-rigid registration method for dynamic 3D mesh implementation performed thereon.

[0067] FIG. 1 is a configuration diagram illustrating a non-rigid alignment system for implementing a dynamic 3D mesh according to an embodiment of the present invention. Referring to FIG. 1, the non-rigid alignment system (100) for implementing a dynamic 3D mesh according to an embodiment of the present invention may include a joint-based alignment model (110) and a normal map-based correction model (120).

[0068] The joint-based registration model (110) utilizes skeletal-based hierarchical registration, i.e., hierarchical joint-based registration, to perform registration between a source mesh and a target mesh. In hierarchical joint-based registration, joint positions are integrated as hierarchical prior information to prioritize high-motion regions and achieve higher precision in extremity regions such as hands and feet. This hierarchical awareness optimization is supported by powerful deep features extracted from Multi-View Stereo (MVS), which can enhance registration consistency and surface detail even in variable lighting and occluded regions.

[0069] The normal map-based correction model (120) can generate a UV map and then restore surface details between the source mesh and the target mesh. The normal map-based correction model (120) reconstructs details based on the normal map in the occluded areas of the target mesh and can enhance visual details by utilizing deep learning-based correction techniques. In topological consistency restoration, topological consistency can be ensured by applying a diffusion model to reconstruct vertices in incomplete data areas using UV mapping. By spreading the normal map to the missing vertex areas, very detailed and visually accurate mesh deformation can be achieved. The present invention not only closely matches human perceptual expectations but also demonstrates robustness in complex scenarios and can provide a new standard for non-rigid mesh alignment for three-dimensional dynamic objects.

[0070] The following description uses a human mesh as an example of a dynamic object, but is not limited thereto.

[0071] Figure 2 is a drawing illustrating an example of a matching case.

[0072] Figure 2 illustrates one ideal case and two failure cases when alignment is performed according to the conventional method. Sample source vertices (orange) are mainly paired with ICP-based vertices (red), and black arrows indicate the direction of movement of the target mesh. The green area is the ideal case, where the source mesh is well deformed. However, in the red area, mesh deformation failure occurs due to movement being too large (Failure case 1) or movement parallel to the surface (Failure case 2).

[0073] In dynamic sequences, fragmented data frequently occurs where target vertex locations are missing due to occlusion and limited viewpoints. Existing methods often rely on rough estimates for these incomplete regions, which compromises the consistency of surface details and leads to a degradation in reconstruction quality.

[0074] To address these issues, one embodiment of the present invention proposes a two-stage algorithm specialized for non-rigid registration of dynamic 3D human meshes. The first stage utilizes a Hierarchical Joint-Based Registration approach that guides initial alignment and improves the precision of surface details by leveraging the tree structure of the skeleton and multi-viewpoint stereo (MVS) features. The second stage utilizes a Topology-Consistent Restoration process that restores vertices in incomplete data regions through UV mapping. This two-stage approach ensures structural integrity and improves surface fidelity even in high-deformation regions.

[0075] In a hierarchical joint-based alignment according to one embodiment of the present invention, alignment is conditioned based on a tree hierarchy of skeletal joints to improve the accuracy of initial alignment in the distal region and reduce cumulative errors throughout the structure. To overcome the limitations of conventional color-based surface detail enhancement, which is susceptible to inaccuracies caused by shadows, lighting changes, and occlusion, one embodiment of the present invention may integrate MVS deep features. These features provide stable vertex representation even under variable lighting conditions, enabling precise comparison of the source and target and maintaining high-fidelity details during deformation.

[0076] Partial mesh problems can be solved through phase-consistent restoration according to one embodiment of the present invention. Here, missing target vertices require accurate processing. By using UV mapping, regions containing missing vertex information can be finely restored while maintaining a consistent phase. Meshes sharing the same phase can generate a normal map using a common UV map, which can then be finely refined through diffusion-based restoration (inpainting). This process restores vertices in regions containing missing surface details, thereby improving deformation accuracy.

[0077] A comprehensive framework according to one embodiment of the present invention addresses the shortcomings of existing methods by combining hierarchical joint-based registration and topology-consistent restoration. This approach aligns with human perceptual expectations and can provide robust performance in various poses, high-deformation regions, and partial mesh scenarios. Experimental results demonstrate the effectiveness of this method and can establish a new standard for non-rigid registration in dynamic 3D human modeling.

[0078] Related Work

[0079] Non-rigid registration is essential for reconstructing dynamic and highly deformable scenes, particularly in multi-viewpoint setups. These scenes present unique challenges. The traditional approach, the Iterative Closest Point (ICP) framework, uses proximity-based deformation to align source and target meshes. While this method is effective for rigid registration, it tends to result in lower registration fidelity in high-deformation regions due to cumulative errors and partial visibility. Recently, optimized strategies, hierarchical frameworks, and multi-viewpoint feature integration have been introduced to address these limitations.

[0080] The ICP algorithm has long been utilized as the fundamental approach to alignment, but its inherent limitations have become apparent as it has been extended to non-rigid applications. Amberg et al. proposed a method to extend ICP to non-rigid contexts to progressively adapt responses across deformations, but relied on proximity-based alignment, resulting in degraded alignment fidelity in fast-moving extremity regions such as hands. Mitra et al. handled large-scale deformations through 4D spatiotemporal surface integration, but scalability was limited due to computational cost. To address partial visibility in multi-view environments, Wand et al. proposed a non-linear optimization framework, but its application to large datasets is limited due to high computational costs.

[0081] One embodiment of the present invention, based on such existing research, introduces a hierarchical strategy to increase accuracy in the distal skeletal region and integrates a powerful framework for processing partial scans. This two-stage solution mitigates cumulative errors and supports efficient alignment for large dynamic datasets.

[0082] Sumner et al.'s embedded deformation method plays a significant role in non-rigid registration. It conveys natural movement through a skeletal structure, maintains spatial consistency through affine transformations at each node, and guarantees the integrity of the original form even in high-deformation scenarios. However, there are still limitations in handling continuity in occluded regions.

[0083] Hierarchical joint-based alignment according to one embodiment of the present invention establishes an initial approximate deformation by utilizing a skeletal joint structure. By conditioned alignment according to the skeletal hierarchy, cumulative errors in the distal regions are minimized, and alignment suitable for the complex topology of the human form is achieved. In addition, by combining ICP and MVS-based feature refinement, the precision lacking in existing implicit deformation methods in high-motion and detailed regions can be compensated for.

[0084] MVS-based feature matching technology improves correspondence in complex multi-viewpoint setups by providing stable feature representation under various lighting conditions. This approach enhances matching accuracy even in situations where existing texture and color information may be degraded.

[0085] Optimization techniques also aim to further enhance the robustness of the alignment. By proposing an Accelerated Principal-Minimize (AMM) algorithm for non-rigid alignment based on the Welsch function, robust performance against noise and disturbances can be provided. One embodiment of the present invention balances computational efficiency and alignment accuracy based on this optimization method, and integrates hierarchical deformation and UV-based reconstruction to achieve smooth and consistent deformation even in partially occluded regions.

[0087] Non-Rigid Registration

[0088] When capturing human motion using multi-view RGB video, conventional non-rigid registration techniques often rely on rigid, smooth, fitting, and bound loss functions. These loss functions identify structural mismatches between the source mesh (S) and the target mesh (T) within an Iterative Closest Point (ICP)-based framework and achieve registration by progressively modifying the vertex positions of the source mesh. However, existing methods have two major limitations.

[0089] The first limitation is the accumulation of deformation errors caused by skeletal movement. During the deformation process, target vertices are assigned to source vertices to bring them closer to the target. As the spatial distance between the source and target vertices increases, the confidence in this vertex pair decreases. Therefore, distance-based thresholding is applied to prioritize closer vertices during initial deformation, while the adjustment of more distant vertices is delayed. Even if the interaction of rigidity, smoothness, and fit loss seeks optimal vertex positions, vertices deformed later become more vulnerable to accumulated errors, which can lead to noticeable artifacts and alignment errors.

[0090] FIG. 3 is a drawing illustrating an example of a tree hierarchy of joints and a 3D human mesh according to an embodiment of the present invention.

[0091] The aforementioned problem can be solved by utilizing the tree hierarchy of the human skeleton. As shown in Fig. 3, the human skeleton can naturally correspond to a tree hierarchy. The root, located at the lower center of the spine, provides core stability to the body and can distribute movement to extremity regions such as the limbs and head. This central region primarily maintains balance and stability, and movement can be minimized. Consequently, vertices located further from the root experience greater deformation and distance differences, and errors can accumulate in the extremity regions of the hierarchy. To mitigate these structural problems, one embodiment of the present invention can apply initial approximate deformation by utilizing skeletal joint information. A deformation graph can be generated first, and each node can be connected to the nearest skeletal joint index. By using the positional difference between the source and target joint coordinates to apply approximate and structure-maintaining deformation to the source mesh, subsequent accumulated errors can be minimized.

[0092] A second limitation may occur when only partial visibility of the target mesh is available. Incomplete data (e.g., missing vertices of elbows) results in unpaired vertices in the source mesh, which may lack reliable correspondences. Typically, only approximate transformations are achieved for these vertices based on the movement of nearby vertices, leading to reduced deformation accuracy. To address this, one embodiment of the present invention utilizes a finely tuned diffusion network in the post-processing step to restore missing regions while maintaining surface consistency. Considering that non-rigid registration aims to maintain mesh topology during deformation, one embodiment of the present invention may utilize a consistent UV map shared by the source and the deformed mesh.

[0093] Through such an embodiment of the present invention, it is possible to optimize source vertices with reliable correspondences while simultaneously distinguishing unpaired vertices. By mapping all vertices into a shared UV space, surface normal vectors and vertex locations can be recorded within the pixels of the UV map to generate a detailed normal map. Areas corresponding to missing vertices are masked, and these areas can be restored using a finely tuned reconstruction network to ensure smooth continuity and reduce the visibility of missing data.

[0095] Hierarchical Joint-Based Registration

[0096] The initial stage of Hierarchical Joint-Based Registration predicts human 3D pose through multi-viewpoint images, and each joint This is the process of extracting coordinates. This skeletal data can provide spatial constraints to structure the deformation process according to the hierarchical characteristics of the human skeleton. For each graph node in the source mesh, the index of an adjacent joint is calculated and aligned with the corresponding joint in the target mesh to apply deformation based on the difference in joint positions. This enables deformation that reflects the core structure of the human body in the initial stages, allowing the source mesh to adapt to basic skeletal movements before performing detailed adjustments.

[0097] After setting the source mesh aligned as described above to an initialized position, existing embedded deformation can be applied. Generally, this deformation process relies heavily on the ICP algorithm to define target vertex positions and sequentially deform the mesh based on them. However, existing ICP has limitations in capturing complex geometric details and falls short, particularly in areas requiring fine detail. To compensate for these limitations, previous studies have introduced color information to utilize surface patterns as additional matching cues; however, color-based correspondence can be distorted under dynamic lighting conditions or in the presence of shadows, which can reduce matching accuracy.

[0098] Figure 4 is a diagram showing the results of a transformation using MVS (Multi-View Stereo) features and a comparison of other methods.

[0099] Figure 4(a) shows the reference data, Scan GT, which serves as an evaluation criterion for assessing the accuracy of the alignment. (b) shows the results of a conventional traditional method, and (c) shows the results of a method utilizing color information. (d) shows the results of an example utilizing MVS features according to an embodiment of the present invention. In the top row, the face shape is generally maintained, but it can be seen that while details such as the eyes are distorted in the traditional method (b), they are represented more accurately in the embodiment of the present invention (d). Similar results are observed in the sleeve area in the bottom row as well, and through this comparison, it can be confirmed that the MVS-based deformation of an embodiment of the present invention can handle complex surface details that are difficult to process with existing methods or color-based methods.

[0100] Conventional ICP has inherent limitations in processing fine geometric details, making it difficult to align particularly thin or complex regions (e.g., hands). In one embodiment of the present invention, this problem is solved by introducing additional guidelines in MVS features and vertex normals.

[0101] In one embodiment of the present invention, ICP is used to narrow down target vertex candidates, and then the target vertex is finally refined using MVS features and vertex normals. This enables more reliable vertex alignment even in areas where precision is low with ICP alone, and improves the overall fidelity of the deformation.

[0102] The overall deformation process according to one embodiment of the present invention defines an energy-based objective function to optimize various elements in a balanced manner. The objective function is defined as shown in the following mathematical formula 1.

[0103] [Mathematical Formula 1]

[0104]

[0105] Here, represents rigid body energy, soft energy, data energy, boundary energy, and MVS feature energy, respectively.

[0106] Each term of the objective function contributes to various aspects of the transformation. and It ensures structural consistency, and It sorts vertices based on source-target correspondence, and It constrains the boundary region, It integrates MVS feature information to support accurate alignment on complex surfaces.

[0107] An approach according to one embodiment of the present invention can improve the accuracy and visual quality of a deformation mesh by combining rigid skeletal alignment and MVS-based precision deformation. Hierarchical alignment in the initial skeletal data can establish a strong structural foundation that reflects the hierarchical movements of human anatomy. Subsequently, MVS features can be integrated to precisely adjust the initial deformation and achieve vertex correspondence that matches the complex geometric details of the target mesh. This can mitigate the accumulation of errors often seen in existing methods and maintain the structural and perceptual consistency of the entire mesh.

[0109] Topology-Consistent Restoration

[0110] Partial or incomplete 3D mesh data presents complex challenges in non-rigid registration, requiring a method to infer realistic transformations in data-missing regions as well as accurate deformations in visible regions. Existing approaches, such as Embedded Deformation, approximate vertices in missing regions by relying on deformation parameters of visible vertices and estimate vertex motion in invisible regions by blending these transformations. However, this approach yields insufficient results when data is extensively missing, and simple interpolation alone fails to capture detailed motion patterns necessary to maintain structural consistency in highly deformed regions.

[0111] Topology-Consistent Restoration according to one embodiment of the present invention can enable more sophisticated and topology-aware deformation by utilizing the fixed topology of a 3D mesh. By recognizing the inherent structural consistency of the 3D mesh, this approach can approximate the movement of invisible vertices based on deformation patterns observed in adjacent visible regions. Consequently, this movement is transmitted smoothly across the mesh, and both geometric fidelity and structural continuity can be maintained during the deformation process.

[0112] To implement this, we can construct a stable reference space capable of accommodating both visible and inferred vertices by utilizing the topology shared by the source and target meshes through UV mapping. This UV map can identify regions containing missing vertices and enable inference of surface properties based on adjacent visible regions. Using this mapped space, we generate a normal map that captures the surface geometric details of the visible regions. We can identify missing regions in the normal map and apply a mask to those areas to specify locations requiring restoration. The normal values ​​of the masked regions are calculated based on information obtained from adjacent visible regions. The i-th pixel p in the UV-mapped normal map i Vertex information for is calculated as shown in the following mathematical formula 2.

[0113] [Mathematical Formula 2]

[0114]

[0115] Here v i1 ,v i2 ,v i3 is pixel p in the UV map i These are the vertices forming the triangle surrounding , and α i ,β i ,γ i is p iThese are the barycentric coordinates for the three vertices.

[0116] The center coordinates are defined as in the following mathematical formula 3.

[0117] [Mathematical Formula 3]

[0118]

[0119] Here is the area of ​​the triangle formed by vertices v1, v2, and v3, and , and are respectively p i It is the area of ​​the sub-triangle containing.

[0120] Therefore, p i The normal vector for is calculated as shown in the following mathematical formula 4.

[0121] [Mathematical Formula 4]

[0122]

[0123] Here, is pixel p i It is the interpolated normal of, and the center coordinate α i ,β i ,γ i Using vertex v i1 ,v i2 ,v i3 It can be calculated as the weighted sum of the normals.

[0124] This initial calculation provides a baseline normal map that fills the entire masked area with basic information. For the inpainting process, a finely tuned Stable Diffusion model DN specifically applied to this task can be used. The Stable Diffusion model operates on the masked normal map to restore missing normal values ​​in a manner consistent with the structural context provided by neighboring areas using the learned diffusion process.

[0125] This inpainting process can be expressed by the following mathematical formula 5.

[0126] [Mathematical Formula 5]

[0127]

[0128] Here, N is the original normal map, M is a binary mask representing the missing area (M = 1 for missing pixels, M = 0 for given pixels), and N' is the restored, i.e., inpainted normal map.

[0129] This model can generate contextually consistent values ​​based on visible data to iteratively refine normals within masked areas, ultimately producing a smooth and consistent normal map that accurately reflects the intended geometry of the mesh. At each iteration step t, the inpainting process can update the masked normal values ​​as shown in the following Equation 6.

[0130] [Mathematical Formula 6]

[0131]

[0132] The restored normal map N' is used to adjust vertex positions across the mesh. The position v' of each vertex v is adjusted as shown in Equation 7 below, based on the difference between the normal vector N(f) of the original face and the restored normal vector N'(f) across neighboring faces.

[0133] [Mathematical Formula 7]

[0134]

[0135] Here, F is the set of faces adjacent to vertex v, and λ is a weight that adjusts the influence of the restored normal. This Equation 7 transfers the structural consistency inferred from the visible region to the invisible region, ensuring stability and alignment during the mesh deformation process.

[0136] Repeating the above process refines the deformation more finely and ensures consistency in both visible and invisible regions of the mesh. The restored normal map N' not only restores surface details but also provides a reliable foundation for non-rigid registration, enabling accurate reconstruction even in cases of severe data loss.

[0138] FIG. 5 is a flowchart illustrating a non-rigid alignment method for implementing a dynamic three-dimensional mesh according to an embodiment of the present invention.

[0139] The non-rigid alignment method for implementing a dynamic 3D mesh illustrated in FIG. 5 (hereinafter abbreviated as 'non-rigid alignment method') can be performed in the system illustrated in FIG. 1.

[0140] Referring to FIG. 5, the non-rigid alignment method includes the step of performing skeleton-based hierarchical alignment between a source mesh and a target mesh based on a skeleton tree structure (S510).

[0141] The non-rigid matching method includes the step of generating a UV map for a source mesh and calculating a normal map based on the UV map (S520).

[0142] The non-rigid matching method includes the step of performing correction on the target mesh using a normal map based on a deep learning-based correction network (S530).

[0143] Step S510, which performs skeleton-based hierarchical alignment, may use skeleton-based hierarchical alignment to minimize structural differences between a source mesh and a target mesh. The vertices of the source mesh are aligned with the target mesh, and to do this, alignment can be performed in a tree structure using joint data of a human skeleton.

[0144] Specifically, major joint points can be aligned first based on the skeletons of the source and target meshes. The alignment proceeds by starting from the center (e.g., spine) and extending toward the extremities; this prevents large deformations that may occur in the extremity regions and ensures the stability of the alignment process. Since the likelihood of cumulative errors increases with larger movements in distant regions such as hands and feet, this hierarchical alignment allows for securing structural stability against initial deformations.

[0145] By calculating a normal map based on a UV map and performing correction on the target mesh using the normal map based on a deep learning-based correction network (S520 to S530), accurate vertex positions in the occluded areas of the target mesh can be restored, and the details of the occluded areas can be reconstructed through a deep learning-based correction technique.

[0147] Hereinafter, with reference to FIGS. 6 to 9, various embodiments of a non-rigid alignment method for implementing a dynamic 3D mesh will be described.

[0149] FIG. 6 is a flowchart illustrating a skeleton-based hierarchical alignment method according to one embodiment of the present invention.

[0150] Referring to FIG. 6, the system (100) can prepare the alignment of major joint points based on the skeletons of the source mesh and the target mesh (S610). For example, the system (100) can identify major joint points based on the skeletons for the source mesh and the target mesh.

[0151] The system (100) can start aligning the major joint points with respect to the center (S620) and perform alignment while extending from the center to the ends of the major joint points (S630).

[0152] For example, the system (100) can set the first mesh frame as the source mesh and consider the remaining meshes as the target mesh to proceed with alignment. The system (100) can utilize the tree structure of the skeleton to sequentially extend alignment starting from the center to the end regions. The system (100) supports high-precision alignment by utilizing multi-view stereo (MVS) features and can provide robust alignment performance even against lighting changes and occlusion phenomena.

[0154] FIG. 7 is a flowchart illustrating a method for generating a normal map based on a UV map according to an embodiment of the present invention.

[0155] Referring to FIG. 7, the system (100) can unfold the surface of a 3D model of a source mesh into a 2D space (S710) and assign 2D UV coordinates to each vertex unfolded into the 2D space (S720).

[0156] The system (100) can map surface information of the 3D model to a normal map based on assigned 2D UV coordinates (S730).

[0157] Although the above content was explained with the source mesh in mind, it can also be applied to the target mesh.

[0158] That is, UV maps are generated first, and these UV maps are created by unfolding a 3D mesh onto a 2D plane, storing surface data while maintaining the same topological information between the source mesh and the target mesh. This provides a common reference space among meshes with the same topology and enables consistent surface alignment even if the mesh is deformed.

[0159] Subsequently, a normal map is generated based on the UV map, representing the surface orientation (vertex normal vector) of each mesh. This normal map can primarily be used to restore details in the occluded areas of the target mesh.

[0160] We will explain in more detail the UV map generation process and the concept of utilizing the topology shared by the source mesh and target mesh through the UV map.

[0161] A UV map is the process of unfolding the surface of a 3D object onto a 2D plane to create a coordinate system for mapping textures or aligning surface information.

[0162] As a method for generating a UV map, a step (S710) of unfolding the surface of a 3D mesh into 2D is performed first. Since the surface of a 3D model has a complex shape, an unwrapping process can be used to unfold it onto a 2D plane. That is, vertices, edges, and faces of the 3D model can be selected, and the surface can be transformed and unfolded into 2D space. Since it is necessary to maintain the information of the 3D surface while ensuring it is represented on a 2D plane with as little distortion as possible, the 3D surface can be appropriately cut and unfolded. During this unfolding process, the 3D coordinates of each vertex are converted to corresponding positions (U, V) in the 2D UV coordinate system.

[0163] Afterwards, 2D UV coordinates can be assigned to each vertex in 2D space (S720). 2D UV coordinates are assigned to each vertex, and based on these coordinates, surface information of the 3D model can be mapped to a 2D texture or normal map. Each vertex is positioned according to the U and V axes of the 2D plane, and these coordinates can be connected to a surface in 3D space.

[0164] Afterwards, a normal map can be applied (S730). The UV map generated as above can be used to apply textures, normal maps, displacement maps, etc. to the surface of a 3D model. Textures are mapped accurately based on the coordinates of the UV map, which allows the 3D model to be rendered more precisely with textures.

[0165] Through UV maps as described above, the source mesh and the target mesh can share topology. Utilizing the topology shared by the source and target meshes via UV maps implies that the two meshes possess the same vertex structure and connectivity (topology). This allows for consistent alignment between the two meshes.

[0166] Topology describes how vertices, edges, and faces are connected to one another within a mesh. For two meshes to share the same topology means that they have the same number of vertices and the same connection methods between them. This implies that even if the two meshes are deformed or posed differently, the same vertices will have the same connection structure at the same locations.

[0167] When a UV map is generated, each vertex of a 3D model has a corresponding 2D coordinate in the UV coordinate system, and these coordinates maintain the same topological information regardless of the mesh's deformation state. Therefore, if the UV map generated from the source mesh shares the same topology as the target mesh, the source and target meshes can be aligned using the same UV coordinate system. This allows for consistent results in alignment or restoration operations between the two meshes, as they share the same reference space in the UV coordinate system, even if the source and target meshes are deformed differently.

[0169] FIG. 8 is a flowchart illustrating a method for restoring a hidden area of ​​a target mesh according to an embodiment of the present invention.

[0170] Referring to FIG. 8, the system (100) generates a UV map and a normal map for a source mesh as described above with reference to FIG. 7 (S810), and can set a hidden area of ​​a target mesh by referring to the normal map information of the source mesh (S820).

[0171] In other words, for example, if you intend to restore the occluded areas of a target mesh after generating a UV map and applying a normal map to a source mesh, you can restore the occluded areas of the target mesh by referencing the source mesh's normal map information on the UV map because the source mesh and the target mesh share the same topology. This is possible because the two meshes have the same structural connection.

[0173] FIG. 9 is a flowchart illustrating a normal map-based correction method based on a deep learning-based correction network according to an embodiment of the present invention.

[0174] Referring to FIG. 9, the system (100) can perform initial vertex matching based on the UV maps of the source mesh and the target mesh (S910). Subsequently, the system (100) calculates an energy item to adjust the position of each vertex (S920), and can adjust the vertex position according to the energy item (S930).

[0175] In other words, after generating normal maps, the system can perform corrections using a deep learning-based correction network. The deep learning model precisely corrects vertex positions and normal vectors in occluded areas, ensuring structural consistency even in missing data or occluded regions. This process helps accurately restore complex surface details and provides visually consistent alignment results. Through this deep learning-based restoration process, fine surface information missed by conventional methods can be effectively restored, and the overall quality of the mesh can be significantly improved. This process can provide reliable alignment results even in situations where data is incomplete.

[0176] Below, the deep learning model is explained in more detail.

[0177] The deep learning model receives various data as input for the alignment and restoration of the source mesh and the target mesh. The main input data may include the UV map and normal map of the source mesh, the vertex coordinates and UV map of the target mesh, and deep features.

[0178] UV maps and normal maps generated from the source mesh are important input data that deep learning models reference in registration and restoration tasks. Through UV maps, the 2D coordinates of each vertex and 3D surface information can be connected, and through normal maps, the surface direction (normal vector) of each vertex can be identified.

[0179] The vertex coordinates and UV map of the target mesh can also be used as input. This data provides the information necessary to restore occluded or lost parts in the target mesh by comparing them with the source mesh.

[0180] Deep learning models can utilize deep features to find vertex pairs between a source mesh and a target mesh, for example, using deep features extracted from Multi-View Stereo (MVS). This allows for the identification of surface information that is robust to lighting changes or shadows.

[0181] Deep learning-based calibration can perform vertex adjustment by performing energy-based optimization through the following procedure.

[0182] First, initial alignment and depth feature matching can be performed. A deep learning model can perform initial vertex alignment based on the UV maps of the source mesh and the target mesh. Here, the vertices of the target mesh can be adjusted to be correctly aligned by referencing the normal map of the source mesh.

[0183] Deep features can be used to find vertex pairs between a source mesh and a target mesh, enabling more sophisticated source-target vertex matching than existing methods that perform matching using only color or proximity information.

[0184] Subsequently, energy-based optimization can be performed. Energy-based optimization is a crucial step for adjusting the position of each vertex, allowing for adjustment by optimizing the following energy parameters. Rigid Energy (E_rigid) constrains the mesh to deform like a rigid body, preventing distortion of the overall structure. Smoothness Energy (E_smooth) ensures smooth deformation between adjacent vertices, making the overall deformation appear natural. Data Energy (E_data) maximizes the alignment of vertex positions between the source and target meshes, thereby increasing the accuracy of the alignment. Boundary Energy (E_bound) ensures consistent alignment of vertices at the mesh boundaries. MVS Feature Energy (E_MVS): Reflects features extracted from Multi-View Stereo (MVS) to ensure accurate reconstruction even in the occluded areas of the target mesh. Each of these energy parameters helps the deep learning model perform vertex alignment between the source and target meshes more accurately, and the optimized energy allows for the appropriate adjustment of vertices.

[0185] Subsequently, a vertex adjustment process can be performed. After the deep learning model matches pairs of vertices between the source mesh and the target mesh, the positions of the vertices are adjusted through energy-based optimization.

[0186] The vertex adjustment process can be implemented through source-target vertex pair matching, vertex position adjustment based on energy parameters, and vertex position updates. For source-target vertex pair matching, a deep learning model can match vertex pairs between the source mesh and the target mesh based on depth features and UV maps. Vertex position adjustment based on energy parameters can be performed based on optimized energy values. For example, smoothness energy can be adjusted to reduce the difference between adjacent vertices and maintain a smooth surface, while data energy can be adjusted so that the source and target vertices are located as close as possible. Vertex position updates can be performed according to the optimized values ​​calculated for each energy parameter. In this process, the vertices of the target mesh are adjusted based on information from the source mesh (specifically normal maps), which can improve structural consistency and visual quality.

[0187] Ultimately, energy-based optimization can be utilized to improve alignment between the source mesh and the target mesh, and to restore occluded or missing parts in the target mesh. The vertices of the target mesh are adjusted using values ​​calculated from each energy item, and a deep learning model can guarantee accurate vertex positions by referencing the source mesh's normal map and depth features.

[0188] This deep learning-based correction adjusts the vertex positions of the target mesh by referencing the normal maps of the source mesh and utilizes depth features to improve the accuracy of the alignment. Additionally, it guarantees consistency and smoothness between source and target vertices based on energy parameters, and enables consistent alignment even in boundary regions. Furthermore, through optimized vertex position adjustment, occluded or missing parts of the target mesh can be naturally restored, and structural consistency matching the source mesh can be maintained.

[0190] FIG. 10 is intended to provide a general and simplified description of a suitable computing environment in which a system may be implemented. Referring to FIG. 10, a computing device is illustrated as an example of a system (100).

[0191] The computing device may include at least a processing unit (1003) and a system memory (1001).

[0192] The processing unit (1003) may include, for example, a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller unit (MCU), an application processor (AP), an electronic control unit (ECU), and / or at least one other electronic device capable of performing various calculations and control processing. These devices may be implemented, for example, by using one or more semiconductor chips, circuits, or related components alone or in combination. The computing device may include a plurality of processing units that cooperate when executing a program.

[0193] Depending on the exact configuration and type of the computing device, the system memory (1001) may be volatile (e.g., RAM), non-volatile (e.g., ROM, flash memory, etc.), or a combination thereof. The system memory (1001) includes a suitable operating system (1002) for controlling the operation of the platform, which may be, for example, the WINDOWS operating system from Microsoft. The system memory (1001) may also include one or more software applications, such as program modules, applications, etc.

[0194] The computing device may include additional data storage devices (1004), such as magnetic disks, optical disks, or tapes. These additional storage devices may be removable storage and / or fixed storage. Computer-readable storage media may include volatile and non-volatile, removable and fixed media implemented by any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. System memory (1001) and storage (1004) are all merely examples of computer-readable storage media. Computer-readable storage media may include, but are not limited to, RAM, ROM, EERPOM, flash memory or other memory techniques, CD-ROM, DVD or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that stores desired information and can be accessed by the computing device (1000).

[0195] Input devices (1005) of a computing device may include, for example, a keyboard, a mouse, a pen, a voice input device, a touch input device, and comparable input devices. Output devices (1006) may include, for example, a display, a speaker, a printer, and other types of output devices. Since these devices are widely known in the art, a detailed description is omitted.

[0196] The computing device may include a communication device (1007) that allows the device to communicate with other devices through a network in a distributed computing environment, such as a wired / wireless network, a satellite link, a cellular link, a local area network, and a comparable mechanism. The communication device (1007) is one example of a communication medium, and the communication medium may contain computer-readable instructions, data structures, program modules, or other data. For example, the communication medium includes, but is not limited to, wired media such as a wired network or direct wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media.

[0198] Experiment Results and Analysis

[0199] Dynamic 3D Human Mesh Dataset

[0200] A dataset for capturing dynamic 3D human motion was generated to validate the system, i.e., the framework, according to one embodiment of the present invention. This dataset was collected by shooting video at 60 frames per second (fps) using a system of 30 synchronized RGB cameras; each sequence consists of a length of approximately 5 to 10 seconds and provides a high frame rate and resolution suitable for capturing high-speed deformation motions. The collected RGB images were reconstructed into 3D scans of each subject through a process, which provides important surface geometric details for evaluating the accuracy of non-rigid registration. The dataset includes 12 subjects, each performing up to 10 unique motion types. This dataset provides a rich reference point for testing the adaptability of the non-rigid registration method under various conditions, including diverse movements.

[0201] Evaluation Metric

[0202] To quantitatively evaluate the accuracy of our method, we measure the Root Mean Square Error (RMSE) between the modified vertex and the target vertex. This is defined as shown in Equation 8 below.

[0203] [Mathematical Formula 8]

[0204]

[0205] Here is the deformed vertex position, represents the target vertex location, and |V| represents the total number of vertices. This metric allows for a comprehensive evaluation of matching accuracy. The RMSE value is measured at the model's original scale, and unless otherwise noted, the unit is expressed in meters.

[0206] Comparison with State-of-the-Art (SOTA) Methods

[0207] To verify the performance of the topology-consistent restoration framework, this study conducted comparative experiments with the approaches of N-ICP, RPTS, and SVR-10. These methods can solve optimization problems involving alignment terms, normalization terms, and in some methods, rigid body terms.

[0208] In our evaluation, we use mean RMSE and median RMSE as key metrics to assess the matching accuracy and consistency across the reconstructed surface.

[0209] Figure 11 is a diagram illustrating examples of experimental results for each method on a dataset, and Figure 12 is a table illustrating quantitative values ​​in this dataset.

[0210] Referring to the results in Fig. 12, it can be seen that the method according to the present invention has the highest performance and achieved an improvement in alignment accuracy of 3.5% and 14.7% compared to the method showing the second highest performance. In particular, in dynamic and fast scenes (S3, S9), the method according to the present invention recorded superior accuracy compared to other methods.

[0211] Figure 11 visualizes the qualitative benchmark and error map. Referring to the results, it can be seen that the results according to the present invention achieve a smooth transition and preserve details in high curvature and occluded regions where artifacts occur or alignment precision is lacking in other methods. These improvements are due to the use of MVS features as depth cues and normal maps for detail restoration, enabling consistent alignment while maintaining the structural integrity of the mesh.

[0213] Effectiveness of Our Method

[0214] The hierarchical approach according to the present invention can reduce cumulative errors in complex deformations by utilizing a skeletal structure to preferentially process the alignment of high-motion regions and distal regions.

[0215] Figure 13 is a diagram comparing the results of applying and not applying hierarchical joint-based alignment to evaluate the effect of the hierarchical structure.

[0216] Referring to FIG. 13, when the hierarchical structure is activated, it can be seen that the method according to the present invention achieves more precise alignment in extremity regions such as hands and significantly reduces residual inaccuracy.

[0217] Figure 14 shows a table illustrating the advantages of hierarchical joint-based alignment.

[0218] Referring to the table in Figure 14, an improvement of approximately 11.24% was observed compared to the case where the hierarchy was not applied, which can be seen as having the greatest impact on the matching results. In addition, consistently higher performance was recorded even when MVS features were included, confirming that MVS features are effective in capturing details and maintaining structural consistency.

[0220] Effect of UV Normal Map Diffusion

[0221] Figure 15 is a diagram visualizing the matching results depending on whether normal map diffusion is used, in order to emphasize the effect of phase consistency restoration using UV normal map diffusion.

[0222] Referring to the results in Fig. 15, it is shown that when using UV normal maps containing holes, details are lost and the result appears as a smooth mesh, whereas when using complete UV normal maps, the details are accurately preserved. The use of normal maps resulted in a performance improvement of about 1%, which is the smallest contributing factor but indicates that it is still an important approach for restoring details.

[0224] Methods according to various embodiments of the present application may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiments, or may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiments, and vice versa.

[0225] Those skilled in the art related to the embodiments of the present invention will understand that they may be implemented in modified forms without departing from the essential characteristics of the description. Therefore, the disclosed methods should be considered in an illustrative rather than a restrictive sense. The scope of the invention is defined by the claims, not by the detailed description of the invention, and all variations within the scope of the claims should be interpreted as being included within the scope of the invention. Explanation of the symbols

[0227] 100 : Non-rigid registration system for dynamic 3D mesh implementation 110: Joint-based registration model 120: Normal map-based correction model

Claims

Claim 1 A non-rigid alignment method for dynamic 3D mesh implementation, comprising: a step of performing skeleton-based hierarchical alignment between a source mesh and a target mesh based on a skeleton tree structure; a step of generating a UV map for the source mesh and generating a normal map based on the UV map; and a step of performing correction on the target mesh based on the normal map using a deep learning-based correction network. Claim 2 A non-rigid alignment method for dynamic 3D mesh implementation according to claim 1, wherein the step of performing skeleton-based hierarchical alignment between a source mesh and a target mesh based on the skeleton tree structure comprises: a step of identifying major joint points based on the skeleton for the source mesh and the target mesh; a step of starting alignment with respect to a center for the major joint points; and a step of performing alignment while extending from the center to the ends of the major joint points. Claim 3 A non-rigid alignment method for implementing a dynamic 3D mesh, wherein, in claim 1, the step of generating a UV map for the source mesh and generating a normal map based on the UV map comprises: a step of unfolding the surface of the 3D model of the source mesh into a 2D space; and a step of assigning 2D UV coordinates to each vertex unfolded into the 2D space. Claim 4 A non-rigid alignment method for dynamic 3D mesh implementation, wherein, in claim 3, the step of generating a UV map for the source mesh and generating a normal map based on the UV map further comprises the step of mapping surface information of the 3D model to the normal map based on assigned 2D UV coordinates. Claim 5 A non-rigid alignment method for dynamic 3D mesh implementation, wherein, in claim 4, the step of generating a UV map for the source mesh and generating a normal map based on the UV map further comprises the step of setting a hidden area of ​​the target mesh by referring to the normal map information of the source mesh. Claim 6 A non-rigid matching method for dynamic 3D mesh implementation according to claim 1, wherein the step of performing correction on the target mesh based on the normal map using the deep learning-based correction network comprises: a step of performing initial vertex matching based on the UV maps of the source mesh and the target mesh; a step of calculating an energy item to adjust the position of each vertex; and a step of adjusting the vertex position according to the energy item. Claim 7 In Clause 6, the above energy item is calculated through the following mathematical formula, and Here, A non-rigid matching method for dynamic 3D mesh implementation, each representing rigid body energy, smooth energy, data energy, boundary energy, and MVS feature energy. Claim 8 In paragraph 4, the vertex information for the i-th pixel pi of the normal map is a mathematical formula It is determined according to, and here, v i1 ,v i2 ,v i3 is pixel p in the UV map i These are the vertices forming the triangle surrounding , and α i ,β i ,γ i is p i A non-rigid alignment method for implementing a dynamic 3D mesh, which is the barycentric coordinates for the three vertices. Claim 9 In paragraph 8, the above-mentioned center coordinate α i ,β i ,γ i is, mathematical formula Satisfying, and here, is the area of ​​the triangle formed by vertices v1, v2, and v3, and , and are respectively p i A non-rigid matching method for implementing a dynamic 3D mesh, which is the area of ​​a sub-triangle including Claim 10 In Clause 9, the above p i Normal vector N(p) for i ) is, mathematical formula A non-rigid matching method for dynamic 3D mesh implementation satisfying [condition]. Claim 11 A non-rigid registration system for dynamic 3D mesh implementation, comprising: a joint-based registration model that performs skeleton-based hierarchical registration between a source mesh and a target mesh based on a skeleton tree structure; and a normal map-based correction model that generates a UV map for the source mesh, generates a normal map based on the UV map, and performs correction for the target mesh based on the normal map using a deep learning-based correction network. Claim 12 In claim 11, the joint-based alignment model identifies major joint points based on a skeleton for the source mesh and the target mesh, starts alignment with respect to the major joint points with respect to a center, and then performs alignment while extending from the center to the ends of the major joint points, a non-rigid alignment system for dynamic 3D mesh implementation. Claim 13 In claim 11, the normal map-based correction model is a non-rigid matching system for dynamic 3D mesh implementation that unfolds the surface of the 3D model of the source mesh into a 2D space and assigns 2D UV coordinates to each vertex unfolded into the 2D space. Claim 14 In claim 13, the normal map-based correction model is a non-rigid matching system for dynamic 3D mesh implementation that maps surface information of the 3D model to a normal map based on assigned 2D UV coordinates. Claim 15 In claim 14, the normal map-based correction model sets the occluded area of ​​the target mesh by referring to the normal map information of the source mesh, a non-rigid matching system for dynamic 3D mesh implementation. Claim 16 In claim 11, the normal map-based correction model performs initial vertex matching based on UV maps of the source mesh and the target mesh, calculates an energy item to adjust the position of each vertex, and then adjusts the vertex position according to the energy item, a non-rigid matching system for dynamic 3D mesh implementation. Claim 17 In Clause 16, the above energy item is calculated through the following mathematical formula, and Here, A non-rigid matching system for dynamic 3D mesh implementation, representing rigid body energy, smooth energy, data energy, boundary energy, and MVS feature energy, respectively. Claim 18 In Clause 17, the vertex information for the i-th pixel pi of the above normal map is a mathematical formula It is determined according to, and here, v i1 ,v i2 ,v i3 is pixel p in the UV map i These are the vertices forming the triangle surrounding , and α i ,β i ,γ i is p i A non-rigid matching system for dynamic 3D mesh implementation, which is the barycentric coordinates for the three vertices of. Claim 19 In paragraph 18, the above-mentioned center coordinate α i ,β i ,γ i is satisfied with the mathematical equation, and here, is the area of ​​the triangle formed by vertices v1, v2, and v3, and , and are respectively p i A non-rigid matching system for dynamic 3D mesh implementation, which is the area of ​​a sub-triangle including Claim 20 In Paragraph 19, the above p i Normal vector N(p) for i ) is, mathematical formula A non-rigid matching system for dynamic 3D mesh implementation satisfying

Images