Three-dimensional face modeling method based on arc-shaped speckle structured light RGB-D

Abstract

The three-dimensional face modeling method based on the arc-shaped speckle structured light RGB-D mainly comprises: three RGB-D depth cameras arranged along the arc-shaped face contour are used to collect continuous multiple frames of RGB-D data in sequence; based on the external parameters between the cameras, the point clouds of each view are transformed to a unified coordinate system to form an initial point cloud; an external parameter constraint cross-view RGB feature registration network is used to extract the view-invariant face features and establish the cross-view feature correspondence, the pixel correspondence is improved to three-dimensional corresponding points, the external parameter constraint and the three-dimensional corresponding points are fused to complete the coarse registration; a stable region constraint three-view rigid and non-rigid joint fine registration network is used to output a rigid pose correction amount, a local deformation correction amount and an uncertainty; the correction amount is applied and weighted fusion is carried out according to the uncertainty, and a three-dimensional face model is output after hole filling, smoothing and texture attachment. The present disclosure can inhibit the mismatching in the symmetrical region, and improve the modeling accuracy and integrity.

Description

Technical Field

[0001] This disclosure relates to the fields of 3D vision, neural network image feature registration and 3D digital modeling technology, specifically to a 3D face modeling method based on arc speckle structured light RGB-D. Background Technology

[0002] RGB-D depth cameras can acquire both color and depth images. Speckle structured light depth cameras obtain depth by projecting a speckle-coded pattern onto the target surface and decoding the speckle deformation, making them suitable for acquiring facial surface information. Since a single camera cannot simultaneously cover the left and right sides of the face, ears, and the outer contour of the face, multiple cameras can be positioned along the arc of the facial contour to obtain multi-view RGB-D data with overlapping areas.

[0003] When multiple active speckle structured light RGB-D depth cameras project simultaneously, speckle crosstalk may occur. Sequential acquisition also results in data from different viewpoints being acquired at different times. Slight rotations, translations, blinks, mouth movements, or facial changes of the subject can cause static extrinsic parameters to fail to directly provide point cloud alignment results that meet the modeling accuracy.

[0004] Existing multi-view face registration methods typically use facial landmarks, manual features, or camera pose for initial registration, followed by refinement using point cloud registration methods. However, when the face is nearly symmetrical, there are significant perspective changes in the profile, local textures are weak, or there are changes in expression or occlusion, incorrect correspondences can easily occur, thus affecting the accuracy of point cloud registration.

[0005] General learning-based image matching, if searching across the entire image, is prone to mismatches in symmetrical facial regions or between similar skin textures. Therefore, it is necessary to utilize known camera geometry to limit the search range of RGB learning features and evaluate the matching reliability, so that the network can obtain stable cross-view correspondences that can be used for 3D registration. Summary of the Invention

[0006] To at least solve one of the above-mentioned technical problems, the present invention provides the following technical solution.

[0007] A 3D face modeling method based on arc-shaped speckle structured light RGB-D includes the following steps: Step S1: Acquire continuous multi-frame RGB-D data from the left, front, and right viewpoints, collected sequentially by three RGB-D depth cameras set along the arc of the face contour. Based on the intrinsic parameters of each camera and the extrinsic parameters between cameras, transform the point cloud data of each viewpoint to a unified coordinate system with the front viewpoint as the reference to form the initial point cloud of each viewpoint. Step S2: Using an extrinsic-constrained cross-view RGB feature registration network, extract view-invariant face features from the RGB images of the left, front, and right views, and establish cross-view feature correspondences within the candidate search region defined by the inter-camera extrinsic parameters, and output the matching confidence of each feature. Step S3: The corresponding pixels in the cross-view feature correspondence are elevated to three-dimensional corresponding points based on their corresponding depth values. The constraints of the camera extrinsic parameters and the constraints of the three-dimensional corresponding points are fused to perform a weighted robust pose solution, and the coarse registration transformations of the left view and the right view relative to the front view are obtained. Step S4: Using a three-view rigid and non-rigid joint fine registration network, based on the left-view, front-view, and right-view point clouds after the coarse registration transformation, output the rigid pose correction, local deformation correction, and corresponding uncertainty. Step S5: Apply the rigid pose correction amount and the local deformation correction amount to correct the point cloud of each viewpoint. Perform weighted fusion of the point cloud of each viewpoint according to the uncertainty of the correspondence relationship. Then, after hole filling, feature preservation smoothing and RGB texture attachment, output a three-dimensional face model.

[0008] Furthermore, in step S1, the time-division sequential acquisition includes performing continuous multi-frame acquisition in the order of the first frontal frame group, the left frame group, the second frontal frame group, the right frame group, and the third frontal frame group, and only the speckle projection of the corresponding camera is enabled during any active depth acquisition period.

[0009] Further, in step S2, the extrinsic parameter-constrained cross-view RGB feature registration network includes a parameter-sharing multi-scale feature extraction backbone, a viewpoint conditional modulation layer, a stable region weighting layer, and a cross-view feature matching layer; the multi-scale feature extraction backbone extracts learning feature maps of different scales from the RGB images of the three viewpoints; the viewpoint conditional modulation layer conditionsally modulates the learning feature maps according to the camera imaging ray direction, camera relative pose encoding, and normalized pixel position of each pixel; the stable region weighting layer predicts the probability that each pixel belongs to a stable facial region and weights the learning feature maps.

[0010] Further, in step S2, the candidate search region is constructed as follows: when the pixel to be matched in the source view has a valid depth value, the pixel to be matched is back-projected into three-dimensional space according to the intrinsic parameters of the source camera, and then the predicted position of the pixel to be matched in the target view is obtained according to the extrinsic parameters from the source camera to the target camera. The anisotropic two-dimensional projection uncertainty is formed by combining the extrinsic parameter calibration error, depth measurement uncertainty and projection Jacobian, and the candidate search region is established accordingly. When the pixel to be matched does not have a reliable depth value, an epipolar line is formed in the target image according to the intrinsic and extrinsic parameters of the camera, and a strip-shaped region of a predetermined width near the epipolar line is taken as the candidate search region.

[0011] Further, in step S2, establishing cross-view feature correspondence includes: for the pixel to be matched in the source view, calculating feature similarity for each candidate position in the candidate search area, using geometric deviation as an additional penalty term, and obtaining the conditional probability that each candidate position belongs to the true corresponding position after normalization; selecting the candidate position with the highest probability as the discrete corresponding position or weighting and summing the candidate positions according to the conditional probability to obtain the sub-pixel corresponding position; calculating the matching uncertainty according to the matching probability distribution in the candidate search area, wherein the matching uncertainty adopts the form of matching entropy, and the matching uncertainty increases when multiple candidate positions have close matching probabilities, and decreases when the true corresponding position forms a single-peak high-probability distribution.

[0012] Furthermore, in step S2, establishing cross-view feature correspondence also includes: for the corresponding position obtained through forward matching, performing reverse matching with the corresponding position as the source position of the target view; if the reverse matching result cannot return to the vicinity of the original pixel to be matched, then the corresponding position is eliminated or its weight is reduced; for features that have a common visible area in the left view, front view, and right view, the transitivity or cyclic consistency between the left view to the front view, the front view to the right view, and the reverse mapping is checked to eliminate ambiguous correspondences; for side face features that only have a common visible area with the front view, a bidirectional consistency check with the front view as a common reference is adopted.

[0013] Furthermore, the stable facial regions predicted by the stable region weighted layer include the forehead, bridge of the nose, cheekbone, and temporal region. The stability probability output by the stable region weighted layer ranges from 0 to 1 and is used as a weight for feature matching and matching confidence calculation, so that the contribution of relatively stable facial regions to coarse registration is greater than that of easily deformable regions.

[0014] Furthermore, in step S3, the objective function of the weighted robust pose solution includes an RGB learning feature matching constraint term and an extrinsic parameter prior constraint term. The RGB learning feature matching constraint term is weighted based on the comprehensive matching confidence of the corresponding three-dimensional points, and the extrinsic parameter prior constraint term is weighted based on the extrinsic parameter quality matrix determined by the calibration reprojection error and the repeated calibration error. By adjusting the relative weights of the two constraints through a balancing factor, the three-dimensional residual corresponding to the high-confidence learning is minimized while the pose correction amount deviates from the reliable extrinsic parameter.

[0015] Further, in step S4, the three-view rigid and non-rigid joint fine registration network includes a geometric feature encoding module, a cross-view feature fusion module with the frontal view as a common reference, a rigid pose correction branch, a local deformation correction branch, and an uncertainty branch; the rigid pose correction branch predicts the rigid pose correction amount based on the features of the stable face region; the local deformation correction branch predicts the local deformation correction amount based on the features of the easily deformable face region; and the uncertainty branch outputs the point-by-point correspondence uncertainty.

[0016] A 3D face modeling system based on arc-shaped speckle structured light RGB-D includes: The data acquisition module is configured to acquire continuous multi-frame RGB-D data from the left, front, and right viewpoints, which are collected sequentially by three RGB-D depth cameras set along the arc of the face contour. Based on the intrinsic parameters of each camera and the extrinsic parameters between cameras, the point cloud data of each viewpoint is transformed into a unified coordinate system with the front viewpoint as the reference, forming the initial point cloud of each viewpoint. The feature matching module is configured to use a cross-view RGB feature registration network constrained by extrinsic parameters to extract view-invariant face features from the RGB images of the left view, front view and right view, and establish cross-view feature correspondences within the candidate search area defined by the extrinsic parameters between the cameras, and output the matching confidence of each feature. The coarse registration module is configured to elevate the corresponding pixels in the cross-view feature correspondence to three-dimensional corresponding points based on their corresponding depth values, and perform weighted robust pose solving by fusing the constraints of the inter-camera extrinsic parameters and the constraints of the three-dimensional corresponding points to obtain the coarse registration transformations of the left and right viewpoints relative to the frontal viewpoint. The fine registration module is configured to use a three-view rigid and non-rigid joint fine registration network to output rigid pose correction, local deformation correction and corresponding uncertainty based on the left, front and right view point clouds after coarse registration transformation. The fusion output module is configured to apply the rigid pose correction amount and the local deformation correction amount to correct the point clouds of each viewpoint, perform weighted fusion of the point clouds of each viewpoint according to the uncertainty of the correspondence relationship, and then output a three-dimensional face model through hole filling, feature preservation smoothing and RGB texture attachment.

[0017] Furthermore, the data acquisition module is configured to perform continuous multi-frame acquisition in the order of the first frontal frame group, the left frame group, the second frontal frame group, the right frame group, and the third frontal frame group, and only the speckle projection of the corresponding camera is enabled during any active depth acquisition period.

[0018] Furthermore, the extrinsic-constrained cross-view RGB feature registration network in the feature matching module includes a parameter-shared multi-scale feature extraction backbone, a viewpoint conditional modulation layer, a stable region weighting layer, and a cross-view feature matching layer; the multi-scale feature extraction backbone is configured to extract learning feature maps of different scales from RGB images of three viewpoints; the viewpoint conditional modulation layer is configured to conditionally modulate the learning feature maps according to the camera imaging ray direction, camera relative pose encoding, and normalized pixel position of each pixel; the stable region weighting layer is configured to predict the probability that each pixel belongs to a stable facial region and weight the learning feature maps.

[0019] Furthermore, the feature matching module is configured to: when the pixel to be matched in the source view has a valid depth value, back-project the pixel to be matched to a three-dimensional space based on the intrinsic parameters of the source camera, and then obtain the predicted position of the pixel to be matched in the target view based on the extrinsic parameters from the source camera to the target camera. Combine the extrinsic parameter calibration error, depth measurement uncertainty, and projection Jacobian to form an anisotropic two-dimensional projection uncertainty, and establish the candidate search region accordingly; when the pixel to be matched does not have a reliable depth value, form an epipolar line in the target image based on the camera's intrinsic and extrinsic parameters, and use a strip-shaped region of a predetermined width near the epipolar line as the candidate search region.

[0020] Furthermore, the feature matching module is configured to: for a pixel to be matched in the source view, calculate feature similarity at each candidate position within the candidate search area, use geometric deviation as an additional penalty term, and after normalization, obtain the conditional probability that each candidate position belongs to the true corresponding position; select the candidate position with the highest probability as the discrete corresponding position, or sum the candidate positions according to the conditional probability to obtain the sub-pixel corresponding position; calculate the matching uncertainty based on the matching probability distribution within the candidate search area, wherein the matching uncertainty is in the form of matching entropy, and the matching uncertainty increases when multiple candidate positions have close matching probabilities, and decreases when the true corresponding positions form a single-peak high-probability distribution.

[0021] Furthermore, the feature matching module is also configured to: for the corresponding position obtained through forward matching, perform reverse matching with the corresponding position as the source position of the target viewpoint; if the reverse matching result cannot return to the vicinity of the original pixel to be matched, then remove the corresponding position or reduce its weight; for features that have a common visible area in the left viewpoint, front viewpoint, and right viewpoint, check the transitivity or cyclic consistency between the left viewpoint to the front viewpoint, the front viewpoint to the right viewpoint, and the reverse mapping to remove ambiguous correspondences; for side face features that only have a common visible area with the front viewpoint, use bidirectional consistency verification with the front viewpoint as a common reference.

[0022] Furthermore, the stable facial regions predicted by the stable region weighted layer include the forehead, bridge of the nose, cheekbone, and temporal region. The stable probability output by the stable region weighted layer ranges from 0 to 1 and is used as weights for feature matching and matching confidence calculation.

[0023] Furthermore, the objective function of the coarse registration module for the weighted robust pose solution includes an RGB learning feature matching constraint term and an extrinsic parameter prior constraint term. The RGB learning feature matching constraint term is weighted based on the comprehensive matching confidence of the corresponding three-dimensional points, and the extrinsic parameter prior constraint term is weighted based on the extrinsic parameter quality matrix determined by the calibration reprojection error and the repeated calibration error. The relative weights of the two constraints are adjusted by a balancing factor.

[0024] Furthermore, the three-view rigid and non-rigid joint fine registration network in the fine registration module includes a geometric feature encoding module, a cross-view feature fusion module with the frontal view as a common reference, a rigid pose correction branch, a local deformation correction branch, and an uncertainty branch; the rigid pose correction branch predicts the rigid pose correction amount based on the features of the stable face region; the local deformation correction branch predicts the local deformation correction amount based on the features of the easily deformable face region; and the uncertainty branch outputs the point-by-point correspondence uncertainty.

[0025] Beneficial effects: This disclosure reduces symmetric mismatches and invalid candidates in full-image search by utilizing camera geometric priors to limit the search region and RGB learned features to determine true correspondences. It also improves the reliability of coarse registration by stabilizing region weights and matching uncertainties. The time-series acquisition scheme effectively avoids optical crosstalk when multiple speckle structured light cameras project simultaneously. A rigid-non-rigid joint fine registration network jointly estimates the rigid motion of the head and the non-rigid deformation of facial expressions in the same model, and outputs point-by-point uncertainties for subsequent weighted fusion, further improving the accuracy and completeness of multi-view 3D face modeling. This disclosure features low cost, high speed, high modeling accuracy, and a more complete model for 3D face modeling. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the arc-shaped speckle structured light RGB-D depth camera array and system module disclosed herein; Figure 2 This is a flowchart illustrating the overall process of the 3D face modeling method disclosed herein. Figure 3 This is a schematic diagram of coarse registration that integrates external camera features for RGB learning matching in this disclosure. Figure 4 This is a diagram of the three-view rigid and non-rigid joint fine registration network structure based on stable region constraints disclosed in this publication. Figure 5 This diagram illustrates the continued retention of registration confidence level and source information in subsequent processing. Figure 6 This is a schematic diagram of the timing of continuous multi-frame acquisition by the three speckle structured light RGB-D depth cameras disclosed in this invention; Figures 7(A) and 7(B) are diagrams of the cross-view RGB feature registration network structure under extrinsic constraints of this disclosure. Detailed Implementation

[0027] The following is in conjunction with the appendix Figure 1 Figure 7(B) and the examples provide further detailed description of this disclosure. It is understood that the specific examples described herein are for illustrative purposes only and are not intended to limit the scope of this disclosure. It should also be noted that, for ease of description, only the parts relevant to this disclosure are shown in the accompanying drawings.

[0028] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0029] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0030] Unless otherwise stated, the exemplary implementations or embodiments shown are to be understood as providing exemplary features of various details that provide ways in which the technical concepts of this disclosure can be implemented in practice. Therefore, unless otherwise stated, the features of various implementations or embodiments may be additionally combined, separated, interchanged and / or rearranged without departing from the technical concepts of this disclosure.

[0031] It should be noted that, unless otherwise explicitly specified and limited, the terms "connected" and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a physical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this disclosure based on the specific circumstances.

[0032] The terminology used herein is for the purpose of describing particular embodiments and is not intended to be limiting. As used herein, unless the context clearly indicates otherwise, the singular forms "a" and "the" are intended to include the plural forms as well. Furthermore, when the terms "contains" and "or include" and variations thereof are used in this specification, it indicates the presence of the stated features, integrals, steps, operations, parts, components, and / or groups thereof, but does not preclude the presence or addition of one or more other features, integrals, steps, operations, parts, components, and / or groups thereof. It should also be noted that, as used herein, the terms "substantially," "approximately," and other similar terms are used as approximate terms rather than as terms of degree, so that they are used to explain the inherent deviations of measured values, calculated values, and / or provided values ​​that will be recognized by those skilled in the art.

[0033] In one embodiment, the 3D face modeling method of this disclosure integrates data acquisition, two-level neural registration, and model generation into the following seven main steps: Step S101: Obtain the camera intrinsic parameters, RGB and depth registration parameters, and camera extrinsic parameters of the RGB-D depth camera.

[0034] Step S102: Sequentially acquire multiple consecutive frames of RGB-D data, complete quality evaluation, time compensation and frame group fusion, and generate a three-view point cloud.

[0035] Step S103: Perform extrinsic parameter initial transformation, extract cross-view stable face features from the RGB neural network, establish cross-view RGB learning feature correspondence in the candidate region constrained by extrinsic parameters, transform the corresponding pixels into three-dimensional corresponding points through depth upscaling, and fuse camera extrinsic parameters with RGB learning feature correspondence to complete coarse registration.

[0036] Step S104: Construct a coarse registration prior, and use a three-view rigid and non-rigid joint fine registration network to predict the rigid residuals, local deformation residuals and corresponding uncertainty.

[0037] Step S105: Apply rigid residuals and local deformation residuals, and generate a unified face point cloud based on the uncertainty credible fusion.

[0038] Step S106: Identify and fill in the measurement holes, generate a 3D face mesh and smooth it.

[0039] Step S107: Attach RGB textures and output a unified face point cloud, gray mesh model, and textured 3D face model.

[0040] In another embodiment, such as Figure 1 As shown, three RGB-D cameras—one on the left, one on the front, and one on the right—are mounted on an adjustable curved bracket, with adjacent fields of view overlapping on the face surface. After adjusting the array, the camera intrinsic parameters, the RGB-depth registration relationship, and the extrinsic parameters of the left and right cameras relative to the front camera are reacquired.

[0041] In another embodiment, such as Figure 6 As shown, the frontal frame group C0, left frame group L, second frontal frame group C1, right frame group R, and third frontal frame group C2 are acquired sequentially. During any active depth acquisition period, only the speckle projection of the corresponding camera is enabled to reduce mutual interference. Each frame group selects qualified frames from continuous RGB-D data and fuses them to form a viewpoint cloud. For asynchronous acquisition, the face motion during acquisition can be estimated using adjacent frontal frame groups, compensating the left and right viewpoint data to a common reference time. Frame selection, point cloud fusion, and time compensation can be implemented using existing methods.

[0042] In another embodiment, such as Figure 3 As shown, firstly, the extrinsic parameters of the left and right cameras relative to the front camera are used to transform the point clouds of the left and right views to an initial common coordinate system based on the front camera. The extrinsic parameters provide a deterministic initial pose, and are combined with calibration reprojection error, repeated calibration error, and depth measurement uncertainty to form an extrinsic parameter calibration quality index. Subsequently, the left, front, and right RGB images, along with the projection constraints generated by the inter-camera extrinsic parameters, are input into a cross-view RGB feature registration network constrained by the extrinsic parameters. This network includes a parameter-sharing three-view feature extraction branch, a view conditional modulation layer, a stable region weighting layer, and a cross-view feature matching layer. The feature extraction branch outputs multi-scale feature maps ranging from local texture details to facial structural semantics; the view conditional modulation layer encodes camera light direction, relative pose of the extrinsic parameters, and pixel position into the features, enabling the same facial region to form comparable view-invariant face features under different viewpoints.

[0043] To map the same face surface position from different viewpoints to a comparable feature space, the left, front, and right RGB branches share backbone network parameters. As one implementation, the shared backbone consists of a four-level feature pyramid composed of local convolutional coding units and hierarchical context coding units. Lower-level features are used to preserve edges, local textures, and fine structures, while higher-level features describe the semantics of a larger facial structure. Features at each scale are fused from top to bottom to form a unified feature map for matching. The number of channels at each level can be set to 64, 128, 256, and 256, and the dimension of the fused matching features can be set to 128. The number of scales, channels, and coding unit types can be adjusted according to computational resources.

[0044] , s=1,...,S in, Represents the RGB image of the viewpoint v. These represent the left-side view, the frontal view, and the right-side view, respectively. Indicates perspective The RGB image is processed by the multi-scale feature extraction backbone with parameter sharing to obtain the learned features at the s-th scale, where s represents the total number of feature scales. This represents the feature mapping corresponding to the multi-scale feature extraction backbone at the s-th scale; This represents the network parameters used by all three viewpoints. Parameter sharing ensures that the images from all three viewpoints are encoded under the same feature transformation rules, thus avoiding additional cross-viewpoint feature shifts introduced by differences in branch parameters.

[0045] Viewpoint conditional modulation is added to the shared features. For viewpoint... pixel position in Construct the viewpoint condition vector The viewpoint condition vector includes at least the camera imaging ray direction corresponding to the pixel, the camera pose code relative to the common reference camera, and the normalized pixel position. The viewpoint condition coding network is based on... Generate feature scaling and feature offset, and perform conditional modulation on shared features to enable the network to explicitly distinguish between appearance differences caused by changes in viewing direction and feature differences caused by differences in the actual surface position of the face.

[0046] in, Indicates perspective Pixel position in; Represents pixels The corresponding viewpoint condition vector; Represents pixels The corresponding camera imaging ray direction; Indicates perspective The pose encoding of the camera relative to a common reference camera; Represents pixels Normalized pixel position; This represents the s-th scale learned feature before conditional modulation. This represents the learned features after conditional modulation; and The scaling and offset terms are generated by the viewpoint conditional coding network. For feature normalization operation, This represents channel-wise multiplication. Viewpoint conditional modulation can also be achieved through conditional normalization, channel attention, or spatial attention.

[0047] The stable region weighting layer simultaneously predicts the probability that each pixel belongs to a stable facial region, prioritizing the preservation of forehead, bridge of nose, cheekbone, and temporal features, while reducing the feature weights of eyelids, lips, nostrils, hair highlights, shadow transitions, and occluded areas. The cross-view feature matching layer does not perform an unconstrained search across the entire image; instead, it computes learned feature relevance within candidate regions defined by extrinsic projection windows or epipolar bands. Ambiguous correspondences are eliminated through bidirectional matching consistency (left-positive and positive-left, right-positive and positive-right) and left-positive-right transitivity or cyclic consistency within commonly visible regions. The layer then outputs the corresponding pixel, matching confidence, and matching uncertainty.

[0048] The stable region weighted layer outputs stable probabilities from a lightweight pixel prediction head. Its value ranges from 0 to 1. This probability does not directly delete easily deformable regions completely, but is used as a weight for feature matching and subsequent confidence calculation, so that relatively stable regions such as the forehead, bridge of the nose, and cheekbones contribute more to coarse registration under facial expression changes, while retaining effective features with discriminative power in regions such as the periorbital area and periorbital area.

[0049] For the pixels to be matched in the source view When its depth is valid, first determine the source camera's intrinsic parameters. The predicted position in the target camera is obtained by back-projecting to three-dimensional space and then using the extrinsic parameters from the source camera to the target camera. The two-dimensional projection uncertainty is formed by combining the external parameter calibration error, depth measurement uncertainty, and projection Jacobian. Based on this, candidate search regions for anisotropy are established.

[0050] in, This represents the pixel to be matched in the source view; Indicates the position of candidate pixels in the target's viewpoint; This indicates the predicted target viewpoint position obtained based on the source camera intrinsic parameters, source pixel depth, and extrinsic parameters from the source camera to the target camera. This represents the pixel obtained by combining extrinsic parameter calibration error, depth measurement uncertainty, and projection Jacobian. The two-dimensional projection uncertainty matrix in the target's viewpoint. Represents pixels Candidate search area from the target's perspective This is a parameter controlling the search range. When the source pixels do not have reliable depth, epipolar lines in the target image are formed based on camera intrinsic and extrinsic parameters, and a band-shaped region of a certain width near the epipolar lines is used as a candidate search region. Therefore, the network does not need to perform unconstrained matching across the entire target image.

[0051] Cross-view feature matching layer Local cross-attention or relevance calculation is performed internally. Source pixel features are used as query features, and target pixel features within the candidate region are used as key and value features; simultaneously, the geometric deviation of the candidate pixel relative to the predicted position of the extrinsic parameters is used as an additional penalty term. After normalization, the conditional probability that each candidate position belongs to the true corresponding position is obtained. : in, and These are the source and target features after viewpoint modulation and stable region weighting, respectively. and For learnable projection matrices, For the feature dimension, For temperature parameters, This represents the geometric deviation of the candidate position from the predicted position by the extrinsic parameters. The candidate position with the highest probability can be selected as the discrete correspondence, or the candidate positions can be assigned according to... Weighted summation is performed to obtain sub-pixel correspondences. : To characterize whether there are multiple solutions or ambiguities in the learned matching, the matching uncertainty is calculated based on the matching probability distribution within the candidate regions. The uncertainty increases when multiple candidate positions have similar matching probabilities; the uncertainty decreases when the true corresponding positions form a clear unimodal high-probability distribution.

[0052] in, To match the uncertainty in the form of entropy, This is the confidence score for normalized feature matching. This indicates the number of candidate locations in the candidate search area. For cases where the number of candidate points varies significantly, the uncertainty of the network's individual prediction can be jointly calibrated with the uncertainty of the probability distribution mentioned above.

[0053] For the corresponding obtained through forward matching , and then Perform a reverse match on the source location. If the reverse result cannot be returned... If the corresponding feature is located nearby, it is considered to have high ambiguity. Correspondences with bidirectional consistency errors exceeding the preset tolerance range are discarded or have their weight reduced. For features that share a common visible area in the left, front, and right views, the transitive or cyclic consistency between the left view and the front view, the front view and the right view, and the available reverse mappings can also be checked. For side profile features that only share a visible area with the front view, bidirectional consistency with the front view as a common reference is used, without requiring a direct pixel correspondence between the left and right views.

[0054] For each pair of valid learned feature correspondences, the corresponding depth pixels are found using the registration parameters from the RGB image to the depth image. When the depth is valid and the local depth variance is below a threshold, the corresponding pixels are back-projected into 3D points, thus obtaining cross-view 3D correspondence points. The RGB learned feature matching confidence score is formed based on learned feature similarity, bidirectional consistency, three-view transfer or cyclic consistency, RGB reprojection residuals, local depth validity, and stable region probability. The final matching confidence score is jointly determined by feature matching confidence score, stable region probability, bidirectional consistency, available three-view transfer or cyclic consistency, extrinsic geometric residuals, and depth validity. Correspondences with low confidence or high uncertainty can be directly discarded before 3D upscaling, or they can be retained and given lower weights in subsequent coarse pose solving.

[0055] A weighted robust pose solution is performed by fusing camera extrinsic parameters with 3D correspondences of RGB learned features. When the extrinsic parameter calibration quality is high but the learned feature correspondences are scarce, the extrinsic parameter constraint weights are increased; when the number of learned feature correspondences is sufficient, the matching uncertainty is low, and the reprojection residual is small, the RGB learned feature matching constraint weights are increased. The solution provides coarse registration transformations of the left and right views relative to the frontal view. Furthermore, based on the rotation and translation differences, corresponding spatial distribution, and depth uncertainty between the extrinsic parameter transformation and the RGB learned feature matching estimation transformation, a coarse registration error covariance is formed and used as the explicit input for fine registration of the point cloud neural network.

[0056] For the first [unclear] which was established and depth enhancement was completed through RGB learning features Group of three-dimensional corresponding points ,by This indicates its overall matching confidence level, in order to This indicates that the initial transformation given by the camera extrinsic parameters can be solved by finding the correction transformation in the vicinity of the extrinsic parameter prior. This minimizes the three-dimensional residual corresponding to high-confidence learning while suppressing excessive deviations of the correction from the reliable extrinsic parameters. Its objective function can be expressed as: in, For robust loss function, This indicates the quality of the external parameters determined by the calibration reprojection error and the repeatability calibration error. This solution is used to balance the learned RGB correspondence with the prior extrinsic parameters. It is only used to transform the reliable two-dimensional correspondence output by the network into a three-dimensional coarse registration result.

[0057] In another embodiment, such as Figure 4 As shown, after coarse registration, the three-view point clouds, RGB learned features, and matching confidence are input into a joint rigid and non-rigid fine registration network. The network can be composed of a geometric feature encoding module, a cross-view feature fusion module using the frontal view as a common reference, a rigid pose correction branch, a local deformation correction branch, and an uncertainty branch. Stable regions are mainly used to estimate overall rigidity correction, while easily deformable regions such as the periorbital area and mouth area are used to estimate local deformation; the coarse registration error is used to limit the candidate search range. The network outputs rigidity correction, local deformation, and registration uncertainty, and uses these to correct and fuse the three-view point clouds. Point cloud feature encoding, rigid pose solving, local deformation propagation, and weighted fusion can be implemented using existing methods.

[0058] In another embodiment, the training dataset for training the cross-view RGB feature registration network with extrinsic constraints and the three-view rigid and non-rigid joint fine registration network is constructed as follows: After completing the calibration of the intrinsic parameters, RGB and depth registration parameters, and inter-camera extrinsic parameters of the three RGB-D depth cameras, multiple sets of three-view RGB images and corresponding depth data were obtained using the same left-view, front-view, and right-view acquisition methods as in the 3D face modeling process. Face point clouds for each view were then generated based on the depth data. The corresponding camera intrinsic parameters and inter-camera extrinsic parameters were saved for each set of training data.

[0059] For the cross-view RGB feature registration network constrained by extrinsic parameters, based on the depth data of each viewpoint, camera intrinsic parameters, and inter-camera extrinsic parameters, the effective depth pixels in one viewpoint are back-projected into 3D space and then projected into another viewpoint to obtain the initial pixel correspondence. This correspondence is then confirmed or corrected by the annotators, and corresponding pixels that meet the depth consistency and visibility conditions are used as positive samples. For non-corresponding pixels within the candidate search region, approximately symmetrical areas of the face, areas with similar textures, or occluded areas are selected as negative samples or hard negative samples. The forehead, bridge of the nose, cheekbone, and temporal region are labeled by the annotators according to the preset face region division to generate stable region labels. This forms the training data for the feature registration network, which consists of three-view RGB images, camera intrinsic and extrinsic parameters, cross-view correspondence, and stable region labels.

[0060] For the three-view rigid and non-rigid joint fine registration network, the annotators selected point clouds with good registration quality from the collected three-view RGB-D data as training samples, and manually annotated relatively stable areas such as the forehead, bridge of the nose, cheekbone and temporal region, as well as easily deformable areas such as the periorbital region, cheek and jaw.

[0061] Based on the training samples, pre-defined rotation, translation, and local deformation perturbations are applied to the point clouds from either the left or right viewpoints to simulate the rigid and non-rigid registration errors that still exist after coarse registration. Since the applied perturbation parameters are known, the corresponding rotation and translation corrections are used as supervision labels for rigid pose correction, and the point position difference before and after local deformation is used as supervision labels for local deformation correction. Simultaneously, low-reliability regions are constructed by adding corresponding errors, depth gaps, or local occlusions to form supervision labels for correspondence uncertainty. This yields the dataset used to train the three-view rigid and non-rigid joint fine registration network.

[0062] By constructing multiple sets of training samples in the manner described above, a training dataset corresponding to the input and output of the cross-view RGB feature registration network and the three-view rigid and non-rigid joint fine registration network with the aforementioned extrinsic constraints can be obtained.

[0063] The training loss of the RGB feature registration network can include cross-view feature contrast loss, bidirectional matching consistency loss, three-view transitive or cyclic consistency loss, stable region weighting loss, RGB reprojection loss, and matching uncertainty calibration loss. Positive samples are the corresponding features of the same face surface location from different viewpoints, while negative samples include symmetrical but different facial regions, occluded regions, and similar texture regions within the extrinsic projection window to reduce mismatches caused by left-right symmetry and repetitive textures.

[0064] During training, the construction of feature comparison samples focuses on the matching ambiguities unique to faces. In addition to random negative samples, regions that are approximately symmetrical on the left and right sides of the face but have different 3D positions, regions with similar textures but not corresponding within the candidate search region, and occluded regions are considered as difficult negative samples; effective projections of the same face surface position from different viewpoints are considered as positive samples. Through the constraint of difficult negative samples, the network reduces the probability of mismatching symmetrical regions based solely on local appearance similarity.

[0065] The overall training objective of the RGB feature registration network can be expressed as follows, with the weights of each loss term set according to the training stage and data quality: in, For cross-view facial feature comparison loss, For bidirectional matching consistency loss, For the loss of three-view propagation or cyclic consistency within the common visible area, To stabilize regional forecasts and weighted losses, For geometric reprojection consistency loss, To match the uncertainty calibration loss, the RGB feature extraction and matching module can be trained first during training, and then fine-tuned by combining coarse pose solving; the specific training order does not affect the basic process of the above network in the inference stage to establish cross-view RGB learning feature correspondences based on extrinsic parameters constraining candidate regions.

[0066] To train the point cloud fine registration network, rotation, translation, and corresponding error perturbations can be applied to the real extrinsic parameters and RGB learned feature matching estimation transformations. Noise can be added to the matching confidence and depth values, and local deformations can be applied to the periorbital, periorbital, cheek, and mandible areas. The fine registration training loss includes at least the following: rigid pose correction loss for stable regions, local deformation loss for easily deformable regions, point-to-surface geometric consistency loss, RGB learned feature reprojection consistency loss, three-view closed-loop consistency loss, local deformation smoothing loss, zero displacement constraint loss for stable regions, and uncertainty calibration loss. The two-level network can be trained in stages and then jointly fine-tuned, or it can be trained end-to-end.

[0067] In another embodiment, such as Figure 5 As shown, during fusion, information such as source viewpoint, depth quality, and confidence level of coarse or fine registration are preserved, and the fusion weight of each viewpoint is determined accordingly. The fused point cloud is generated using conventional hole processing, triangular mesh generation, feature preservation smoothing, and multi-view RGB texture attachment methods. During processing, measurement holes are distinguished from real openings such as nostrils, mouth slits, and ear canal recesses to avoid erroneous closure.

[0068] The system outputs a unified face point cloud, a gray mesh model, and a textured 3D face model, and can simultaneously output point-level confidence and source information. When there are insufficient effective frames, insufficient effective RGB correspondences, or network uncertainty exceeds a threshold, the corresponding viewpoint weights are reduced; if the model is insufficient to cover the necessary modeling area, model generation is terminated and a quality failure flag is output.

[0069] In one embodiment, a three-dimensional face modeling method is provided, including the following steps: Acquire RGB-D data from multiple perspectives using multiple RGB-D depth cameras; The point cloud data from each viewpoint is transformed to a unified coordinate system based on the intrinsic parameters of each camera and the extrinsic parameters between cameras. A cross-view feature registration network is used to extract view-invariant features from RGB images of each viewpoint, and cross-view feature correspondence is established using camera geometric constraints. The corresponding pixels in the cross-view feature correspondence are upgraded to three-dimensional corresponding points, and coarse registration is completed based on the three-dimensional corresponding points. Rigid-non-rigid joint registration is performed on the point clouds of each view after coarse registration to obtain the corrected point clouds of each view. By fusing the corrected point clouds from each viewpoint, a 3D face model is generated.

[0070] Acquiring RGB-D data from multiple RGB-D depth cameras from various perspectives aims to obtain coverage information of the face from different angles, thus addressing the problem of insufficient coverage from a single camera. Transforming the point clouds from each perspective to a unified coordinate system based on inter-camera extrinsics provides geometric priors for subsequent feature matching, reducing the search space. Utilizing a cross-view feature registration network to extract view-invariant features and establishing feature correspondences using camera geometric constraints combines the learning ability of neural networks with known geometric constraints to suppress mismatches in symmetrical regions. Upgrading corresponding pixels to 3D corresponding points and completing coarse registration transforms 2D feature matching into alignment constraints in 3D space to achieve initial pose estimation. Rigid-non-rigid joint registration is performed on the coarsely registered point clouds to handle the overall rigid motion of the head and the non-rigid deformation of facial expressions, thereby improving registration accuracy.

[0071] In another embodiment, the plurality of RGB-D depth cameras are three, arranged in an arc along the contour of the face, forming a left camera, a front camera, and a right camera respectively, with the fields of view of adjacent cameras overlapping on the face surface.

[0072] The three cameras are arranged in an arc along the facial contour to achieve complete coverage of the left and right sides of the face, ears, and the outer contour of the face with a smaller number of cameras; the overlapping of adjacent fields of view is to establish the basis for feature correspondence between different viewpoints.

[0073] In another embodiment, the cross-view feature registration network includes a parameter-sharing multi-scale feature extraction backbone, which extracts learning feature maps of different scales from RGB images of each viewpoint.

[0074] The multi-scale feature extraction backbone extracts learning feature maps at different scales from RGB images from various perspectives in order to simultaneously preserve local texture details and facial structure semantic information in order to establish a more robust cross-view feature correspondence.

[0075] In another embodiment, the cross-view feature registration network further includes a view condition modulation layer, which conditionally modulates the learned feature map based on the camera imaging ray direction of each pixel, the camera relative pose encoding, and the normalized pixel position.

[0076] The viewpoint conditional modulation layer incorporates the camera imaging ray direction and relative pose encoding into the features. This is to enable the same facial region to form comparable viewpoint-invariant facial features under different viewpoints, thereby eliminating the influence of differences in shooting angles.

[0077] In another embodiment, the cross-view feature registration network further includes a stable region weighting layer, which predicts the probability that each pixel belongs to a stable facial region and weights the learned feature map.

[0078] The stable region weighting layer predicts the probability that each pixel belongs to a stable facial region such as the forehead, bridge of the nose, and cheekbone. This is to make relatively stable regions contribute more to coarse registration under facial expression changes, thereby reducing the interference of facial expression changes.

[0079] In another embodiment, the step of establishing cross-view feature correspondence using camera geometric constraints includes: for a pixel to be matched in the source view, determining a candidate search region in the target view based on the source camera intrinsic parameters and the extrinsic parameters from the source camera to the target camera, and performing feature matching only within the candidate search region.

[0080] Using camera geometric constraints to determine candidate search regions is to avoid performing unconstrained matching across the entire target image, thereby reducing false matching of symmetrical and similar texture regions.

[0081] In another embodiment, establishing cross-view feature correspondence further includes: performing reverse matching on the correspondence obtained through forward matching to verify bidirectional consistency, and performing transitive consistency or cyclic consistency checks on the feature correspondences between the left, front, and right views to eliminate ambiguous correspondences, and outputting the matching confidence of each feature correspondence.

[0082] Bidirectional consistency check and transitive consistency check are used to filter out ambiguous matches generated in symmetrical or similar texture regions in order to output feature correspondences with quantified confidence.

[0083] In another embodiment, the coarse registration based on the three-dimensional corresponding points includes: using the matching confidence of the three-dimensional corresponding points as weights, performing weighted robust pose solving by fusing camera extrinsic constraints and RGB learning feature matching constraints; increasing the weight of extrinsic constraints when the extrinsic calibration quality is high and the learning feature correspondences are scarce; and increasing the weight of learning feature matching constraints when the number of learning feature correspondences is sufficient and the matching confidence is high.

[0084] The weighted robust pose solution dynamically adjusts the relative weights of the two types of constraints based on the extrinsic calibration quality and the learned feature matching quality, in order to achieve adaptive fusion between the calibration extrinsic parameters and the learned feature matching to obtain the optimal coarse registration result.

[0085] In another embodiment, the rigid-non-rigid joint registration includes: using a fine registration network to predict the rigid pose correction, the local deformation correction, and the uncertainty of the corresponding relationship based on the point clouds of each viewpoint after coarse registration; and applying the rigid pose correction and the local deformation correction to correct the point clouds of each viewpoint.

[0086] The fine registration network simultaneously predicts rigid pose corrections and local deformation corrections in order to jointly estimate the rigid head motion and non-rigid facial expression deformations in the same model to more accurately align point clouds from different viewpoints.

[0087] In another embodiment, the rigid-non-rigid joint registration further includes: the fine registration network includes a rigid pose correction branch and a local deformation correction branch, wherein the rigid pose correction branch predicts the rigid pose correction amount based on features of the stable face region, and the local deformation correction branch predicts the local deformation correction amount based on features of the easily deformable face region.

[0088] The rigid pose correction branch estimates the overall rigidity correction based on the features of the stable region, while the local deformation correction branch estimates the local deformation based on the features of the easily deformable region. This is to decouple the rigid motion and the local deformation to obtain more accurate estimates respectively.

[0089] In another embodiment, fusing the corrected point clouds from each viewpoint includes: weighting and fusing the point clouds from each viewpoint according to the uncertainty of the correspondence relationship, wherein points with lower uncertainty have higher fusion weights and points with higher uncertainty have lower fusion weights.

[0090] Weighted fusion based on the uncertainty of the correspondence is intended to automatically reduce the weight of ambiguous matching regions during the fusion process to prevent the spread of errors.

[0091] In another embodiment, generating a 3D face model includes: processing holes in the fused point cloud, generating triangular meshes, preserving feature smoothing, and attaching multi-view RGB textures.

[0092] Hole handling, triangular mesh generation, feature preservation smoothing, and multi-view RGB texture attachment are all designed to generate 3D face models with complete geometry and realistic textures.

[0093] In one embodiment, a three-dimensional face modeling method based on arc-shaped speckle structured light RGB-D is provided, comprising the following steps: Step S1: Acquire continuous multi-frame RGB-D data from the left, front, and right viewpoints, collected sequentially by three RGB-D depth cameras set along the arc of the face contour. Based on the intrinsic parameters of each camera and the extrinsic parameters between cameras, transform the point cloud data of each viewpoint to a unified coordinate system with the front viewpoint as the reference to form the initial point cloud of each viewpoint. Step S2: Using an extrinsic-constrained cross-view RGB feature registration network, extract view-invariant face features from the RGB images of the left, front, and right views, and establish cross-view feature correspondences within the candidate search region defined by the inter-camera extrinsic parameters, and output the matching confidence of each feature. Step S3: The corresponding pixels in the cross-view feature correspondence are elevated to three-dimensional corresponding points based on their corresponding depth values. The constraints of the camera extrinsic parameters and the constraints of the three-dimensional corresponding points are fused to perform a weighted robust pose solution, and the coarse registration transformations of the left view and the right view relative to the front view are obtained. Step S4: Using a three-view rigid and non-rigid joint fine registration network, based on the left-view, front-view, and right-view point clouds after the coarse registration transformation, output the rigid pose correction, local deformation correction, and corresponding uncertainty. Step S5: Apply the rigid pose correction amount and the local deformation correction amount to correct the point cloud of each viewpoint. Perform weighted fusion of the point cloud of each viewpoint according to the uncertainty of the correspondence relationship. Then, after hole filling, feature preservation smoothing and RGB texture attachment, output a three-dimensional face model.

[0094] Step S1 acquires multi-view RGB-D data collected in a time-division manner using an arc array and transforms it to a unified coordinate system based on extrinsic parameters. This is to obtain multi-view face data with geometric priors to establish initial alignment. Step S2 uses a cross-view RGB feature registration network with extrinsic constraints to extract view-invariant features and establish feature correspondences within candidate regions limited by extrinsic parameters. This can suppress mismatches in symmetrical regions by combining geometric constraints with learned features. Step S3 elevates the feature correspondences to three-dimensional corresponding points and fuses extrinsic parameters with feature constraints for weighted pose solving. This is to transform two-dimensional feature matching into three-dimensional alignment constraints to obtain coarse registration transformation. Step S4 uses a fine registration network to output rigid pose correction, local deformation correction, and uncertainty. The technical role is to handle rigid body motion and local deformation respectively and quantify matching reliability. Step S5 applies corrections and performs weighted fusion based on uncertainty, followed by post-processing to output a three-dimensional face model. This is to prioritize the retention of high-confidence data to generate a complete model through a confidence-driven fusion strategy.

[0095] In another embodiment, in step S1, the time-division sequential acquisition includes performing continuous multi-frame acquisition in the order of the first frontal frame group, the left frame group, the second frontal frame group, the right frame group, and the third frontal frame group, and only the speckle projection of the corresponding camera is enabled during any active depth acquisition period.

[0096] The continuous multi-frame acquisition is performed in the order of the first frontal frame group, the left frame group, the second frontal frame group, the right frame group, and the third frontal frame group. During any active depth acquisition period, only the speckle projection of the corresponding camera is enabled. This is to avoid optical crosstalk caused by the simultaneous projection of multiple speckle structured light cameras. At the same time, multiple sets of frontal frame groups are used to estimate the face movement during the acquisition period to compensate for the time deviation of asynchronous acquisition.

[0097] In another embodiment, in step S2, the extrinsic-constrained cross-view RGB feature registration network includes a parameter-shared multi-scale feature extraction backbone, a viewpoint conditional modulation layer, a stable region weighting layer, and a cross-view feature matching layer; the multi-scale feature extraction backbone extracts learning feature maps at different scales from the RGB images of the three viewpoints; the viewpoint conditional modulation layer conditionsally modulates the learning feature maps according to the camera imaging ray direction, camera relative pose encoding, and normalized pixel position of each pixel; the stable region weighting layer predicts the probability that each pixel belongs to a stable facial region and weights the learning feature maps.

[0098] The parameter-shared multi-scale feature extraction backbone extracts learned feature maps at different scales from RGB images from three perspectives. This is to ensure that the images from the three perspectives are encoded under the same feature transformation rules to avoid additional cross-view feature shifts introduced by differences in branch parameters. The viewpoint conditional modulation layer conditionsally modulates the features based on the camera imaging ray direction and relative pose encoding. This is to eliminate differences in shooting angles and form viewpoint-invariant facial features. The stable region weighting layer predicts the probability that each pixel belongs to a stable facial region and weights the features. This is to give stable regions such as the forehead and bridge of the nose higher weights in feature matching to reduce interference from changes in appearance.

[0099] In another embodiment, in step S2, the candidate search region is constructed as follows: when the pixel to be matched in the source view has a valid depth value, the pixel to be matched is back-projected into three-dimensional space based on the intrinsic parameters of the source camera, and then the predicted position of the pixel to be matched in the target view is obtained based on the extrinsic parameters from the source camera to the target camera. The candidate search region is established by combining the extrinsic parameter calibration error, depth measurement uncertainty, and projection Jacobian to form anisotropic two-dimensional projection uncertainty. When the pixel to be matched does not have a reliable depth value, an epipolar line is formed in the target image based on the camera's intrinsic and extrinsic parameters, and a strip-shaped region of a predetermined width near the epipolar line is taken as the candidate search region.

[0100] When the source pixel has a valid depth value, an anisotropic candidate search region is established by combining extrinsic parameter projection with uncertainty in order to perform a fine search near the high-precision predicted position and narrow the matching range. When the source pixel does not have a reliable depth value, a strip-shaped candidate region is established by using epipolar constraints in order to limit the search range by using geometric constraints even when there is no reliable depth information, so as to avoid full-map search.

[0101] In another embodiment, step S2, establishing cross-view feature correspondence, includes: for the pixel to be matched in the source view, calculating feature similarity for each candidate position within the candidate search area, using geometric deviation as an additional penalty term, and obtaining the conditional probability that each candidate position belongs to the true corresponding position after normalization; selecting the candidate position with the highest probability as the discrete corresponding position or weighting and summing the candidate positions according to the conditional probability to obtain the sub-pixel corresponding position; calculating the matching uncertainty based on the matching probability distribution within the candidate search area, wherein the matching uncertainty is in the form of matching entropy, and the matching uncertainty increases when multiple candidate positions have close matching probabilities, and decreases when the true corresponding positions form a single-peak high-probability distribution.

[0102] The purpose of calculating the matching probability by using geometric deviation as an additional penalty term within the candidate search region is to add geometric consistency constraints on top of feature similarity to exclude geometrically unreasonable candidate positions; the purpose of calculating the uncertainty in the form of matching entropy based on the matching probability distribution is to quantify whether there are multiple solutions or ambiguities in the matching so as to serve as the basis for subsequent weighted fusion.

[0103] In another embodiment, step S2, establishing cross-view feature correspondence further includes: for the corresponding position obtained by forward matching, performing reverse matching with the corresponding position as the source position of the target view; if the reverse matching result cannot return to the vicinity of the original pixel to be matched, then the corresponding position is eliminated or its weight is reduced; for features that have a common visible area in the left view, front view, and right view, the transitivity or cyclic consistency between the left view to the front view, the front view to the right view, and the reverse mapping is checked to eliminate ambiguous correspondences; for side face features that only have a common visible area with the front view, a bidirectional consistency check with the front view as a common reference is adopted.

[0104] Bidirectional consistency checks are used to filter out ambiguous correspondences where the forward and reverse matches are inconsistent, in order to reduce mismatches caused by symmetrical regions or similar textures; three-view transitive or cyclic consistency checks are used to filter out correspondences that cannot be transitively closed between the three views, in order to further eliminate ambiguous matches by utilizing multi-view redundant information.

[0105] In another embodiment, the stable facial regions predicted by the stable region weighting layer include the forehead, bridge of the nose, cheekbone, and temporal region. The stable probability output by the stable region weighting layer ranges from 0 to 1 and is used as a weight for feature matching and matching confidence calculation, so that the contribution of relatively stable facial regions to coarse registration is greater than that of easily deformable regions.

[0106] The stability region weighted layer predicts that the stability probability of the forehead, bridge of the nose, cheekbone and temple is between 0 and 1. This is to give higher weight to relatively stable areas of the face under facial expression changes in feature matching, while retaining effective features that still have discriminative power in areas such as the periorbital region.

[0107] In another embodiment, in step S3, the objective function of the weighted robust pose solution includes an RGB learning feature matching constraint term and an extrinsic parameter prior constraint term. The RGB learning feature matching constraint term is weighted based on the comprehensive matching confidence of the corresponding 3D points, and the extrinsic parameter prior constraint term is weighted based on the extrinsic parameter quality matrix determined by the calibration reprojection error and the repeated calibration error. The relative weights of the two constraints are adjusted by a balancing factor to minimize the 3D residual corresponding to high confidence learning while suppressing the pose correction from deviating from the reliable extrinsic parameter.

[0108] The objective function of the weighted robust pose solution includes both RGB learning feature matching constraints and extrinsic prior constraints. By adjusting the relative weights of the two constraints through a balancing factor, adaptive fusion can be achieved between calibration extrinsic and learning feature matching to obtain the optimal balance between the two information sources.

[0109] In another embodiment, in step S4, the three-view rigid and non-rigid joint fine registration network includes a geometric feature encoding module, a cross-view feature fusion module with the frontal view as a common reference, a rigid pose correction branch, a local deformation correction branch, and an uncertainty branch; the rigid pose correction branch predicts the rigid pose correction amount based on features of the stable face region; the local deformation correction branch predicts the local deformation correction amount based on features of the easily deformable face region; and the uncertainty branch outputs the point-by-point correspondence uncertainty.

[0110] The rigid pose correction branch predicts the rigid correction amount based on the features of stable regions, while the local deformation correction branch predicts the local deformation correction amount based on the features of easily deformable regions. This is to estimate the rigid motion and local deformation separately to avoid mutual interference. The uncertainty branch outputs point-by-point uncertainty in order to quantify the reliability of registration at each point to guide subsequent weighted fusion.

[0111] In another embodiment, a three-dimensional face modeling system based on arc-shaped speckle structured light RGB-D is provided, comprising: The data acquisition module is configured to acquire continuous multi-frame RGB-D data from the left, front, and right viewpoints, which are collected sequentially by three RGB-D depth cameras set along the arc of the face contour. Based on the intrinsic parameters of each camera and the extrinsic parameters between cameras, the point cloud data of each viewpoint is transformed into a unified coordinate system with the front viewpoint as the reference, forming the initial point cloud of each viewpoint. The feature matching module is configured to use a cross-view RGB feature registration network constrained by extrinsic parameters to extract view-invariant face features from the RGB images of the left view, front view and right view, and establish cross-view feature correspondences within the candidate search area defined by the extrinsic parameters between the cameras, and output the matching confidence of each feature. The coarse registration module is configured to elevate the corresponding pixels in the cross-view feature correspondence to three-dimensional corresponding points based on their corresponding depth values, and perform weighted robust pose solving by fusing the constraints of the inter-camera extrinsic parameters and the constraints of the three-dimensional corresponding points to obtain the coarse registration transformations of the left and right viewpoints relative to the frontal viewpoint. The fine registration module is configured to use a three-view rigid and non-rigid joint fine registration network to output rigid pose correction, local deformation correction and corresponding uncertainty based on the left, front and right view point clouds after coarse registration transformation. The fusion output module is configured to apply the rigid pose correction amount and the local deformation correction amount to correct the point clouds of each viewpoint, perform weighted fusion of the point clouds of each viewpoint according to the uncertainty of the correspondence relationship, and then output a three-dimensional face model through hole filling, feature preservation smoothing and RGB texture attachment.

[0112] The data acquisition module acquires multi-view RGB-D data collected in a time-division manner from an arc array and transforms it to a unified coordinate system based on extrinsic parameters to obtain multi-view face data with geometric priors. The feature matching module uses a cross-view RGB feature registration network with extrinsic constraints to extract view-invariant features and establish feature correspondences within candidate regions limited by extrinsic parameters. The technical role is to suppress mismatches in symmetrical regions by combining geometric constraints with learned features. The coarse registration module elevates the feature correspondences to three-dimensional corresponding points and fuses constraints for weighted pose solving to obtain the coarse registration transformation. The fine registration module uses a neural network to output rigidity correction, local deformation correction, and uncertainty to handle rigid body motion and local deformation respectively and quantify matching reliability. The fusion output module applies corrections and performs weighted fusion based on uncertainty, followed by post-processing to output the model. This generates a complete model through a confidence-driven fusion strategy.

[0113] In another embodiment, the data acquisition module is configured to perform continuous multi-frame acquisition in the order of the first frontal frame group, the left frame group, the second frontal frame group, the right frame group, and the third frontal frame group, with only the speckle projection of the corresponding camera enabled during any active depth acquisition period.

[0114] The data acquisition module performs time-division acquisition in five-frame group sequence, and only the speckle projection of the corresponding camera is enabled during any active depth acquisition period. This is to avoid optical crosstalk caused by multiple speckle structured light cameras projecting simultaneously.

[0115] In another embodiment, the extrinsic-constrained cross-view RGB feature registration network in the feature matching module includes a parameter-shared multi-scale feature extraction backbone, a viewpoint conditional modulation layer, a stable region weighting layer, and a cross-view feature matching layer; the multi-scale feature extraction backbone is configured to extract learning feature maps of different scales from RGB images of three viewpoints; the viewpoint conditional modulation layer is configured to conditionally modulate the learning feature maps based on the camera imaging ray direction, camera relative pose encoding, and normalized pixel position of each pixel; the stable region weighting layer is configured to predict the probability that each pixel belongs to a stable facial region and weight the learning feature maps.

[0116] The multi-scale feature extraction backbone with shared parameters extracts learning feature maps at different scales to encode images from three perspectives under the same feature transformation rules. The viewpoint conditional modulation layer modulates the features based on the camera imaging ray direction and relative pose encoding to eliminate differences in shooting angles and form viewpoint-invariant face features. The stable region weighting layer predicts the probability of stable regions and weights the features to give stable regions higher weight in feature matching.

[0117] In another embodiment, the feature matching module is configured to: when the pixel to be matched in the source view has a valid depth value, back-project the pixel to be matched to a three-dimensional space based on the intrinsic parameters of the source camera, and then obtain the predicted position of the pixel to be matched in the target view based on the extrinsic parameters from the source camera to the target camera. Combine the extrinsic parameter calibration error, depth measurement uncertainty, and projection Jacobian to form anisotropic two-dimensional projection uncertainty, and establish the candidate search region accordingly; when the pixel to be matched does not have a reliable depth value, form an epipolar line in the target image based on the camera's intrinsic and extrinsic parameters, and use a strip-shaped region of a predetermined width near the epipolar line as the candidate search region.

[0118] The feature matching module adaptively constructs candidate search regions using 3D projection constraints or epipolar constraints based on the validity of depth data, so that geometric constraints can be used to limit the matching search range regardless of the availability of reliable depth information.

[0119] In another embodiment, the feature matching module is configured to: for a pixel to be matched in the source view, calculate feature similarity at each candidate position within the candidate search area, use geometric deviation as an additional penalty term, and after normalization, obtain the conditional probability that each candidate position belongs to the true corresponding position; select the candidate position with the highest probability as the discrete corresponding position or sum the candidate positions according to the conditional probability to obtain the sub-pixel corresponding position; calculate the matching uncertainty based on the matching probability distribution within the candidate search area, wherein the matching uncertainty is in the form of matching entropy, and the matching uncertainty increases when multiple candidate positions have close matching probabilities, and decreases when the true corresponding positions form a single-peak high-probability distribution.

[0120] The feature matching module calculates the matching probability by combining feature similarity and geometric deviation within the candidate region in order to consider both appearance similarity and geometric consistency in feature matching. The uncertainty in the form of matching entropy is calculated based on the matching probability distribution in order to quantify whether there are multiple solutions or ambiguities in the matching.

[0121] In another embodiment, the feature matching module is further configured to: for corresponding positions obtained through forward matching, perform reverse matching with the corresponding position as the source position of the target viewpoint; if the reverse matching result cannot return to the vicinity of the original pixel to be matched, then eliminate the corresponding position or reduce its weight; for features that have a common visible area in the left viewpoint, front viewpoint, and right viewpoint, check the transitivity or cyclic consistency between the left viewpoint to the front viewpoint, the front viewpoint to the right viewpoint, and the reverse mapping to eliminate ambiguous correspondences; for side face features that only have a common visible area with the front viewpoint, use bidirectional consistency verification with the front viewpoint as a common reference.

[0122] The feature matching module filters out ambiguous correspondences through bidirectional consistency checks and transitive consistency checks in order to reduce mismatches in symmetrical and similar texture regions by utilizing redundant information from multiple perspectives.

[0123] In another embodiment, the stable facial regions predicted by the stable region weighting layer include the forehead, bridge of the nose, cheekbone, and temporal region. The stable probability output by the stable region weighting layer ranges from 0 to 1 and is used as weights for feature matching and matching confidence calculation.

[0124] The stability region weighted layer predicts the stability probability of regions such as the forehead, bridge of the nose, cheekbone, and temple with values ​​ranging from 0 to 1. This is to give stable regions higher weight in feature matching so as to reduce the interference of appearance changes on matching.

[0125] In another embodiment, the objective function of the coarse registration module for the weighted robust pose solution includes an RGB learning feature matching constraint term and an extrinsic parameter prior constraint term. The RGB learning feature matching constraint term is weighted based on the comprehensive matching confidence of the three-dimensional corresponding points, and the extrinsic parameter prior constraint term is weighted based on the extrinsic parameter quality matrix determined by the calibration reprojection error and the repeated calibration error. The relative weights of the two constraints are adjusted by a balancing factor.

[0126] The objective function of the coarse registration module includes both learning feature matching constraints and extrinsic parameter prior constraints, and adjusts the weights through a balancing factor in order to achieve adaptive fusion between calibration extrinsic parameters and learning feature matching.

[0127] In another embodiment, the three-view rigid and non-rigid joint fine registration network in the fine registration module includes a geometric feature encoding module, a cross-view feature fusion module with the frontal view as a common reference, a rigid pose correction branch, a local deformation correction branch, and an uncertainty branch; the rigid pose correction branch predicts the rigid pose correction amount based on features of the stable face region; the local deformation correction branch predicts the local deformation correction amount based on features of the easily deformable face region; and the uncertainty branch outputs the point-by-point correspondence uncertainty.

[0128] The rigid pose correction branch and the local deformation correction branch of the fine registration module predict their respective correction amounts based on the stable region and the easily deformable region, in order to decouple rigid motion and local deformation to obtain more accurate estimates respectively; the uncertainty branch outputs point-by-point uncertainty in order to quantify the reliability of registration at each point.

[0129] In another embodiment, a three-dimensional face modeling system based on arc-shaped speckle structured light RGB-D is provided, comprising: Arc-shaped support; Three RGB-D depth cameras are set in an arc shape along the contour of the face on the arc-shaped bracket, forming a left camera, a front camera and a right camera respectively. The fields of view of adjacent cameras have overlapping areas on the face surface. processor; A memory coupled to the processor, the memory storing computer program instructions; The processor implements the method described above when executing the computer program instructions in the memory.

[0130] The arc-shaped bracket is used to fix three RGB-D depth cameras and set them along the arc of the face contour in order to obtain face data from the left, front, and right perspectives simultaneously in a defined geometric layout; the three cameras constitute the left camera, the front camera, and the right camera respectively, and their adjacent fields of view overlap, in order to establish a spatial basis for feature correspondence between the three perspectives; the processor implements the above method when executing computer program instructions stored in memory, so as to enable the general-purpose hardware system to have the function of executing the method of this disclosure.

[0131] In another embodiment, the three RGB-D depth cameras are speckle structured light depth cameras, each of which has a built-in infrared speckle projector. The time-division sequential acquisition includes controlling the activation of each infrared speckle projector according to a predetermined timing sequence.

[0132] Each RGB-D depth camera has a built-in infrared speckle projector to project a speckle-coded pattern onto the face surface to obtain depth information through speckle deformation decoding. The activation of each projector is controlled according to a predetermined timing sequence to avoid optical crosstalk caused by multiple speckle projectors being activated simultaneously.

[0133] In another embodiment, the arc-shaped bracket is an adjustable arc-shaped bracket, and the mounting angle and position of the left camera and the right camera relative to the front camera can be adjusted, and the external parameters between the cameras can be reacquired after adjustment.

[0134] The curved support is adjustable to adapt to different face sizes and contours to optimize the field of view coverage; after adjustment, the inter-camera extrinsic parameters are reacquired to ensure that the accuracy of geometric constraints is not affected by the support adjustment.

[0135] In another embodiment, the processor includes a GPU or NPU, the memory stores network parameters of the extrinsic-constrained cross-view RGB feature registration network and the three-view rigid and non-rigid joint fine registration network, and the processor calls the GPU or NPU to perform inference operations of the network.

[0136] In another embodiment, a method for reconstructing the surface of a three-dimensional object is provided, comprising the following steps: Step S1: Acquire RGB-D data from multiple imaging devices at multiple different perspectives; Step S2: Define the candidate search region based on the geometric constraints between the imaging devices, and establish cross-view feature correspondence within the candidate search region; Step S3: Upgrade the corresponding pixels in the cross-view feature correspondence to correspondences in three-dimensional space, and complete the coarse registration of multi-view point clouds based on the correspondences; Step S4: Perform rigid-non-rigid joint registration on the point clouds of each view after coarse registration to obtain the corrected point clouds of each view. Step S5: Fuse the corrected point clouds from each viewpoint to generate the surface model of the three-dimensional object.

[0137] Step S1 acquires RGB-D data from multiple imaging devices from multiple perspectives to obtain coverage information of the 3D object from different angles; Step S2 limits the candidate search area and establishes feature correspondences based on the geometric constraints between imaging devices. The technical role is to use known geometric relationships to narrow the matching search range and reduce false matches; Step S3 improves the corresponding pixels into 3D correspondences and completes coarse registration to obtain the initial alignment result; Step S4 performs rigid-non-rigid joint registration on the coarsely registered point cloud to simultaneously handle rigid body motion and local non-rigid deformation to obtain more accurate alignment; Step S5 fuses the corrected point cloud to generate a surface model to output the complete 3D reconstruction result.

[0138] In another embodiment, the multiple imaging devices are RGB-D depth cameras, the multiple different viewpoints include a left viewpoint, a front viewpoint, and a right viewpoint, and the geometric constraints between the imaging devices include camera intrinsic parameters and camera extrinsic parameters.

[0139] In another embodiment, step S2 includes: extracting learning features from RGB images of each viewpoint using a cross-view feature registration network; performing viewpoint conditional modulation on the learning features based on the camera imaging ray direction and camera relative pose encoding of each pixel to obtain viewpoint-invariant features; and matching the viewpoint-invariant features within the candidate search area to establish the cross-view feature correspondence.

[0140] The purpose of using a cross-view feature registration network to extract learned features and perform view condition modulation is to obtain view-invariant features that are comparable across different viewpoints to reduce the impact of differences in shooting angles. Matching view-invariant features within the candidate region is to use geometric constraints to narrow the search range.

[0141] In another embodiment, step S2 further includes: performing reverse matching on the correspondences obtained through forward matching to verify bidirectional consistency, and performing transitive consistency or cyclic consistency checks on features with common visible areas among the three perspectives to eliminate ambiguous correspondences; and outputting the matching confidence of each feature.

[0142] The purpose of eliminating ambiguous correspondences through bidirectional consistency verification and transitive consistency verification is to use multi-perspective redundant information to filter out mismatches caused by symmetrical regions or similar textures and output feature correspondences with quantified credibility.

[0143] In another embodiment, step S4, the rigid-non-rigid joint registration includes: using a fine registration network to predict the rigid pose correction, the local deformation correction, and the uncertainty of the corresponding relationship based on the point clouds of each viewpoint after coarse registration; and applying the rigid pose correction and the local deformation correction to correct the point clouds of each viewpoint.

[0144] Using a precise registration network to simultaneously predict rigid pose corrections and local deformation corrections is to jointly estimate rigid body motion and local non-rigid deformation in the same model; applying two types of corrections to modify the point cloud is to simultaneously compensate for head rigid body displacement and facial surface changes.

[0145] In another embodiment, in step S5, the fusion includes weighted fusion of point clouds from each viewpoint based on the uncertainty of the correspondence relationship; the generation of the surface model of the three-dimensional object includes outputting a textured three-dimensional model after hole filling, feature preservation smoothing and RGB texture attachment.

[0146] Weighted fusion of point clouds from different perspectives based on uncertainty is to automatically reduce the weight of ambiguous matching regions during the fusion process; outputting a textured 3D model after hole filling, smoothing and texture attachment is to obtain a complete 3D reconstruction result with realistic texture.

[0147] In another embodiment, a three-dimensional object surface reconstruction system is provided, comprising: The data acquisition module is configured to acquire RGB-D data collected from multiple imaging devices from multiple different perspectives; The feature matching module is configured to define a candidate search region based on the geometric constraints between the imaging devices, and to establish a cross-view feature correspondence within the candidate search region; The coarse registration module is configured to elevate the corresponding pixels in the cross-view feature correspondence to a correspondence in three-dimensional space, and complete the coarse registration of the multi-view point cloud based on the correspondence. The fine registration module is configured to perform rigid-non-rigid joint registration on the point clouds of each view after coarse registration to obtain the corrected point clouds of each view. The fusion output module is configured to fuse the corrected point clouds from each viewpoint to generate the surface model of the three-dimensional object.

[0148] The data acquisition module acquires RGB-D data from multiple imaging devices at multiple perspectives to obtain coverage of the 3D object from different angles; the feature matching module limits candidate regions based on geometric constraints and establishes feature correspondences, the technical role of which is to reduce mismatches by utilizing geometric priors; the coarse registration module improves pixel correspondences to 3D correspondences and completes coarse registration to obtain initial alignment; the fine registration module performs rigid-non-rigid joint registration to handle rigid body motion and local deformation simultaneously; the fusion output module fuses point clouds to generate a surface model to output a complete 3D reconstruction result.

[0149] In another embodiment, the multiple imaging devices are RGB-D depth cameras, the multiple different viewpoints include a left viewpoint, a front viewpoint, and a right viewpoint, and the geometric constraints between the imaging devices include camera intrinsic parameters and camera extrinsic parameters.

[0150] The imaging device is an RGB-D depth camera, and the geometric constraints include intrinsic and extrinsic parameters, in order to provide geometric relationships between color images, depth images, and the camera to support cross-view feature matching.

[0151] In another embodiment, the feature matching module is configured to: extract learned features from RGB images of each viewpoint using a cross-view feature registration network; perform viewpoint conditional modulation on the learned features according to the camera imaging ray direction and camera relative pose encoding of each pixel to obtain viewpoint-invariant features; and match the viewpoint-invariant features within the candidate search area to establish the cross-view feature correspondence.

[0152] The feature matching module uses the network to extract learned features and modulate the viewpoint conditions in order to obtain viewpoint-invariant features and reduce the impact of differences in shooting angles.

[0153] In another embodiment, the feature matching module is further configured to: perform reverse matching on the correspondences obtained through forward matching to verify bidirectional consistency, and perform transitive consistency or cyclic consistency checks on features with common visible areas among the three views to eliminate ambiguous correspondences; and output the matching confidence of each feature.

[0154] The feature matching module eliminates ambiguous correspondences through bidirectional consistency and transitive consistency checks in order to reduce mismatches by utilizing redundant information from multiple perspectives.

[0155] In another embodiment, the fine registration module is configured to: use a fine registration network to predict the rigid pose correction, the local deformation correction, and the uncertainty of the corresponding relationship based on the point clouds of each viewpoint after coarse registration; and apply the rigid pose correction and the local deformation correction to correct the point clouds of each viewpoint.

[0156] The fine registration module uses a network to simultaneously predict rigidity correction and local deformation correction, in order to jointly estimate rigid body motion and local deformation in the same model.

[0157] In another embodiment, the fusion output module is configured to: perform weighted fusion of point clouds from each viewpoint based on the uncertainty of the correspondence; and output a textured 3D model after hole filling, feature preservation smoothing and RGB texture attachment.

[0158] The fusion output module performs weighted fusion based on uncertainty and outputs a textured model after post-processing in order to generate a complete 3D reconstruction result with realistic texture through a confidence-driven fusion strategy.

[0159] In another embodiment, a computer-readable storage medium is provided having computer program instructions stored thereon, which, when executed by a processor, implement the method described in any of the preceding embodiments.

[0160] In another embodiment, an electronic device is provided, comprising: A memory that stores computer program instructions; A processor, coupled to the memory, implements the method described above when executing instructions in the memory.

[0161] The two-level neural registration architecture disclosed herein includes a coarse registration network and a fine registration network. The coarse registration network takes three-view RGB images and camera extrinsic parameters as input, constructs candidate search regions based on intrinsic and extrinsic parameters, performs local cross-attention matching on multi-scale learned features within each candidate region, and outputs cross-view feature correspondence and matching confidence. The fine registration network takes three-view point clouds, RGB learned features, matching confidence, and coarse registration error covariance as input, extracts geometric features of the point cloud through a geometric feature encoding module, aggregates three-view information through a cross-view feature fusion module with the frontal view as a common reference, and then outputs three types of information through a rigid pose correction branch, a local deformation correction branch, and an uncertainty branch, respectively.

[0162] The coarse registration network takes both image data and extrinsic parameter data as input to fuse learned feature matching and geometric constraints at the input. The fine registration network takes the coarse registration error covariance as explicit input to inform the network of the uncertainty of coarse registration so as to optimize the fine registration result more effectively. The three independent branches output stiffness correction, local deformation correction and uncertainty, respectively, to decouple the three types of information to avoid mutual interference.

[0163] In one embodiment, the specific structure of the extrinsic-constrained cross-view RGB feature registration network is as follows: the parameter-sharing multi-scale feature extraction backbone consists of a four-level feature pyramid composed of local convolutional coding units and hierarchical context coding units, with the number of channels at each level being 64, 128, 256, and 256, respectively; the features at each scale are fused from top to bottom to form a unified matching feature map with a dimension of 128; the view conditional modulation layer uses conditional instance normalization to modulate the feature map at each scale separately; the stable region weighting layer consists of three convolutional layers, outputting a stable probability map for a single channel; the cross-view feature matching layer uses a local cross-attention mechanism to calculate feature similarity within candidate regions.

[0164] The hyperparameters are determined as follows: the number of channels at each level (64, 128, 256, and 256) can be determined by performing a grid search on the constructed dataset, with a search range of 32-512 and a step size of powers of 2; the matching feature dimension (128) can be determined by comparing the performance of 64, 128, and 256 dimensions on the validation set, selecting the dimension that achieves the best balance between registration accuracy and computational efficiency; temperature parameter... It is initially set to 0.1 and can be adaptively adjusted during training based on the entropy of the matching probability distribution.

[0165] The four-level feature pyramid has 64 / 128 / 256 / 256 channels, which is to preserve detailed features at the lower levels and capture semantic features at the higher levels to balance matching accuracy and robustness; the matching feature dimension is 128, which is to achieve a balance between high-dimensional expressive power and computational efficiency.

[0166] In another embodiment, the specific structure of the three-view rigid and non-rigid joint fine registration network is as follows: The geometric feature encoding module adopts the PointNet++ architecture to extract geometric features from the point clouds of each viewpoint. The number of encoding layers is 3, and the number of sampling points in each layer is 512, 256, and 128, respectively. The cross-view feature fusion module takes the geometric features of the frontal viewpoint as a reference and aligns the features of the left and right views to the frontal viewpoint coordinate system through a cross-attention mechanism. The rigid pose correction branch consists of three fully connected layers and outputs a 6-dimensional pose correction (3-dimensional rotation + 3-dimensional translation). The local deformation correction branch adopts a graph neural network structure to predict the 3D deformation offset of each point on the point cloud neighborhood graph. The uncertainty branch consists of two fully connected layers and outputs the scalar uncertainty of each point.

[0167] The hyperparameters were determined as follows: the number of encoding layers and sampling points in PointNet++ were determined by comparing the registration errors under different configurations on the dataset; the number of fully connected network layers in the rigid pose correction branch was determined through cross-validation, and the number of hidden layer neurons was set to 256; the neighborhood radius of the graph neural network in the local deformation correction branch was set to 0.05m, and determined by comparing the deformation smoothness under different neighborhood radii; the Adam optimizer was used during training, with an initial learning rate of... The batch size is 8, and the number of training rounds is no less than 200.

[0168] The geometric feature encoding module uses the PointNet++ architecture to extract point cloud features in order to effectively encode geometric structure information in three-dimensional space; the local deformation correction branch uses a graph neural network structure to propagate deformation information on the neighborhood graph of the point cloud to achieve smooth local deformation estimation.

[0169] In one embodiment, the extrinsic-constrained cross-view RGB feature registration network of this disclosure constructs difficult negative samples during the training phase as follows: For each anchor pixel (located in the left-view face image), its positive sample is the corresponding pixel of the same face surface position in the frontal view; the difficult negative samples include three categories: the first category is pixels that are symmetrical to the anchor pixel in the face but have different three-dimensional positions (e.g., the left corner of the eye corresponds to the right corner of the eye), the second category is pixels with similar local textures in the candidate search region but with different depth values, and the third category is pixels located near the occlusion boundary. Through these three categories of difficult negative samples, the network learns to distinguish between true cross-view correspondences and false correspondences caused by symmetrical regions or similar textures.

[0170] The first type of hard negative sample targets the left-right symmetrical structure of the face, in order to make the network rely not only on local appearance similarity but also on spatial location and geometric constraints during matching to reduce the probability of left-right confusion; the second type of hard negative sample targets weak texture regions, in order to enable the network to make correct judgments in regions that are similar in appearance but do not actually correspond; the third type of hard negative sample targets occlusion boundaries, in order to improve the robustness of the network under partial occlusion conditions.

[0171] In another embodiment, the training data augmentation method of the fine registration network of this disclosure is as follows: applying... Rotational disturbance and Translational perturbations are used to generate training samples simulating coarse registration errors; random dropout (dropout probability 0.1-0.3) is applied to the matching confidence to simulate partial failure of feature matching; Gaussian noise (standard deviation equal to the depth value) is applied to the depth values. to ), simulating the measurement noise of a depth sensor; applying a sensor around the eyes of the face. Local deformation disturbances are applied in the perioral region. Local deformation perturbations are used to simulate changes in facial expressions.

[0172] Applying rotational and translational perturbations to the external parameters enables the fine registration network to correct the rigid body errors remaining from the coarse registration during inference; applying Gaussian noise to the depth values ​​improves the network's robustness to noise from depth sensor measurements; applying local deformation perturbations around the eyes and mouth enables the network to learn to estimate non-rigid deformations caused by facial expressions.

[0173] In another embodiment, the hole processing flow of this disclosure is as follows: Consistent clustering based on normal direction is performed on the fused face point cloud to identify missing regions as holes; for each hole, the position and normal information of its boundary points are obtained; new sampling points are generated within the hole region using radial basis function interpolation, and the density of the new sampling points is consistent with the density of the point cloud surrounding the hole; the filled region is locally smoothed to ensure a continuous transition in the normal direction between the filled points and the surrounding point cloud; finally, the filled region is merged with the original point cloud to form a complete point cloud. When identifying holes, a distinction is made between real physiological openings such as nostrils, mouth slits, and ear canal recesses and measurement holes; real physiological openings are not filled.

[0174] Consistent clustering based on normal direction is used to identify holes, which is to distinguish the measurement missing area from the real physiological opening by utilizing the continuity of the facial surface and the uniformity of the normal direction. The radial basis function interpolation method generates new sampling points to reconstruct the missing surface in a natural and smooth manner by using the hole boundary information. Local smoothing makes the filling point transition continuously with the surrounding normal, in order to avoid visual discontinuities and sharp edges in the filling area.

[0175] This disclosure combines camera geometric priors with cross-view feature matching of neural networks, using extrinsic constraints to limit the feature search region to suppress mismatches in symmetrical facial regions, and improves the reliability of coarse registration by stabilizing region weights and matching uncertainty. In the fine registration stage, a rigid-non-rigid joint network is used to simultaneously estimate the rigid motion of the head and the non-rigid deformation of facial expressions in the same model, and the output pointwise uncertainty is used for confidence-weighted fusion, overcoming the technical bias of traditional methods where rigid and non-rigid registration are performed in stages and errors accumulate. Compared with the existing marker coarse registration plus ICP fine registration scheme, this disclosure is expected to achieve significant improvements in registration accuracy, symmetrical region mismatch rate and processing efficiency.

[0176] Although embodiments of the present disclosure have been described above in conjunction with the accompanying drawings, the present disclosure is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and instructive, and not restrictive. Those skilled in the art, guided by this specification, can make many modifications and variations without departing from the scope of protection of the claims of this disclosure, and all such modifications and variations are within the scope of protection of this disclosure.

Claims

1. A three-dimensional face modeling method based on arc-shaped speckle structured light RGB-D, characterized in that, Includes the following steps: Step S1: Acquire continuous multi-frame RGB-D data from the left, front, and right viewpoints, collected sequentially by three RGB-D depth cameras set along the arc of the face contour. Based on the intrinsic parameters of each camera and the extrinsic parameters between cameras, transform the point cloud data of each viewpoint to a unified coordinate system with the front viewpoint as the reference to form the initial point cloud of each viewpoint. Step S2: Using an extrinsic-constrained cross-view RGB feature registration network, extract view-invariant face features from the RGB images of the left, front, and right views, and establish cross-view feature correspondences within the candidate search region defined by the inter-camera extrinsic parameters, and output the matching confidence of each feature. Step S3: The corresponding pixels in the cross-view feature correspondence are elevated to three-dimensional corresponding points based on their corresponding depth values. The constraints of the camera extrinsic parameters and the constraints of the three-dimensional corresponding points are fused to perform a weighted robust pose solution, and the coarse registration transformations of the left view and the right view relative to the front view are obtained. Step S4: Using a three-view rigid and non-rigid joint fine registration network, based on the left-view, front-view, and right-view point clouds after the coarse registration transformation, output the rigid pose correction, local deformation correction, and corresponding uncertainty. Step S5: Apply the rigid pose correction amount and the local deformation correction amount to correct the point cloud of each viewpoint. Perform weighted fusion of the point cloud of each viewpoint according to the uncertainty of the correspondence relationship. Then, after hole filling, feature preservation smoothing and RGB texture attachment, output a three-dimensional face model.

2. The method according to claim 1, characterized in that, In step S1, the time-division sequential acquisition includes performing continuous multi-frame acquisition in the order of the first frontal frame group, the left frame group, the second frontal frame group, the right frame group, and the third frontal frame group. During any active depth acquisition period, only the speckle projection of the corresponding camera is enabled.

3. The method according to claim 1, characterized in that, In step S2, the extrinsic parameter-constrained cross-view RGB feature registration network includes a parameter-shared multi-scale feature extraction backbone, a viewpoint conditional modulation layer, a stable region weighting layer, and a cross-view feature matching layer. The multi-scale feature extraction backbone extracts learning feature maps at different scales from the RGB images of the three viewpoints. The viewpoint conditional modulation layer modulates the learning feature maps based on the camera imaging ray direction, camera relative pose encoding, and normalized pixel position of each pixel. The stable region weighting layer predicts the probability that each pixel belongs to a stable facial region and weights the learning feature maps accordingly.

4. The method according to claim 1, characterized in that, In step S2, the candidate search region is constructed as follows: when the pixel to be matched in the source view has a valid depth value, the pixel to be matched is back-projected into three-dimensional space based on the intrinsic parameters of the source camera, and then the predicted position of the pixel to be matched in the target view is obtained based on the extrinsic parameters from the source camera to the target camera. The candidate search region is established by combining the extrinsic parameter calibration error, depth measurement uncertainty, and projection Jacobian to form anisotropic two-dimensional projection uncertainty. When the pixel to be matched does not have a reliable depth value, an epipolar line is formed in the target image based on the camera's intrinsic and extrinsic parameters, and a strip-shaped region of a predetermined width near the epipolar line is taken as the candidate search region.

5. The method according to claim 1, characterized in that, In step S2, establishing cross-view feature correspondence includes: for the pixel to be matched in the source view, calculating feature similarity for each candidate position in the candidate search area, using geometric deviation as an additional penalty term, and obtaining the conditional probability that each candidate position belongs to the true corresponding position after normalization; selecting the candidate position with the highest probability as the discrete corresponding position or weighting and summing the candidate positions according to the conditional probability to obtain the sub-pixel corresponding position; calculating the matching uncertainty according to the matching probability distribution in the candidate search area, wherein the matching uncertainty is in the form of matching entropy, the matching uncertainty increases when multiple candidate positions have close matching probabilities, and the matching uncertainty decreases when the true corresponding position forms a single-peak high-probability distribution.

6. The method according to claim 5, characterized in that, In step S2, establishing cross-view feature correspondence further includes: for the corresponding position obtained through forward matching, performing reverse matching with the corresponding position as the source position of the target view; if the reverse matching result cannot return to the vicinity of the original pixel to be matched, then the corresponding position is removed or its weight is reduced; for features that have a common visible area in the left view, front view, and right view, the transitivity or cyclic consistency between the left view to the front view, the front view to the right view, and the reverse mapping is checked to remove ambiguous correspondences; for side face features that only have a common visible area with the front view, a bidirectional consistency check with the front view as a common reference is adopted.

7. The method according to claim 3, characterized in that, The stable facial regions predicted by the stable region weighted layer include the forehead, bridge of the nose, cheekbone, and temporal region. The stability probability output by the stable region weighted layer ranges from 0 to 1 and is used as a weight for feature matching and matching confidence calculation, so that the contribution of relatively stable facial regions to coarse registration is greater than that of easily deformable regions.

8. The method according to claim 1, characterized in that, In step S3, the objective function of the weighted robust pose solution includes an RGB learning feature matching constraint term and an extrinsic parameter prior constraint term. The RGB learning feature matching constraint term is weighted based on the comprehensive matching confidence of the corresponding three-dimensional points, and the extrinsic parameter prior constraint term is weighted based on the extrinsic parameter quality matrix determined by the calibration reprojection error and the repeated calibration error. The relative weights of the two constraints are adjusted by a balancing factor to minimize the three-dimensional residual corresponding to high confidence learning while suppressing the deviation of the pose correction from the reliable extrinsic parameter.

9. The method according to claim 1, characterized in that, In step S4, the three-view rigid and non-rigid joint fine registration network includes a geometric feature encoding module, a cross-view feature fusion module with the frontal view as a common reference, a rigid pose correction branch, a local deformation correction branch, and an uncertainty branch. The rigid pose correction branch predicts the rigid pose correction amount based on the features of the stable face region. The local deformation correction branch predicts the local deformation correction amount based on the features of the easily deformable face region. The uncertainty branch outputs the point-by-point correspondence uncertainty.

Images