A robot face self-adaptive projection method and system based on real-time visual feedback
By constructing a 3D digital twin model and using real-time visual feedback, the problem of dynamic facial deformation in robots was solved, achieving high-precision adaptive projection and improving the robustness of the projection system and user experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SICHUAN EMBODIED HUMANOID ROBOT TECHNOLOGY CO LTD
- Filing Date
- 2025-12-17
- Publication Date
- 2026-06-09
Smart Images
Figure CN121708220B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robot perception technology, and more specifically, to a robot facial adaptive projection method and system based on real-time visual feedback. Background Technology
[0002] Existing projection calibration technologies mostly rely on static templates or manual adjustments, which cannot adapt to dynamic deformations (such as surface changes in a robot's face due to collisions or thermal expansion). For example, while traditional structured light systems (such as depth cameras based on single-frame stripes) can reconstruct planes, their accuracy in accurately reproducing concave and convex surfaces (such as nasal bridge protrusions and eye socket depressions) is insufficient (error > 1mm), and feature registration relies on preset models, making it difficult to handle asymmetrical deformations. Furthermore, existing systems lack closed-loop feedback, making it impossible to verify projection distortion in real time. This results in misalignment of key areas such as eye spacing and mouth position in different character models (such as cartoon faces) during projection, affecting user experience. A search of comparative documents shows that structured light 3D reconstruction (such as Gray code + phase shift method) and curvature analysis algorithms (such as principal curvature calculation) are publicly available, but they have not been integrated into the robot head to achieve template-free adaptive registration. Summary of the Invention
[0003] The purpose of this invention is to provide a robot facial adaptive projection method and system based on real-time visual feedback to solve the problems in the prior art.
[0004] This invention is achieved through the following technical solution:
[0005] In a first aspect, the present invention provides a robot facial adaptive projection method based on real-time visual feedback, comprising:
[0006] The original image data to be projected is obtained. After the original image data is projected onto the robot's face, the first projected image is obtained. The first projected image is preprocessed to obtain a RAW image.
[0007] The process involves constructing a 3D digital twin model of the current RAW image, defining the target 3D model, calibrating the geometric parameters based on the target 3D model, outputting an ordered point cloud, and triangulating it to generate a mesh surface.
[0008] Calculate the curvature properties of the vertices of each grid in the grid surface to obtain a global curvature map. Based on the global curvature map, extract facial features to obtain the first semantic network label composed of each feature combination.
[0009] The first semantic network label is matched and detected. If it matches, no processing is performed. If it does not match, the expected image is obtained. Based on the expected image, the projection parameters are obtained inversely through a 3D digital twin model. The projection is performed based on the projection parameters, and the second projection image of the current projection is obtained. The second semantic network label is obtained by extracting facial features from the second projection image. The second semantic network label is matched and detected. If it matches, the current projection parameters are saved. If it does not match, the projection parameters are iteratively adjusted and an iteration upper limit is set. When the iteration upper limit is reached, the original image data is projected with an error.
[0010] Preferably, the curvature properties of the vertices of each mesh in the computational mesh surface include:
[0011] Mesh surfaces are generated based on Delaunay triangulation. Vertices and all adjacent vertices within their one or two ring neighborhoods are selected to form a local point set.
[0012] For each vertex, perform PCA fitting on the local point set to establish a local coordinate system. Fit the height to a quadratic surface and calculate the curvature based on the fitting coefficients.
[0013] Preferably, the high-fitting quadratic surface includes:
[0014]
[0015] The fitting coefficients include:
[0016]
[0017]
[0018] Calculating curvature includes:
[0019]
[0020] in, For height, for Term coefficient, , For the coordinates of the tangent plane, for Term coefficient, for Term coefficient, for Term coefficient, for Term coefficient, For constant terms, For the mean curvature, For Gaussian curvature, Principal curvature.
[0021] Preferably, the facial feature extraction based on the global curvature map includes:
[0022] Iterate through the highest vertex in the global curvature map and set confirmation conditions. If the confirmation conditions are met, set it as the nose tip feature and use it as the center anchor point. If the confirmation conditions are not met, report an error.
[0023] Eye and mouth features are identified based on the central anchor point, and first semantic network labels are generated based on the obtained nose tip features, eye features, and mouth features.
[0024] Preferably, the extraction of eye features includes:
[0025] Set a curvature threshold, find concave points smaller than the curvature threshold adjacent to the center anchor point based on the global curvature map, fit ellipse to the concave points to obtain concavity features, and obtain eye features based on the concavity features.
[0026] Preferably, the identification of mouth features includes:
[0027] Multi-layer horizontal contours are extracted, contour symmetry is quantized based on Fourier descriptors, saddle points are searched based on local set features of the mouth region, and the center line and target corner points are fitted. Mouth features are obtained based on the center line and target corner points.
[0028] Preferably, the matching detection includes feature point reprojection error detection, 3D model consistency detection, and curvature attribute consistency detection;
[0029] The feature point reprojection error detection includes:
[0030] The nose tip features, eye features, and mouth features are back-projected onto the two-dimensional camera image coordinate system using pre-calibrated camera parameters to obtain several sets of projected feature points. The nose tip features, eye features, and mouth features are then detected and extracted from the first projected image using a two-dimensional image algorithm to obtain the real feature points.
[0031] Calculate the Euclidean distance between the projected feature points and the real feature points, and set the reprojection error. Determine if the Euclidean distance is less than the reprojection error. If it is, then it is a match; otherwise, it is a mismatch.
[0032] Preferably, the three-dimensional model consistency detection includes:
[0033] The target 3D model is iteratively registered with the reference 3D model to obtain the optimal rigid body transformation.
[0034] After ICP convergence, the average distance and Hausdorff distance between corresponding points in the two models are calculated, and the average distance threshold and Hausdorff distance threshold are set. If the average distance is less than or equal to the average distance threshold and the Hausdorff distance threshold is less than or equal to the Hausdorff distance threshold, then it is a match; otherwise, it is a mismatch.
[0035] Preferably, the curvature attribute consistency detection includes:
[0036] Calculate the first curvature value of any target feature among the nose tip feature, eye feature, and mouth feature in the baseline 3D model, and obtain the second curvature value of the target feature in the target 3D model;
[0037] The curvature matching rate is calculated based on the first curvature value and the second curvature value, and a matching rate threshold is set. If the curvature matching rate is less than the matching rate threshold, no match is made; if it is greater than or equal to the matching rate threshold, a match is made.
[0038] Secondly, a robot face adaptive projection system based on real-time visual feedback is provided for executing the aforementioned robot face adaptive projection method based on real-time visual feedback, comprising:
[0039] The image processing module is configured to acquire the raw image data to be projected, project the raw image data onto the robot's face, acquire the first projected image, preprocess the first projected image, and obtain a RAW image.
[0040] The process involves constructing a 3D digital twin model of the current RAW image, defining the target 3D model, calibrating the geometric parameters based on the target 3D model, outputting an ordered point cloud, and triangulating it to generate a mesh surface.
[0041] Calculate the curvature properties of the vertices of each grid in the grid surface to obtain a global curvature map. Based on the global curvature map, extract facial features to obtain the first semantic network label composed of each feature combination.
[0042] The feedback module is configured to perform matching detection on the first semantic network label. If a match is found, no processing is performed. If no match is found, the desired image is obtained. Based on the desired image, the projection parameters are obtained inversely through a 3D digital twin model. The projection is performed based on the projection parameters, and the second projection image of the current projection is obtained. The second semantic network label is obtained by extracting facial features from the second projection image. The second semantic network label is then matched. If a match is found, the current projection parameters are saved. If no match is found, the projection parameters are iteratively adjusted, and an iteration limit is set. When the iteration limit is reached, the original image data is projected with an error.
[0043] The technical solution of the present invention has at least the following advantages and beneficial effects:
[0044] The method described above mainly includes constructing a target 3D digital twin model of the current RAW image, calibrating geometric parameters based on the target 3D model, outputting an ordered point cloud, triangulating it to generate a mesh surface, calculating the curvature attribute of each vertex in the mesh surface to obtain a global curvature map, extracting facial features based on the global curvature map to obtain a first semantic network label composed of each feature combination, and performing matching detection on the first semantic network label. This method supplements the existing method with precise geometric attribute reproduction algorithms (such as local quadratic fitting curvature and mirror asymmetric quantization), filling the gap and improving the system's robustness. Attached Figure Description
[0045] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0046] Figure 1 This is a schematic diagram of the process of the present invention;
[0047] Figure 2 This is a schematic diagram of the system structure of the present invention. Detailed Implementation
[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.
[0049] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. The naming or numbering of steps in this application does not imply that the steps in the method flow must be executed in the chronological / logical order indicated by the naming or numbering. The execution order of named or numbered process steps can be changed according to the desired technical objective, as long as the same or similar technical effect is achieved.
[0050] The module division in this application is a logical division. In actual application, there may be other division methods. For example, multiple modules may be combined into or integrated into another system, or some features may be ignored or not executed.
[0051] The independently described modules or sub-modules may or may not be physically separated; they may be implemented in software or hardware, and some modules or sub-modules may be implemented in software, with the processor calling the software to implement the function of these modules or sub-modules, while other modules or sub-modules may be implemented in hardware, such as through hardware circuits. Furthermore, some or all of the modules can be selected to achieve the purpose of this application's solution according to actual needs.
[0052] Please refer to Figures 1-2 This invention provides a robot facial adaptive projection method based on real-time visual feedback, comprising:
[0053] S101: Obtain the original image data to be projected, project the original image data onto the robot's face, obtain the first projected image, preprocess the first projected image to obtain a RAW image;
[0054] A hybrid encoding method is employed, combining Gray code (N=8-10 bits, providing absolute fringe index) and phase shifting (3-4 steps of phase shifting, θ=120°) to ensure absolute positioning and relative accuracy. Pattern sequences are dynamically generated (first coarse Gray code, then fine phase shifting), with adaptive projection intensity (based on ALS, 0.5-2W) to ensure that distortion after surface modulation is decodeable, and high-frequency triggering (<100ms / frame) is supported.
[0055] Image acquisition: After projection, the camera is synchronously triggered to acquire images (delay <1ms, GPIO signal), obtaining the feature pattern deformed by facial surface modulation. Preprocessing includes denoising (median filtering, kernel 3x3) and multi-frame fusion (M=3), exposure <1ms anti-blurring, and outputting RAW images for subsequent decoding.
[0056] S102: Construct a 3D digital twin model of the current RAW image, calibrate the geometric parameters based on the target 3D model, output an ordered point cloud, and triangulate it to generate a mesh surface;
[0057] The pattern decoding algorithm uses Gray code decoding (thresholding intensity I>128 to binary, Gray_to_Binary conversion) and phase-shift decoding (least squares fitting of the wrapped phase). , unfold By combining the calculation coefficients (where k is determined by the Gray code) to achieve relative decoding, the sub-pixel accuracy is improved (<0.05 pixels), and small surface distortions (such as the concavity and convexity of the bridge of the nose) can be captured.
[0058] Based on pre-calibrated geometric parameters (baseline B = 10-20cm, included angle) ), calculate 3D coordinates through triangulation ( d is the parallax. 3D coordinates For), output an ordered point cloud (density ≥ 1 point / mm², covering a projection surface of 100k-500k points).
[0059] Finally, point cloud filtering (statistical filtering) is performed. After generating the mesh surface using Delaunay triangulation, the curvature is calculated for each vertex on the mesh.
[0060] Finally, the curvature properties of each vertex are generated to obtain a global curvature map, which serves as precise input for subsequent geometric analysis and feature registration.
[0061] S103: Calculate the curvature properties of the vertices of each grid in the grid surface to obtain a global curvature map. Based on the global curvature map, extract facial features to obtain the first semantic network label composed of each feature combination.
[0062] S104: Perform matching detection on the first semantic network label. If it matches, no processing is performed. If it does not match, the expected image is obtained. Based on the expected image, the projection parameters are obtained inversely through the 3D digital twin model. Projection is performed based on the projection parameters, and the second projection image of the current projection is obtained. The second semantic network label is obtained by extracting facial features from the second projection image. The second semantic network label is matched. If it matches, the current projection parameters are saved. If it does not match, the projection parameters are iteratively adjusted, and an iteration upper limit is set. When the iteration upper limit is reached, the original image data is projected with an error.
[0063] The method described above mainly includes constructing a target 3D model from the current RAW image using a 3D digital twin model; calibrating geometric parameters based on the target 3D model; outputting an ordered point cloud and triangulating it to generate a mesh surface; calculating the curvature attribute of each vertex in the mesh surface to obtain a global curvature map; extracting facial features based on the global curvature map to obtain a first semantic network label composed of each feature combination; and performing matching detection on the first semantic network label. This method supplements the existing precise geometric attribute reproduction algorithms (such as local quadratic fitting curvature and mirror asymmetry quantization), filling the gap and improving system robustness. It addresses how to capture and reproduce the micrometer-level concavity, curvature, and asymmetry of the projection surface in real time, avoiding idealization bias in the reconstructed model; how to automatically register key features (such as nose tip, eye socket, and mouth) based on pure geometric inference to achieve adaptive projection adjustment for different character models without the need for preset templates; and how to construct a closed-loop feedback mechanism to ensure projection error <0.5mm and support high-frequency triggering (e.g., once per second).
[0064] In one exemplary embodiment of the present invention, calculating the curvature properties of the vertices of each mesh in the mesh surface includes:
[0065] Mesh surfaces are generated based on Delaunay triangulation. Vertices and all adjacent vertices within their one or two ring neighborhoods are selected to form a local point set.
[0066] For each vertex, perform PCA fitting on the local point set to establish a local coordinate system. Fit the height to a quadratic surface and calculate the curvature based on the fitting coefficients.
[0067] Specifically, highly fitted quadratic surfaces include:
[0068]
[0069] The fitting coefficients include:
[0070]
[0071]
[0072] Calculating curvature includes:
[0073]
[0074] in, For height, for Term coefficient, , For the coordinates of the tangent plane, for Term coefficient, for Term coefficient, for Term coefficient, for Term coefficient, For constant terms, For the mean curvature, For Gaussian curvature, Principal curvature.
[0075] An exemplary embodiment of the present invention, facial feature extraction based on a global curvature map, includes:
[0076] Iterate through the highest vertex in the global curvature map and set confirmation conditions. If the confirmation conditions are met, set it as the nose tip feature and use it as the center anchor point. If the confirmation conditions are not met, report an error.
[0077] Eye and mouth features are identified based on the central anchor point, and first semantic network labels are generated based on the obtained nose tip features, eye features, and mouth features.
[0078] Specifically, based on geometric parameters, automated geometric reasoning and feature extraction are performed to establish a unique "feature fingerprint," i.e., facial features, to achieve adaptive mapping of the projected content. First, the point cloud is scanned to locate the global maximum point on the Z-axis. Within its neighborhood (r=5-10mm), if the average curvature H>0.03 mm... - ¹And the standard deviation std(H) < 0.01 mm - ¹, Confirm the convex peak feature, determine the "nose tip" reference as the center anchor point, and apply an affine transformation to the coordinate system.
[0079] Secondly, the extraction of eye features includes: setting a curvature threshold, finding concave points smaller than the curvature threshold adjacent to the center anchor point based on the global curvature map, fitting ellipses to the concave points to obtain concavity features, and obtaining eye features based on the concavity features.
[0080] The system generates a surface curvature variation map (calculating the principal curvature as the eigenvalue of the Hessian matrix for each vertex) and thresholds negative curvature regions (H < -0.015 mm). - ¹), and use density clustering algorithms such as DBSCAN ( Minimum sample size: 50 Grouping adjacent concave points by the radius of the domain, a "concave point" refers to a vertex on the 3D mesh surface whose local geometry exhibits concave characteristics, defined by an average curvature H < -0.015 mm. - ¹ is used to determine whether a vertex is a "concave point" (negative curvature indicates a local concave surface); "adjacent" means that these concave points are close to each other in 3D space or mesh topology. For example, in a Delaunay triangular mesh, two concave points are considered adjacent if they share an edge or are located in the same local neighborhood (such as a ring / double ring neighborhood).
[0081] Then, ellipse fitting is performed on the candidate regions (satisfying the least squares solution (uh)² / a²+(vk)² / b²=1, aspect ratio 1.2-1.8) to filter specific concavity features with an area >150mm². Based on the specific concavity features, the expected region of the "eye socket" is accurately located (divided into left and right eye sockets according to the X-axis, and the distance between the two eyes is calculated).
[0082] After performing DBSCAN density clustering on all the concave points, several connected clusters of concave points are obtained. Each such cluster constitutes a "candidate region", and each candidate region represents a potential eye socket location, because the human eye socket is a typical, continuous concave region on the face with certain area and shape regularity.
[0083] Therefore, the "candidate region" is the local surface region corresponding to a spatially continuous set of concave points formed by DBSCAN clustering.
[0084] Secondly, the identification of mouth features includes:
[0085] Multi-layer horizontal contours are extracted, contour symmetry is quantized based on Fourier descriptors, saddle points are searched based on local set features of the mouth region, and the center line and target corner points are fitted. Mouth features are obtained based on the center line and target corner points.
[0086] Specifically, mouth feature recognition includes the following steps:
[0087] 1. Extract 10–20 horizontal contour sections at 1.5–3 mm intervals within a 30 mm range downward along the Z-axis from the tip of the nose;
[0088] 2. Calculate the Fourier descriptor quantization profile symmetry for each profile z(t) = x(t) + iy(t), and its k-th order coefficients. Defined as:
[0089]
[0090] Where x(t) and y(t) are the coordinates of the t-th point of the contour, i is the imaginary unit, k is the order of the Fourier coefficients, and N is the total number of contour points.
[0091] And calculate its similarity S with the mirror contour, where the mirror contour is obtained by inverting the original contour about the Y-axis, i.e., z_mirror(t) = x(t)+iy(t); The similarity S is defined as the normalized cross-correlation value of the low-frequency Fourier coefficients (e.g., |k|≤5) of the original contour and the mirror contour. If S>0.9, it is confirmed that the region is symmetrical about the facial midline, and the principal axis of symmetry is established.
[0092] 3. Calculate the Hessian matrix (second derivative matrix) of each vertex in the candidate mouth region (near the axis of symmetry, 15–25 mm below the tip of the nose), and search for saddle points that satisfy trace tr(Hess)≈0 and determinant det(Hess)<0, as vertical references for the mouth.
[0093] 4. Using the detected saddle points and their neighboring points as input, the RANSAC algorithm is used to fit the mouth centerline, requiring an in-point ratio greater than 80%. This straight line is the mouth centerline.
[0094] 5. Calculate the Harris corner response at each vertex on both sides of the centerline: First, construct a structure tensor based on curvature gradient, obtain its eigenvalues λ1, λ2, and let R = min(λ1, λ2), filtering those satisfying R > 0.01 (unit: mm). -2The points are selected from the points, and the two points with the smallest and largest X coordinates are chosen. If the horizontal distance between the two points is greater than 20 mm, they are marked as the key corner points of the left and right corners of the mouth, respectively.
[0095] Through this series of automated geometric reasoning and feature extraction, the system further quantifies the asymmetry (Hausdorff distance of the mirrored point cloud P_m > 0.5 mm). Specifically, the reconstructed 3D point cloud P is mirrored about the principal axis of symmetry established by the center lines of the nose tip and mouth, generating a mirrored point cloud Pm; the Hausdorff distance dH(P,Pm) between the two is calculated. If this distance is greater than 0.5 mm, it is determined that the current robot's face has significant asymmetric deformation (such as due to manufacturing tolerances, collisions, or thermal expansion), and the system will retain this asymmetry as a key geometric feature; otherwise, a symmetric simplified model can be used to improve computational efficiency. This asymmetry index, together with semantic features such as the nose tip, eye sockets, and corners of the mouth, is integrated into a unique "feature fingerprint".
[0096] All outputs are integrated into a unique "feature fingerprint," and a first semantic label mesh is generated (each vertex / face is labeled, such as "left eye socket: region_ID"). This transforms any shape of physical projection surface from an undefined curved surface into a display carrier with clear semantic labels that the system fully "understands." Based on this, the adaptive affine transformation matrix A=s·R·T (s=d_model / d_eyes, R is the PCA pose, T is the nose tip translation) is calculated. d_model is the standard interocular distance (in mm) in the expected content model, such as the preset interocular distance when designing the target projection image (such as a standard face template or cartoon character). d_eyes is the actual measured interocular distance (in mm), which is the Euclidean distance from the center of the left eye socket to the center of the right eye socket extracted in the previous step. This lays an indispensable perceptual foundation for the subsequent accurate and adaptive mapping of digital content to physical surfaces, supporting closed-loop verification (such as reprojection SSIM>0.95) and real-time deformation adaptation (accuracy <0.5mm).
[0097] In one exemplary embodiment of the present invention, after the feature registration stage, the system obtains an accurate digital twin model and its corresponding feature fingerprint. Mismatch detection aims to verify whether the digital twin model is consistent with the state of the current two-dimensional physical projection surface. The detection is performed at three levels, all based on three-dimensional geometry and feature comparison.
[0098] The matching detection includes feature point reprojection error detection, 3D model consistency detection, and curvature attribute consistency detection.
[0099] The feature point reprojection error detection includes:
[0100] The nose tip features, eye features, and mouth features are back-projected onto the two-dimensional camera image coordinate system using pre-calibrated camera parameters to obtain several sets of projected feature points. The nose tip features, eye features, and mouth features are then detected and extracted from the first projected image using a two-dimensional image algorithm to obtain the real feature points.
[0101] Calculate the Euclidean distance between the projected feature points and the real feature points, and set the reprojection error. Determine if the Euclidean distance is less than the reprojection error. If it is, then it is a match; otherwise, it is a mismatch.
[0102] In this embodiment, the reprojection error is set to 3 pixels.
[0103] Secondly, the consistency check of the 3D model includes:
[0104] The target 3D model and the reference 3D model are iteratively registered to the nearest point to obtain the optimal rigid body transformation. The reference 3D model is the 3D model constructed by adjusting the projection method during calibration.
[0105] After ICP convergence, the average distance and Hausdorff distance between corresponding points in the two models are calculated, and the average distance threshold and Hausdorff distance threshold are set. If the average distance is less than or equal to the average distance threshold and the Hausdorff distance threshold is less than or equal to the Hausdorff distance threshold, then it is a match; otherwise, it is a mismatch.
[0106] Secondly, curvature property consistency detection includes:
[0107] Calculate the first curvature value of any target feature among the nose tip feature, eye feature, and mouth feature in the baseline 3D model, and obtain the second curvature value of the target feature in the target 3D model;
[0108] The curvature matching rate is calculated based on the first curvature value and the second curvature value, and a matching rate threshold is set. If the curvature matching rate is less than the matching rate threshold, no match is made; if it is greater than or equal to the matching rate threshold, a match is made.
[0109]
[0110] In the formula, Where N is the curvature matching rate, and N is the number of sampling points. The first curvature value, This is the second curvature value. The tolerance parameter is a scale parameter that controls the tolerance for curvature differences, and the matching rate threshold is set to 0.85.
[0111] If any of the above three checks exceeds the threshold, the pattern optimization process is triggered; otherwise, the digital twin model is considered to be still valid, and the current mapping relationship is used directly for the final content projection.
[0112] Pattern Optimization and New Pattern Generation: Based on feedback from mismatch detection, the system confirms that the current projected pattern no longer fits the physical surface. The core goal of optimization is to generate a pre-distorted pattern so that when projected onto the current physical surface, the desired target image can be seen from the observer's (or main camera's) perspective. A reverse rendering method based on a 3D digital twin model is employed. We expect to obtain the target image I_target: the final image displayed on the physical surface (such as a distortion-free cartoon face or standard phase-shifted fringes). When it is necessary to accurately attach complex textures to the entire curved surface, the following reverse rendering method is used.
[0113] First, import the current accurate digital twin mesh model S(u, v). Apply the desired target image I_target as a diffuse texture to model S. Then, set up the virtual camera; the intrinsic parameters (focal length, principal point) and extrinsic parameters (position, pose) of the virtual camera (View Camera) must strictly match the calibration parameters of the real viewing camera. Next, set up the virtual projector; the intrinsic and extrinsic parameters of the virtual projector must strictly match the calibration parameters of the real projector. Finally, render the pre-distortion pattern from the virtual projector's viewpoint, rendering the model S with the I_target texture. The resulting image I_proj is the pre-distortion pattern, containing all geometric distortion compensations, that needs to be sent to the real projector.
[0114] Reprojection and loop closure verification: The generated new pattern sequence is projected and the modulated image I_new is acquired simultaneously.
[0115] Reconstruction and Registration: Based on I_new, perform 3D reconstruction, point cloud processing, meshing, and feature registration to obtain a new feature fingerprint F' and transformation relationship A' based on the current projection result.
[0116] Closed-loop verification: This verification aims to confirm whether the newly generated pattern I_proj effectively eliminates the mismatch. First, the average Euclidean distance d_new between the key feature points in F' (such as the center of the eye socket and the corners of the mouth) and their target design positions (i.e., the ideal 3D positions corresponding to the desired pattern I_target) is calculated. If d_new < 0.5 mm, the projected content is considered to be accurately aligned to the target area, and the verification is successful.
[0117] If the above verification is successful, the optimized pattern I_proj and transformation relation A' are locked and applied to the final pattern projection. If the verification fails, a new round of optimization iteration is triggered, returning to step 6, and using the previous I_proj or F' as the initial reference for re-optimizing the pattern and generating a new pattern. To prevent infinite loops, a maximum number of iterations is set (e.g., 3 times). If convergence is not achieved after exceeding the number of iterations, an error is reported and the default projection mode is used.
[0118] Secondly, a robot face adaptive projection system based on real-time visual feedback is provided for executing the aforementioned robot face adaptive projection method based on real-time visual feedback, comprising:
[0119] The image processing module is configured to acquire the raw image data to be projected, project the raw image data onto the robot's face, acquire the first projected image, preprocess the first projected image, and obtain a RAW image.
[0120] The process involves constructing a 3D digital twin model of the current RAW image, defining the target 3D model, calibrating the geometric parameters based on the target 3D model, outputting an ordered point cloud, and triangulating it to generate a mesh surface.
[0121] Calculate the curvature properties of the vertices of each grid in the grid surface to obtain a global curvature map. Based on the global curvature map, extract facial features to obtain the first semantic network label composed of each feature combination.
[0122] The feedback module is configured to perform matching detection on the first semantic network label. If a match is found, no processing is performed. If no match is found, the desired image is obtained. Based on the desired image, the projection parameters are obtained inversely through a 3D digital twin model. The projection is performed based on the projection parameters, and the second projection image of the current projection is obtained. The second semantic network label is obtained by extracting facial features from the second projection image. The second semantic network label is then matched. If a match is found, the current projection parameters are saved. If no match is found, the projection parameters are iteratively adjusted, and an iteration limit is set. When the iteration limit is reached, the original image data is projected with an error.
[0123] Hardware modules:
[0124] 1. Miniature projector;
[0125] 2. High-resolution camera calibration;
[0126] The hardware modules need to be pre-calibrated: the projector-camera extrinsic matrix [R|t] (rotation R is SO(3) group, translation t<5mm error), and the camera intrinsic parameters K (radial distortion k1=-0.1, tangential distortion k2=0.01) are obtained by Zhang calibration method. The pre-calibration baseline B=10-20cm, and the included angle α=45°.
[0127] Software architecture:
[0128] The embedded processor modules include: a pattern generator (dynamic encoding), an image processor (preprocessing and decoding), a 3D reconstructor (point cloud and curvature calculation), a feature registrant (geometric inference), a content mapper (affine transformation and UV mapping), and a feedback controller (mismatch detection and optimization).
[0129] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0130] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. This computer software product, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0131] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A robot facial adaptive projection method based on real-time visual feedback, characterized in that, include: The original image data to be projected is obtained. After the original image data is projected onto the robot's face, the first projected image is obtained. The first projected image is preprocessed to obtain a RAW image. The process involves constructing a 3D digital twin model of the current RAW image, defining the target 3D model, calibrating the geometric parameters based on the target 3D model, outputting an ordered point cloud, and triangulating it to generate a mesh surface. Calculate the curvature properties of the vertices of each grid in the grid surface to obtain a global curvature map. Based on the global curvature map, extract facial features to obtain the first semantic network label composed of each feature combination. The first semantic network label is matched and detected. If it matches, no processing is performed. If it does not match, the expected image is obtained. Based on the expected image, the projection parameters are obtained inversely through the 3D digital twin model. The projection is performed based on the projection parameters, and the second projection image of the current projection is obtained. The second semantic network label is obtained by extracting facial features from the second projection image. The second semantic network label is matched and detected. If it matches, the current projection parameters are saved. If it does not match, the projection parameters are iteratively adjusted and an iteration upper limit is set. When the iteration upper limit is reached, the original image data is projected with an error. The facial feature extraction based on the global curvature map includes: Iterate through the highest vertex in the global curvature map and set confirmation conditions. If the confirmation conditions are met, set it as the nose tip feature and use it as the center anchor point. If the confirmation conditions are not met, report an error. Eye and mouth features are identified based on the central anchor point, and first semantic network labels are generated based on the obtained nose tip features, eye features, and mouth features. The extraction of eye features includes: Set a curvature threshold, find concave points smaller than the curvature threshold adjacent to the center anchor point based on the global curvature map, fit ellipse to the concave points to obtain concave features, and obtain eye features based on the concave features. Mouth feature recognition includes: Multi-layer horizontal contours are extracted, contour symmetry is quantized based on Fourier descriptors, saddle points are searched based on local set features of the mouth region, and the center line and target corner points are fitted. Mouth features are obtained based on the center line and target corner points.
2. The robot face adaptive projection method based on real-time visual feedback according to claim 1, characterized in that, The curvature properties of the vertices of each grid in the computational mesh surface include: Mesh surfaces are generated based on Delaunay triangulation. Vertices and all adjacent vertices within their one or two ring neighborhoods are selected to form a local point set. For each vertex, perform PCA fitting on the local point set to establish a local coordinate system. Fit the height to a quadratic surface and calculate the curvature based on the fitting coefficients.
3. The robot face adaptive projection method based on real-time visual feedback according to claim 2, characterized in that, The high-fit quadratic surface includes: The fitting coefficients include: Calculating curvature includes: in, For height, for Term coefficient, , For the coordinates of the tangent plane, for Term coefficient, for Term coefficient, for Term coefficient, for Term coefficient, For constant terms, For the mean curvature, For Gaussian curvature, Principal curvature.
4. The robot face adaptive projection method based on real-time visual feedback according to claim 3, characterized in that, The matching detection includes feature point reprojection error detection, 3D model consistency detection, and curvature attribute consistency detection. The feature point reprojection error detection includes: The nose tip features, eye features, and mouth features are back-projected onto the two-dimensional camera image coordinate system using pre-calibrated camera parameters to obtain several sets of projected feature points. The nose tip features, eye features, and mouth features are then detected and extracted from the first projected image using a two-dimensional image algorithm to obtain the real feature points. Calculate the Euclidean distance between the projected feature points and the real feature points, and set the reprojection error. Determine if the Euclidean distance is less than the reprojection error. If it is, then it is a match; otherwise, it is a mismatch.
5. The robot face adaptive projection method based on real-time visual feedback according to claim 4, characterized in that, The consistency check of the three-dimensional model includes: The target 3D model is iteratively registered with the reference 3D model to obtain the optimal rigid body transformation. After ICP convergence, the average distance and Hausdorff distance between corresponding points in the two models are calculated, and the average distance threshold and Hausdorff distance threshold are set. If the average distance is less than or equal to the average distance threshold and the Hausdorff distance threshold is less than or equal to the Hausdorff distance threshold, then it is a match; otherwise, it is a mismatch.
6. The robot face adaptive projection method based on real-time visual feedback according to claim 5, characterized in that, The curvature attribute consistency detection includes: Calculate the first curvature value of any target feature among the nose tip feature, eye feature, and mouth feature in the baseline 3D model, and obtain the second curvature value of the target feature in the target 3D model; The curvature matching rate is calculated based on the first curvature value and the second curvature value, and a matching rate threshold is set. If the curvature matching rate is less than the matching rate threshold, no match is made; if it is greater than or equal to the matching rate threshold, a match is made.
7. A robot face adaptive projection system based on real-time visual feedback, used to execute the robot face adaptive projection method based on real-time visual feedback as described in any one of claims 1-6, characterized in that, include: The image processing module is configured to acquire the raw image data to be projected, project the raw image data onto the robot's face, acquire the first projected image, preprocess the first projected image, and obtain a RAW image. The process involves constructing a 3D digital twin model of the current RAW image, defining the target 3D model, calibrating the geometric parameters based on the target 3D model, outputting an ordered point cloud, and triangulating it to generate a mesh surface. Calculate the curvature properties of the vertices of each grid in the grid surface to obtain a global curvature map. Based on the global curvature map, extract facial features to obtain the first semantic network label composed of each feature combination. The feedback module is configured to perform matching detection on the first semantic network label. If a match is found, no processing is performed. If no match is found, the desired image is obtained. Based on the desired image, the projection parameters are obtained inversely through a 3D digital twin model. The projection is performed based on the projection parameters, and the second projection image of the current projection is obtained. The second semantic network label is obtained by extracting facial features from the second projection image. The second semantic network label is then matched. If a match is found, the current projection parameters are saved. If no match is found, the projection parameters are iteratively adjusted, and an iteration limit is set. When the iteration limit is reached, the original image data is projected with an error.