Face information recognition optimization method and system

Abstract

The application provides a face information recognition optimization method and system, and relates to the technical field of computer vision, and comprises the following steps: obtaining a face video of a target person and video editing content input by a user; processing the face video through a video depth estimator to obtain a depth map corresponding to each frame of the face video; mapping the face video and the depth map to a camera coordinate system to obtain a dynamic point cloud of the target person; projecting the dynamic point cloud back to a camera plane through a perspective projection function to obtain a video-mask pair composed of a rendered video and a visibility mask; and processing the video-mask pair through a pre-trained video repair model based on the video editing content to obtain a 4D face video of the target person. The video repair model is trained through a composite mask, the occlusion repair function and the editing repair function of the face of the target person are realized, and the face recognition accuracy of the target person is greatly improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a method and system for optimizing facial information recognition. Background Technology

[0002] With the development of artificial intelligence and deep learning technologies, facial expression recognition is becoming increasingly important in the field of medical diagnosis. Especially for patients who cannot express themselves actively (such as ICU patients), automated facial expression analysis can provide doctors with crucial diagnostic support.

[0003] Existing facial recognition methods typically involve fixing a detection camera directly above the patient or installing it on a mobile robot in an intensive care unit to obtain the patient's facial information.

[0004] However, this method, which relies on a fixed camera, creates blind spots when the patient rotates their head, making it unable to effectively capture the patient's facial information. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] To address the shortcomings of existing technologies, this invention provides an optimized method and system for facial information recognition, which solves the problem that traditional detection methods cannot effectively capture blind spots in the detection of patients' facial information.

[0007] (II) Technical Solution

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] In a first aspect, the present invention provides a facial information recognition optimization method, comprising:

[0010] Acquire facial videos of the target person and video editing content input by the user;

[0011] The facial video is processed by a video depth estimator to obtain a depth map corresponding to each frame of the facial video;

[0012] The facial video and the depth map are mapped to the camera coordinate system to obtain the dynamic point cloud of the target person;

[0013] The dynamic point cloud is projected back onto the camera plane using a perspective projection function to obtain a video-mask pair consisting of a rendered video and a visibility mask.

[0014] Based on the video editing content, the video-mask pair is processed by a pre-trained video inpainting model to obtain a 4D facial video of the target person; wherein, the video inpainting model is constructed based on a diffusion model and is obtained by adaptive iterative optimization training using a composite mask as a supervision signal, and the composite mask includes a point cloud mask and an edit mask.

[0015] Preferably, the step of processing the video-mask pair using a pre-trained video inpainting model based on the video editing content to obtain the 4D facial video of the target person includes:

[0016] Based on the video editing content, video frames and video regions are selected to obtain the corresponding selected frames and editing regions;

[0017] The edited area in the selected frame is edited based on the video editing content, and the edited selected frame is used as a guide frame to guide the video repair model to generate video for subsequent frames.

[0018] Preferably, the method for constructing the point cloud mask includes:

[0019] Obtain the training video input by the user;

[0020] The training video is processed to obtain a training video-mask pair consisting of a training rendered video and a training visibility mask;

[0021] The training rendered video is inversely projected to obtain the training dynamic point cloud;

[0022] By applying inverse transform to process the training dynamic point cloud, a new video and point cloud mask are obtained.

[0023] Preferably, the adaptive iterative tuning includes:

[0024] Step S210: Based on the training video-mask pair, the video inpainting model is fine-tuned once using the low-rank adaptive technique LORA to obtain the updated low-rank parameters;

[0025] Step S220: Based on the low-rank parameters and the video inpainting model, process the training video-mask pairs to obtain a 4D video;

[0026] Step S230: Increase the camera viewpoint corresponding to the 4D video to obtain a new training camera viewpoint, and infer a new training video based on the new training camera viewpoint.

[0027] Step S240: Process the new training video to obtain a new training video-mask pair;

[0028] Step S250: Repeat steps S210-S240 until the new training camera view reaches the preset training camera view, then stop the adaptive iterative tuning and use the updated low-rank parameters and video inpainting model as the trained video inpainting model.

[0029] Preferably, the adaptive iterative tuning includes:

[0030]

[0031]

[0032]

[0033] Where, ψ( ) represents the perspective projection function used to extrapolate the viewpoint; η Indicates the learning rate; This represents the mean square error loss for spatial-temporal consistency. N Indicates the number of frames in the video; i Indicates the selected video frame; P Represents a point cloud sequence; T This represents the extrinsic parameter matrix of the camera; K This represents the intrinsic parameter matrix of the camera; This represents the rendered video frame obtained through the perspective projection function; express j The rendered video obtained in each iteration; j Indicates the corresponding iteration cycle number; M express j Mask in round iteration; This indicates a low-rank parameter update; This represents the LORA weight.

[0034] Preferably, after processing the training video-mask pairs based on the low-rank parameters and the video inpainting model to obtain the 4D video, the method further includes temporal inference packaging, the temporal inference packaging step including:

[0035] Calculate the patched area of ​​the 4D video and extract selected frames; obtain a new rendered video through inference from the new camera perspective;

[0036] The selected frames and the new rendered video are encoded using a pre-trained 3D-VAE to obtain selected frame tokens and hollow video tokens;

[0037] The selected frame tokens and the holed video tokens are concatenated along the time dimension to obtain the joint input X of the video inpainting model. input .

[0038] Preferably, the method further includes using the ArcFace module and the Face Encoder module to verify the identity consistency between the facial video and the 4D facial video.

[0039] Secondly, the present invention also provides a facial information recognition optimization system, comprising:

[0040] The data acquisition module acquires facial videos of the target person and video editing content input by the user;

[0041] The first processing module processes the facial video using a video depth estimator to obtain a depth map corresponding to each frame of the facial video.

[0042] The second processing module maps the facial video and the depth map to the camera coordinate system to obtain the dynamic point cloud of the target person;

[0043] The third processing module projects the dynamic point cloud back to the camera plane using a perspective projection function to obtain a video-mask pair consisting of a rendered video and a visibility mask.

[0044] The video inpainting module processes the video-mask pair based on the video editing content using a pre-trained video inpainting model to obtain a 4D facial video of the target person. The video inpainting model is constructed based on a diffusion model and trained using a composite mask as a supervision signal through adaptive iterative tuning. The composite mask includes a point cloud mask and an edit mask.

[0045] Thirdly, the present invention also provides a computer-readable storage medium storing a computer program for extracting features of radar health status parameters, wherein the computer program causes a computer to execute the facial information recognition optimization method as described above.

[0046] Fourthly, the present invention also provides an electronic device, comprising:

[0047] One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing facial information recognition optimization methods as described above.

[0048] (III) Beneficial Effects

[0049] This invention provides a facial information recognition optimization method and system. Compared with the prior art, it has the following advantages:

[0050] This invention trains an initial video inpainting model based on a diffusion model using point cloud masks and editing masks, enabling the trained model to perform both occlusion repair and editing repair. It captures facial video of the target person using a preset camera coordinate system, based on camera trajectory and camera viewpoint. The facial video is then processed using a video depth estimator and perspective projection function to obtain a video-mask pair consisting of a rendered video and a visibility mask. Inputting this video-mask pair into the trained video inpainting model yields the repaired facial video of the target person. The video inpainting model used in this invention employs a Flux inpainting model combined with depth estimation technology, effectively repairing facial occlusions caused by patients wearing masks, tape, or other devices. This ensures that the repaired facial expression is natural and refined, especially in areas with significant occlusion (such as the eyes, nose, and lips), where it can accurately complete the repair. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 A flowchart illustrating a facial information recognition optimization method provided in an embodiment of this application;

[0053] Figure 2 A schematic diagram of a facial information recognition optimization system provided in this application embodiment;

[0054] Figure 3 This is a schematic diagram of the framework of a facial information recognition optimization method provided in an embodiment of this application. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] This application provides a facial information recognition optimization method and system, which constructs a novel framework that can simultaneously generate and edit 4D content from monocular video input. It solves the problem that traditional detection methods cannot effectively capture the blind spots in the detection of patients' facial information, and realizes accurate facial recognition when the target person's face is obscured due to head posture adjustment.

[0057] The technical solution in this application is to solve the above-mentioned technical problems, and the general idea is as follows:

[0058] While existing facial reconstruction models can effectively handle small-scale occlusions, their results are often unsatisfactory when dealing with large-scale facial occlusions (such as patients wearing masks or tape). The reconstructed facial features (such as eyes and lips) exhibit unnatural stitching marks, affecting the realism and accuracy of the reconstructed video. Secondly, existing facial reconstruction techniques often fail to ensure facial identity consistency when performing large-scale dynamic video reconstruction. Because video reconstruction models typically focus only on restoring video content and lack dedicated processing of facial identity information, facial identity information can be lost during the reconstruction process, especially during large-angle camera movements or changes in facial expressions, affecting subsequent facial expression recognition and analysis. Furthermore, existing technologies lack comprehensive consideration of video reconstruction and perspective adjustment. This method, relying on static camera motion paths to acquire input images, cannot handle real-time head shifts, and the temporal consistency issues arising from dynamic adjustments to the camera motion path remain unresolved.

[0059] This application employs a Flux inpainting model combined with depth estimation technology to effectively repair facial occlusions caused by patients wearing masks, tape, or other devices, ensuring natural and refined facial expressions after inpainting, especially in areas with significant occlusion (such as eyes, nose, and lips). This application introduces a Face Encoder module, which, through interaction with the overall image layout during the inpainting process, enhances the correlation between facial features and the background, thereby ensuring consistency of identity after facial inpainting. This module ensures that facial features remain consistent across multiple time frames, avoiding the identity loss problem caused by distortion during the inpainting process in existing technologies. This application employs a self-iterative optimization strategy to dynamically adjust the camera angle based on the patient's head movement, ensuring that the patient's face is always directly in front of the video. This technology allows for accurate facial image inpainting even if the patient's head shifts during acquisition, maintaining the naturalness and consistency of the video. This application effectively solves the temporal inconsistency problem caused by large-angle camera movement or multiple viewpoint changes by introducing temporal packing inference and a self-iterative optimization strategy. By progressively optimizing the camera angle and combining it with data generated in the previous stage, smooth transitions and consistency between different viewpoints and time frames in the video are ensured. This application combines video inpainting with multi-view consistency, ensuring that during multi-view video generation, not only are occluded areas repaired, but content consistency between different viewpoints is also maintained. Through temporal optimization and viewpoint alignment strategies, visual misalignment and inconsistency caused by viewpoint changes in existing methods are avoided.

[0060] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0061] like Figure 1 As shown in the figure, this application provides a facial information recognition optimization method, the specific implementation steps of which include:

[0062] Step S110: Obtain the facial video of the target person and the video editing content input by the user.

[0063] Step S120: Process the facial video using a video depth estimator to obtain the depth map corresponding to each frame of the facial video.

[0064] Step S130: Map the facial video and depth map to the camera coordinate system to obtain the dynamic point cloud of the target person.

[0065] Step S140: Project the dynamic point cloud back to the camera plane using the perspective projection function to obtain a video-mask pair consisting of the rendered video and the visibility mask.

[0066] Step S150: Based on the video editing content, the video-mask pair is processed by a pre-trained video inpainting model to obtain a 4D facial video of the target person; wherein, the video inpainting model is constructed based on a diffusion model and trained by adaptive iterative optimization using a composite mask as a supervision signal, and the composite mask includes a point cloud mask and an edit mask.

[0067] The facial information recognition optimization method provided in this application uses a Flux repair model (i.e., a trained target video repair model) to accurately repair the parts of a patient's face that are obscured by equipment such as belts, masks, and tape, ensuring that the repaired facial expressions are more natural and refined, and improving the realism of the repair effect.

[0068] In one embodiment, step S110 involves acquiring a facial video of the target person and video editing content input by the user.

[0069] In this embodiment, it is understood that the facial video of the target person can be captured in real time by controlling the camera trajectory input by the user, or the video recorded by surveillance cameras, etc. can be extracted.

[0070] It should be clarified that the camera's own imaging characteristics (focal length, pixel size, principal point offset, etc.) are adjusted through camera intrinsic parameters, while the camera angle and camera trajectory are adjusted through camera extrinsic parameters. In this embodiment, the camera's intrinsic parameter matrix is ​​used... K This indicates that the camera's extrinsic parameter matrix is ​​represented by... Optionally, this embodiment obtains the complete camera trajectory by pre-setting the camera's intrinsic and extrinsic parameter matrices, and captures facial video of the target person based on the camera trajectory. Among them, facial video The expression is as follows:

[0071]

[0072] in, N Indicates the total number of frames. I Represents an independent image of a specific frame. H Indicates the height of the image. W Indicates the width of the image. R The height of the image is represented by 3, and the number of channels in the image in this embodiment is represented by 3.

[0073] In one embodiment, step S120 involves processing the facial video using a video depth estimator to obtain a depth map corresponding to each frame of the facial video.

[0074] Specifically, a video depth estimator is used to estimate the depth map of each frame in the facial video. .

[0075] In one embodiment, step S130 involves mapping the facial video and depth map to the camera coordinate system to obtain the dynamic point cloud of the target person.

[0076] Specifically, video frames are combined with their corresponding depth maps to obtain a dynamic point cloud sequence. ;in, The calculation method is as follows:

[0077]

[0078] in, Let be a mapping function, representing the mapping function. 3D point cloud mapped to the camera coordinate system; K This represents the intrinsic parameter matrix of the camera; Represents an image frame; This represents the depth map corresponding to the image frame.

[0079] In one embodiment, step S140 involves projecting the dynamic point cloud back onto the camera plane using a perspective projection function to obtain a video-mask pair consisting of a rendered video and a visibility mask.

[0080] Specifically, the dynamic point cloud is projected back to the camera coordinate system using a perspective projection function to obtain the rendered video. V′ A video-mask pair consisting of a visibility mask and a video mask.

[0081] This application embodiment uses an external parameter matrix sequence. To provide motion trajectories for the camera, based on the extrinsic parameter matrix, the perspective function ψ is used to project the point cloud from each frame to the camera plane and render the image, resulting in a rendered frame. Combine all the rendered frames to obtain the rendered video of the target character. V′ The expression is:

[0082]

[0083] in, This represents the rendered frame obtained from the point cloud sequence; ψ Represents perspective functions; Tabular point cloud; K This represents the intrinsic parameter matrix of the camera; This represents the extrinsic parameter matrix of the camera.

[0084] Since a single monocular depth map is insufficient to reconstruct the entire scene, resulting in occluded or missing areas, the rendering result is usually incomplete. Therefore, during the rendering process, missing areas are identified, and a binary visibility mask is generated. The video inpainting model fills in the masked areas, thereby generating a complete 4D video; where the visibility mask... Represented as:

[0085]

[0086] In this context, pixels that are effectively projected are marked as 1, while areas that extend beyond the original view due to camera movement are marked as 0.

[0087] In one embodiment, step S150 involves processing the video-mask pair using a pre-trained video inpainting model based on the video editing content to obtain a 4D facial video of the target person; wherein the video inpainting model is constructed based on a diffusion model and trained by adaptive iterative optimization using a composite mask as a supervision signal, and the composite mask includes a point cloud mask and an edit mask.

[0088] In this embodiment, the aim is to obtain a complete 4D video by filling in the generated visibility mask using a video inpainting model. Based on the trained video inpainting model, a diffusion inpainting process is used to render the video. V′ Using the corresponding occlusion mask as input, the repaired video is obtained.

[0089] It should be clarified that the video to be repaired in this embodiment is a 4D video. The video repair model obtained through adaptive iterative tuning and training can obtain multiple perspectives of the repaired video and has extremely high spatial perception capabilities.

[0090] Understandably, during the training phase, when a composite mask is used as a supervisory signal, it can be one or more combinations of point cloud masks and edit masks.

[0091] In this embodiment, an edit mask is used to train the video inpainting model, enabling the target video inpainting model to have content editing capabilities. During training, a region of the input video is randomly selected and a corresponding mask sequence is generated. The mask of the first frame is set to '0', indicating that the first frame serves as a guide for video synthesis. During inference, content editing is performed by modifying the first frame, and then the changes are propagated to subsequent frames. The video inpainting model trained in this way possesses editing capabilities. In this embodiment, the specific implementation steps include:

[0092] Step S151: Select video frames and video regions based on the video editing content to obtain the corresponding selected frames and editing regions.

[0093] Step S152: Based on the video editing content, edit the selected frame to select the editing area, and use the edited selected frame as a guide frame to guide the video repair model to generate video for subsequent frames.

[0094] Based on pre-entered video editing content by the user, such as removing the patient's mask, the editing information is transmitted to the video repair model via UMT5. See [link / reference]. Figure 3 The video repair model is processed through steps S151-S152 to obtain a 4D facial video with the target person's mask removed.

[0095] Next, we will introduce the training phase of the video inpainting model in detail. First, we will introduce the generation of the point cloud mask, and the specific generation steps are as follows:

[0096] Step S310: Obtain the training video input by the user.

[0097] Step S320: Process the training video to obtain a training video-mask pair consisting of a training rendered video and a training visibility mask.

[0098] Step S330: Inverse project the training rendered video to obtain the training dynamic point cloud.

[0099] Step S340: The dynamic point cloud is trained by applying inverse transform to obtain a new video and point cloud mask.

[0100] In this embodiment, since the rendered frame does not contain the real content of the occluded area, it is very difficult to supervise the training of the model using the original visibility mask. To solve this problem, this embodiment of the application adopts a dual reprojection strategy to reproject the visibility mask back to the viewpoint of the input video, thereby obtaining a supervision mask that is aligned with the original frame space, and the input video can be used as real data for training and repairing the model.

[0101] Specifically, the rendered video V′ Through depth map D′Back projection onto a new dynamic point cloud P′， And by applying inverse transform Re-render the view to obtain a pair containing the new rendered video. V′′ and a mask indicating the artifact region M′′ The dataset, and the corresponding training videos input by users. Following the same original camera trajectory, M′′ This refers to the point cloud mask. The point cloud mask is used as a supervision signal, and the training video is used as the ground truth to train the video inpainting model until a fully trained video inpainting model is obtained.

[0102] It should be clarified that in this embodiment, the consistency loss between the training video and the output video is measured by a loss function.

[0103] This application also includes adaptive iterative optimization, and the specific implementation steps include:

[0104] Step S210: Based on the training video-mask pair, the video inpainting model is fine-tuned once using the low-rank adaptive technique LORA to obtain the updated low-rank parameters.

[0105] Step S220: Process the training video-mask pairs based on low-rank parameters and the video inpainting model to obtain the 4D video.

[0106] Step S230: Add the camera viewpoint corresponding to the 4D video to obtain a new training camera viewpoint, and infer a new training video based on the new training camera viewpoint.

[0107] Step S240: Process the new training video to obtain a new training video-mask pair.

[0108] Step S250: Repeat steps S210-S240 until the new training camera view reaches the preset training camera view, then stop the adaptive iterative tuning and use the updated low-rank parameters and video inpainting model as the trained video inpainting model.

[0109] In this embodiment, it is understood that the training video input by the user is a video generated by a preset small-angle (exemplary, <30°) camera. In this embodiment, the angle is gradually increased through an angle progressive method, and an iteration is performed for each increase in angle until the preset maximum angle is reached (exemplary, 90°).

[0110] Conventional video inpainting diffusion models face several challenges when handling videos with holes at large angles. Firstly, the fine-tuning performed on individual videos and the use of various masks significantly limit the model's generalization ability. Secondly, when generating large-angle scene reconstructions, video inpainting methods struggle to maintain temporal consistency between frames; this lack of 3D perception greatly affects the inpainting effect. Therefore, to generate videos with content at larger angles, this application proposes a self-iterative optimization approach, gradually increasing the angular range of the masked videos to improve the 3D generation capability of the video inpainting model.

[0111] Specifically, obtaining monocular video Subsequently, multiple video-mask pairs are generated using a narrower perspective; for example, this application uses a 20-degree perspective to generate multiple video-mask pairs. The video-mask is used to fine-tune the low-rank adaptation (LoRA) video inpainting model in one step. After fine-tuning, the LoRA weights are loaded. And infer the video with small angles that have been trained. The specific formula for parameter optimization is as follows:

[0112]

[0113] in, This indicates a low-rank parameter update.

[0114] Then, an iterative approach was used, specifically, at the... j In the next iteration, a new masked video with a wider angular range is generated; then, the model executes a self-iterative process through recursion. In this embodiment, adaptive iterative tuning includes:

[0115]

[0116]

[0117]

[0118] Where, ψ( ) represents the perspective projection function used to extrapolate the viewpoint; η Indicates the learning rate; This represents the mean square error loss for spatial-temporal consistency. N Indicates the number of frames in the video; i Indicates the selected video frame; P Represents a point cloud sequence; T This represents the extrinsic parameter matrix of the camera; K This represents the intrinsic parameter matrix of the camera; This represents the rendered video frame obtained through the perspective projection function; express j The rendered video obtained in each iteration; j Indicates the corresponding iteration cycle number; M express j Mask in round iteration; This indicates a low-rank parameter update; This represents the LORA weight.

[0119] In this embodiment, to maintain consistency between the subject and scene in the generated multi-view video, the video inpainting model performs temporal inference packaging.

[0120] Specifically, after inferring the 4D video corresponding to the training camera's viewpoint based on low-rank parameters and the video inpainting model, temporal inference and packaging are performed. The temporal inference and packaging steps include:

[0121] Step S410: Calculate the patched area of ​​the 4D video and extract the selected frames; obtain a new rendered video through inference from the new camera perspective.

[0122] Step S420: Select frames and new rendered video are selected using pre-trained 3D-VAE encoding to obtain selected frame tokens and holed video tokens.

[0123] Step S430: Concatenate the selected frame tokens and the holed video token along the time dimension to obtain the joint input X of the video inpainting model. input .

[0124] Understandably, after iteratively optimizing the video inpainting model, existing video inpainting techniques are prone to producing inconsistent results between different time frames due to changes in camera perspective. This results in a lack of smooth transitions in the generated video across multiple perspectives, affecting video quality. In dynamic scenes, existing methods cannot effectively handle smooth transitions between multiple perspectives, and the inpainted video may exhibit disjointed and abrupt frames, impacting the viewing experience. To address the temporal consistency issue, this application's embodiments employ a time-packing strategy to maintain consistency across multi-view videos. Specifically, camera trajectories are obtained. Generate video using target video inpainting model Calculate video In each frame, the patch area is selected and then... k A frame, expressed by the formula:

[0125]

[0126] Where S( ) represents a region calculation function; Represents the final set of selected sample frames; top-k-argmax( () indicates selecting the highest score. k Each frame; Indicates video The corresponding mask.

[0127] In the subsequent camera trajectory In reasoning, the chosen k Frames tokens (i.e., the basic processing unit after image / video segmentation) and hollow video token (i.e., image patches) are stitched together along the time dimension, expressed as:

[0128]

[0129] in, It is the input to the video inpainting model, and E( ) indicates a pre-trained 3D-VAE; Indicates based on camera trajectory The video was obtained through reasoning.

[0130] It is important to emphasize that in the target video inpainting model, self-attention is globally applied to all tokens in the spatial-temporal attention layer, eliminating redundant architecture and reducing computational complexity.

[0131] The facial learning and recognition optimization method provided in this application realizes the recognition of the patient's facial information through different camera trajectories and edited first frames, while using a time packing strategy to maintain the consistency of multi-view videos.

[0132] In one embodiment, the method further includes using the ArcFace module and the Face Encoder module to verify the identity consistency between the facial video and the 4D facial video. See also Figure 3 In this embodiment of the application, an ArcFace module and a Face Encoder module are added to verify the identity consistency between the input video and the output video.

[0133] like Figure 3As shown in the figure, the facial information recognition optimization method provided by this invention trains an initial video inpainting model based on a diffusion model using point cloud masks and editing masks, enabling the trained video inpainting model to have occlusion repair and editing repair functions. In actual operation, a facial video of the target person is captured using a preset camera trajectory and camera viewpoint; then, the facial video is processed by a video depth estimator to obtain a depth map corresponding to the facial video; then, the facial video and depth map are mapped to the camera coordinate system to obtain a dynamic point cloud of the target person; then, the dynamic point cloud is projected back to the camera plane using a perspective projection function to obtain a video-mask pair consisting of a rendered video and a visibility mask; then, a pre-trained 3D-VAE module performs temporal packing inference on the video-mask pair to ensure the consistency of multi-view videos; subsequently, the data is input into the trained video inpainting model, and the user can input editing information into the UMT5 module to realize the model's editing function. For example, facial pain assessment indicators of ICU patients can be input to determine the user's pain standard based on the patient's facial expression in order to implement a better treatment plan. The video repair model used in this invention employs a Flux repair model combined with depth estimation technology, which can effectively repair facial occlusions caused by patients wearing masks, tape, or other devices, ensuring that the repaired facial expressions are natural and detailed, especially in areas with large occlusions (such as eyes, nose, and lips), where it can accurately complete the repair.

[0134] like Figure 2 As shown, this embodiment of the invention also provides a facial information recognition optimization system, including:

[0135] The data acquisition module acquires facial videos of the target person and video editing content input by the user.

[0136] The first processing module processes the facial video using a video depth estimator to obtain a depth map corresponding to each frame of the facial video.

[0137] The second processing module maps the facial video and depth map to the camera coordinate system to obtain the dynamic point cloud of the target person.

[0138] The third processing module projects the dynamic point cloud back to the camera plane using a perspective projection function, resulting in a video-mask pair consisting of the rendered video and the visibility mask.

[0139] The video inpainting module processes video-mask pairs based on the video editing content using a pre-trained video inpainting model to obtain a 4D facial video of the target person. The video inpainting model is constructed based on a diffusion model and trained using a composite mask as a supervision signal through adaptive iterative optimization. The composite mask includes a point cloud mask and an edit mask.

[0140] It is understood that the facial information recognition optimization system provided in this embodiment of the invention corresponds to the facial information recognition optimization method described above. The explanations, examples, and beneficial effects of the relevant content can be used to train the corresponding content in the facial information recognition optimization method, and will not be repeated here.

[0141] This invention also provides a computer-readable storage medium storing a computer program for a facial information recognition optimization method, wherein the computer program causes a computer to execute the facial information recognition optimization method as described above.

[0142] This application also provides an electronic device, including: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing the facial information recognition optimization method as described above.

[0143] In summary, compared with existing technologies, it has the following beneficial effects:

[0144] 1. The embodiments of the present invention adopt a diffusion repair model combined with depth estimation technology, which can effectively repair facial occlusion caused by patients wearing masks, tape and other devices, ensuring that the repaired facial expression is natural and delicate, especially in areas with large occlusion (such as eyes, nose and lips) that can be accurately repaired.

[0145] 2. This invention employs a self-iterative optimization strategy to dynamically adjust the virtual camera's perspective based on the patient's head displacement, ensuring the patient's face remains directly in front of the video. This technology allows for accurate facial image repair even if the patient's head shifts during acquisition, maintaining the video's naturalness and consistency.

[0146] 3. This invention introduces a temporal packing inference and self-iterative optimization strategy, which effectively solves the temporal inconsistency problem caused by large-angle camera movement or multiple viewpoint changes. By progressively optimizing the camera viewpoint and combining it with the data generated in the previous stage, a smooth transition and consistency between different viewpoints and time frames in the video are ensured.

[0147] 4. This invention combines video patching with multi-view consistency, ensuring that during multi-view video generation, not only are occluded areas patched, but content consistency between different viewpoints is also maintained. Through temporal optimization and viewpoint alignment strategies, visual misalignment and inconsistency caused by viewpoint changes in existing methods are avoided.

[0148] 5. This invention introduces a Face Encoder module, which enhances the correlation between facial features and the background through interaction with the overall image layout during the repair process, thereby ensuring the consistency of identity after facial repair. This module ensures that facial features remain consistent across multiple time frames, avoiding the identity loss problem caused by distortion during the repair process in existing technologies.

[0149] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0150] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A facial information recognition optimization method, characterized in that, The method includes: Acquire facial videos of the target person and video editing content input by the user; The facial video is processed by a video depth estimator to obtain a depth map corresponding to each frame of the facial video; The facial video and the depth map are mapped to the camera coordinate system to obtain the dynamic point cloud of the target person; The dynamic point cloud is projected back onto the camera plane using a perspective projection function to obtain a video-mask pair consisting of a rendered video and a visibility mask. Based on the video editing content, the video-mask pair is processed by a pre-trained video inpainting model to obtain a 4D facial video of the target person; wherein, the video inpainting model is constructed based on a diffusion model and trained through adaptive iterative optimization using a composite mask as a supervision signal, the composite mask including a point cloud mask and an edit mask; the video inpainting model includes temporal inference packing, the temporal inference packing step including: Calculate the patched area of ​​the 4D video and extract selected frames; infer a new rendered video through the new camera perspective; The selected frames and the new rendered video are encoded using a pre-trained 3D-VAE to obtain selected frame tokens and hollow video tokens; The selected frame tokens and the holed video tokens are concatenated along the time dimension to obtain the joint input Xinput of the video inpainting model.

2. The facial information recognition optimization method according to claim 1, characterized in that, The process of processing the video-mask pair using a pre-trained video inpainting model based on the video editing content to obtain a 4D facial video of the target person includes: Based on the video editing content, video frames and video regions are selected to obtain the corresponding selected frames and editing regions; The edited area in the selected frame is edited based on the video editing content, and the edited selected frame is used as a guide frame to guide the video repair model to generate video for subsequent frames.

3. The facial information recognition optimization method according to claim 1, characterized in that, The method for constructing the point cloud mask includes: Obtain the training video input by the user; The training video is processed to obtain a training video-mask pair consisting of a training rendered video and a training visibility mask; The training rendered video is inversely projected to obtain the training dynamic point cloud; By applying inverse transform to process the training dynamic point cloud, a new video and point cloud mask are obtained.

4. The facial information recognition optimization method according to claim 3, characterized in that, The adaptive iterative tuning includes: Step S210: Based on the training video-mask pair, the video inpainting model is fine-tuned once using the low-rank adaptive technique LORA to obtain the updated low-rank parameters; Step S220: Based on the low-rank parameters and the video inpainting model, process the training video-mask pairs to obtain a 4D video; Step S230: Increase the camera viewpoint corresponding to the 4D video to obtain a new training camera viewpoint, and infer a new training video based on the new training camera viewpoint. Step S240: Process the new training video to obtain a new training video-mask pair; Step S250: Repeat steps S210-S240 until the new training camera view reaches the preset training camera view, then stop the adaptive iterative tuning, and use the updated low-rank parameters and video inpainting model as the trained video inpainting model.

5. The facial information recognition optimization method according to claim 4, characterized in that, The adaptive iterative tuning includes: Where, ψ( ) represents the perspective projection function used to extrapolate the viewpoint; η Indicates the learning rate; This represents the mean square error loss for spatial-temporal consistency. N Indicates the number of frames in the video; i Indicates the selected video frame; P Represents a point cloud sequence; T This represents the extrinsic parameter matrix of the camera; K This represents the intrinsic parameter matrix of the camera; This represents the rendered video frame obtained through the perspective projection function; express j The rendered video obtained in each iteration; j Indicates the corresponding iteration cycle number; M express j Mask in round iteration; This indicates a low-rank parameter update; This indicates the LORA weight.

6. The facial information recognition optimization method according to claim 1, characterized in that, The method also includes using the ArcFace module and the Face Encoder module to verify the identity consistency between the facial video and the 4D facial video.

7. A facial information recognition optimization system, applied to the facial information recognition optimization method as described in claim 1, characterized in that, include: The data acquisition module acquires facial videos of the target person and video editing content input by the user; The first processing module processes the facial video using a video depth estimator to obtain a depth map corresponding to each frame of the facial video. The second processing module maps the facial video and the depth map to the camera coordinate system to obtain the dynamic point cloud of the target person; The third processing module projects the dynamic point cloud back to the camera plane using a perspective projection function to obtain a video-mask pair consisting of a rendered video and a visibility mask. The video inpainting module processes the video-mask pair based on the video editing content using a pre-trained video inpainting model to obtain a 4D facial video of the target person. The video inpainting model is constructed based on a diffusion model and trained using a composite mask as a supervision signal through adaptive iterative tuning. The composite mask includes a point cloud mask and an edit mask.

8. A computer-readable storage medium, characterized in that, It stores a computer program for optimizing facial information recognition, wherein the computer program causes a computer to execute a facial information recognition optimization method as described in any one of claims 1 to 6.

9. An electronic device, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing a facial information recognition optimization method as described in any one of claims 1 to 6.

Images