Facial image restoration processing method and device, storage medium and program product

Abstract

The invention provides a face image restoration processing method and device, a storage medium and a program product. The method comprises the following steps: acquiring a to-be-detected video frame; performing shielding detection processing on the to-be-detected video frame to obtain a shielding detection result; generating a vector field thermodynamic diagram by adopting a preset fusion algorithm of an optical flow displacement field and an image displacement gradient according to the to-be-detected video frame and the shielding region information; obtaining a preset geometry and texture double-constraint matching model; wherein the preset geometry and texture double-constraint matching model is obtained based on training of a preset space-time memory library; according to a preset geometry and texture double-constraint matching model, performing matching processing on the vector field thermodynamic diagram to obtain face patch features; and performing image restoration processing according to the face patch features and the vector field thermodynamic diagram to obtain a target restoration image corresponding to the shielding region information. The accuracy of face shielding repair under the dynamic condition can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, device, storage medium, and program product for repairing facial images. Background Technology

[0002] Dynamic facial occlusion detection has always been one of the challenges in implementing face recognition and tracking applications. The purpose of dynamic facial occlusion detection is to track, match, and repair occlusions of faces appearing in videos, thereby reducing mismatches or mismatches that occur when face detection and face recognition algorithms track moving targets.

[0003] Currently, facial dynamic occlusion detection typically employs single-frame analysis technology, utilizing single-frame generative adversarial networks to repair occluded areas, and improving the realism of the repair through a local discriminator.

[0004] However, existing technologies have limitations, resulting in low accuracy in facial occlusion repair under dynamic conditions. Summary of the Invention

[0005] This application provides a method, device, storage medium, and program product for repairing facial images, in order to solve the problem that the existing technology has limitations, resulting in low accuracy of facial occlusion repair under dynamic conditions.

[0006] In a first aspect, this application provides a method for restoring facial images, comprising:

[0007] Acquire the video frames to be detected;

[0008] The video frame to be detected is subjected to occlusion detection processing to obtain occlusion detection results; wherein, the occlusion detection results include occlusion area information;

[0009] Based on the video frame to be detected and the occlusion area information, a vector field heat map is generated using a preset fusion algorithm of optical flow displacement field and image displacement gradient.

[0010] Obtain a preset geometric and texture dual-constraint matching model; wherein the preset geometric and texture dual-constraint matching model is trained based on a preset spatiotemporal memory bank, and the preset spatiotemporal memory bank is used to store historical frames and facial features in the historical frames;

[0011] According to the preset geometric and texture dual-constraint matching model, the vector field heatmap is matched to obtain face patch features;

[0012] Based on the face patch features and the vector field heatmap, image restoration processing is performed to obtain the target restored image corresponding to the occluded area information.

[0013] In one possible implementation, the step of generating a vector field heatmap based on the video frame to be detected and the occlusion region information using a preset fusion algorithm of optical flow displacement field and image displacement gradient includes: determining the corresponding continuous frame sequence based on the video frame to be detected; calculating the displacement vector of the occlusion object using a sparse iterative optical flow method based on the continuous frame sequence and the occlusion region information; calculating the differential gradient of the occlusion region information and constructing a displacement gradient field based on the differential gradient; determining a first weight and a second weight for gradient optical flow fusion; performing gradient optical flow fusion calculation processing based on the first weight, the displacement vector of the occlusion object, the second weight, and the displacement gradient field to obtain fused vector information; and generating a vector field heatmap based on the fused vector information.

[0014] In one possible implementation, determining the first and second weights for gradient optical flow fusion includes: determining the inter-frame luminance variance based on the continuous frame sequence; determining the blurriness based on the video frame to be detected; and determining the first and second weights for gradient optical flow fusion based on the inter-frame luminance variance and the blurriness.

[0015] In one possible implementation, generating a vector field heatmap based on the fused vector information includes: performing facial key point topology tracking and sparse optical flow chain tracking processing based on the fused vector information to obtain the vector field heatmap.

[0016] In one possible implementation, the occlusion detection result further includes information on the non-occluded region; correspondingly, after performing occlusion detection processing on the video frame to be detected to obtain the occlusion detection result, the method further includes: performing multimodal feature extraction processing and expression encoding processing based on the video frame to be detected and the information on the non-occluded region to obtain multimodal feature data and expression encoding feature data; and updating the preset spatiotemporal memory based on the multimodal feature data and the expression encoding feature data.

[0017] In one possible implementation, before obtaining the preset geometric and texture dual-constraint matching model, the method further includes: acquiring historical video data including complete facial information; extracting multiple historical frames based on the historical video data; performing feature extraction processing on the historical frames to obtain facial features; wherein the facial features include multiple of timestamp index, key point coordinates, texture histogram, expression encoding, and time decay weight; and storing the facial features and the corresponding historical frames in the preset spatiotemporal memory.

[0018] In one possible implementation, the step of matching the vector field heatmap according to the preset geometric and texture dual-constraint matching model to obtain face patch features includes: performing geometric consistency screening on the vector field heatmap in the preset spatiotemporal memory according to the preset geometric and texture dual-constraint matching model to obtain a first matching history frame; performing texture similarity screening on the vector field heatmap in the preset spatiotemporal memory according to the preset geometric and texture dual-constraint matching model to obtain a second matching history frame; and determining face patch features based on the first matching history frame and the second matching history frame.

[0019] In one possible implementation, the step of performing image inpainting processing based on the face patch features and the vector field heatmap to obtain a target repaired image corresponding to the occlusion area information includes: determining a cycle consistency constraint loss parameter for image inpainting processing based on the face patch features, the vector field heatmap, and a preset cycle consistency constraint loss function; and performing image inpainting processing on the vector field heatmap based on the cycle consistency constraint loss parameter to obtain a target repaired image corresponding to the occlusion area information.

[0020] Secondly, this application provides a facial image restoration processing apparatus, comprising:

[0021] The first acquisition module is used to acquire the video frames to be detected;

[0022] The detection module is used to perform occlusion detection processing on the video frame to be detected in order to obtain occlusion detection results; wherein, the occlusion detection results include occlusion area information;

[0023] The generation module is used to generate a vector field heatmap based on the video frame to be detected and the occlusion area information, using a preset fusion algorithm of optical flow displacement field and image displacement gradient.

[0024] The second acquisition module is used to acquire a preset geometric and texture dual-constraint matching model; wherein, the preset geometric and texture dual-constraint matching model is trained based on a preset spatiotemporal memory bank, and the preset spatiotemporal memory bank is used to store historical frames and facial features in the historical frames;

[0025] The first processing module is used to perform matching processing on the vector field heatmap according to the preset geometric and texture dual-constraint matching model to obtain face patch features;

[0026] The second processing module is used to perform image restoration processing based on the face patch features and the vector field heatmap to obtain the target restored image corresponding to the occlusion area information.

[0027] Thirdly, this application provides a facial image restoration processing device, including: a memory and a processor;

[0028] The memory stores computer-executed instructions;

[0029] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0030] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible embodiments of the first aspect.

[0031] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0032] The facial image restoration processing method, device, storage medium, and program product provided in this application obtain a geometric and texture dual-constraint matching model by training based on a spatiotemporal memory; using the geometric and texture dual-constraint matching model, combined with the spatiotemporal memory, the vector field heatmap is matched to obtain facial patch features; based on the facial patch features and the vector field heatmap, image restoration processing is performed to obtain the target restored image corresponding to the occlusion area information. This can improve the accuracy of facial occlusion restoration under dynamic conditions. Attached Figure Description

[0033] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0034] Figure 1 A schematic diagram illustrating an application scenario of the facial image restoration processing method provided in this application embodiment;

[0035] Figure 2 A schematic flowchart illustrating a facial image restoration method provided in one embodiment of this application;

[0036] Figure 3 A schematic diagram of another facial image restoration method provided in an embodiment of this application;

[0037] Figure 4 A schematic diagram illustrating a method for generating a vector field heat map provided in an embodiment of this application;

[0038] Figure 5 A schematic diagram of the structure of the facial image restoration processing device provided in the embodiments of this application;

[0039] Figure 6 A schematic diagram of the structure of the facial image restoration processing device provided in the embodiments of this application.

[0040] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0041] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0042] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0043] Currently, facial dynamic occlusion detection typically employs single-frame analysis techniques, utilizing single-frame generative adversarial networks to repair occluded areas and employing local discriminators to improve the realism of the repair. However, existing occlusion detection and repair technologies rely on single-frame information, resulting in low accuracy in facial occlusion repair in dynamic occlusion scenarios.

[0044] The facial image restoration method provided in this application embodiment generates a vector field heatmap based on a preset fusion algorithm of optical flow displacement field and image displacement gradient. It obtains face patch features through a geometric and texture dual-constraint matching model trained by a spatiotemporal memory bank. The target restoration image is obtained through the face patch features and the vector field heatmap, which can improve the accuracy of facial occlusion restoration under dynamic conditions.

[0045] Figure 1 This is a schematic diagram illustrating an application scenario of the facial image restoration processing method provided in the embodiments of this application, such as... Figure 1 As shown, the scene is a computer device, including: a receiving device 101, a processor 102, and a display device 103.

[0046] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the item recognition method. In other feasible embodiments of this application, the above architecture may include more or fewer components than illustrated, or combine some components, or split some components, or arrange different components, which can be determined according to the actual application scenario and is not limited here. Figure 1 The components shown can be implemented in hardware, software, or a combination of both.

[0047] In the specific implementation process, the receiving device 101 can be an input / output interface or a communication interface, and can acquire the video frame to be detected.

[0048] The processor 102 can process the video frame to be detected to determine the target restoration image corresponding to the occlusion area information.

[0049] The display device 103 can be used to display the target repair image, etc., corresponding to the above-mentioned occluded area information.

[0050] The display device can also be a touch screen, used to receive user commands while displaying the above content, so as to achieve interaction with the user.

[0051] It should be understood that the aforementioned processor can be implemented by reading instructions from memory and executing those instructions, or it can be implemented through chip circuitry.

[0052] Furthermore, the network architecture and business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0053] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0054] Figure 2 This is a schematic flowchart illustrating a facial image restoration method provided in one embodiment of this application. The execution entity of this embodiment can be... Figure 1 The computer equipment shown is not specifically limited in this embodiment. Figure 2 As shown, the method includes:

[0055] S201: Obtain the video frame to be detected.

[0056] S202: Perform occlusion detection processing on the video frame to be detected to obtain occlusion detection results; wherein, the occlusion detection results include occlusion area information.

[0057] Specifically, the video frame to be detected is input into a pre-trained target detection model to output the face bounding box in the video frame; the face bounding box is used to determine whether the face is occluded; if the face is determined to be occluded, the occlusion area information is obtained; and the occlusion detection result is generated based on the occlusion area information.

[0058] Optionally, key points are extracted from the video frame to be detected to obtain facial key points; occlusion areas are detected based on the facial key points to determine whether occlusion has occurred; if occlusion is determined to have occurred, occlusion area information is generated to obtain occlusion detection results.

[0059] Optionally, the occlusion detection result also includes information on the non-occluded area; correspondingly, after step S202, the method further includes: performing multimodal feature extraction and expression encoding processing based on the video frame to be detected and the information on the non-occluded area to obtain multimodal feature data and expression encoding feature data; and updating the preset spatiotemporal memory based on the multimodal feature data and expression encoding feature data.

[0060] By performing multimodal feature extraction and facial expression encoding on the video frames to be detected, and updating the preset spatiotemporal memory, the accuracy of the preset geometric and texture dual-constraint matching model can be improved, thereby improving the accuracy of facial occlusion repair under dynamic conditions.

[0061] Optionally, when performing multimodal feature extraction and expression encoding processing based on the video frame to be detected and the information of the non-occluded area to obtain multimodal feature data and expression encoding feature data, the process includes: obtaining information of the non-occluded area; if it is determined that the area of ​​the non-occluded area information meets a preset area threshold, then performing multimodal feature extraction and expression encoding processing on the video frame to be detected to obtain multimodal feature data and expression encoding feature data.

[0062] The preset area threshold can be 95%, and the area of ​​the unobstructed area information can satisfy the preset area threshold as the unobstructed area area is greater than or equal to 95%.

[0063] Optionally, when performing multimodal feature extraction and expression coding processing on the video frame to be detected to obtain multimodal feature data and expression coding feature data, the process includes: extracting local binary pattern histograms from the video frame to be detected to obtain multimodal feature data; and using a preset lightweight convolutional ensemble neural network to extract micro-expression features from the video frame to be detected to obtain expression coding feature data.

[0064] Optionally, when updating the preset spatiotemporal memory based on multimodal feature data and facial expression coding feature data, the process includes: fusing multimodal feature data and facial expression coding feature data to obtain a fingerprint-like feature vector of the video frame to be detected; and updating the preset spatiotemporal memory based on the fingerprint-like feature vector.

[0065] Optionally, when updating the preset spatiotemporal memory based on the fingerprint feature vector, the process includes: adding dynamic decay weights to the fingerprint feature vector using a preset weighting algorithm to obtain the facial features of the video frame to be detected; and saving the facial features and the corresponding video frame to be detected to the preset spatiotemporal memory.

[0066] Among them, facial features include multiple elements such as timestamp index, key point coordinates, texture histogram, expression encoding, and time decay weights.

[0067] Among them, the texture histogram can be a local binary pattern histogram.

[0068] Optionally, the preset weight algorithm is calculated using the following formula:

[0069]

[0070] In the formula, W t λ represents the dynamic decay weight; λ represents the time decay coefficient, which can take a value of 0.2; t ref Indicates the timestamp index of the reference frame; t cur Represents the timestamp index of the current frame; SSIM represents the structural similarity index, which can be calculated using a window, with a window size of 11x11; I ref Indicates reference frame; I cur Indicates the current frame.

[0071] It should be noted that the dynamic decay weight of facial features in frames further away from the current time is lower (following the exponential decay law), while the dynamic decay weight of facial features in frames with lower structural similarity index (i.e., SSIM) with the current frame will be suppressed.

[0072] It should be noted that the preset spatiotemporal memory capacity is limited to 8 frames. When a new frame is added, the frame with the lowest dynamic decay weight is automatically discarded. The purpose of this is to selectively forget secondary information. This avoids storage bloat while ensuring that the preset spatiotemporal memory retains the feature fragments most relevant to the current scene, laying the foundation for accurate subsequent repair.

[0073] Optionally, when extracting local binary pattern histograms from the video frame to be detected to obtain multimodal feature data, the process includes: acquiring a face image of the video frame to be detected; dividing the face image into a preset number of grids; for the image data of each grid, using a preset local binary pattern operator to extract texture features from the image data to obtain a local binary pattern feature vector; obtaining a local binary pattern feature map based on the local binary pattern feature vectors of multiple grids, wherein the local binary pattern feature map contains multiple grids; performing histogram transformation on each grid of the local binary pattern feature map and then concatenating them to obtain a local binary pattern histogram; and determining multimodal feature data based on the local binary pattern histogram.

[0074] Optionally, the preset number of grids can be 8x8. When extracting texture features from image data, a sliding window can be used with 3x3 neighborhoods as units.

[0075] Optionally, the formula for calculating the local binary pattern feature vector is:

[0076]

[0077] In the formula, LBP U Represents the local binary pattern feature vector; M is the number of neighborhoods; u represents the neighborhood index; g u g represents the grayscale value of the pixel at the u-th neighborhood location; c This represents the pixel grayscale value at the center of the window.

[0078] It should be noted that the local binary pattern feature vector can be an 8-bit binary code, reflecting the gradient changes around the reference pixel. The local binary pattern histogram can be obtained by concatenating eight 32-dimensional vectors, which can be a 256-bit vector.

[0079] Optionally, the preset lightweight convolutional ensemble neural network structure can be: input → 3x3 convolutional kernel (256 channels) → activation function → max pooling → 3x3 convolutional kernel (128 channels) → activation function → fully connected layer → 64-dimensional output.

[0080] Optionally, the preset lightweight convolutional ensemble neural network can be trained by distilling facial features extracted from a pre-trained convolutional neural network architecture as input data, which can effectively reduce network size and computational cost.

[0081] The pre-trained convolutional neural network architecture can be a pre-trained ResNet or a pre-trained VGG.

[0082] S203: Based on the video frame to be detected and the occlusion area information, a vector field heat map is generated using a preset fusion algorithm of optical flow displacement field and image displacement gradient.

[0083] Optionally, based on the video frames to be detected, a corresponding continuous frame sequence is determined; using the sparse iterative optical flow method, the displacement vector of the occlusion object is calculated based on the continuous frame sequence and the occlusion region information; the differential gradient of the occlusion region information is calculated, and a displacement gradient field is constructed based on the differential gradient; the first weight and the second weight of gradient optical flow fusion are determined; gradient optical flow fusion calculation is performed based on the first weight, the displacement vector of the occlusion object, the second weight, and the displacement gradient field to obtain the fused vector information; and a vector field heatmap is generated based on the fused vector information.

[0084] The displacement vector of the occlusion is calculated by sparse iterative optical flow method; the displacement gradient field is constructed by differential gradient of occlusion area information; and a vector field heat map is generated based on the first weight, the displacement vector of the occlusion, the second weight and the displacement gradient field, which can improve the accuracy of the vector field heat map and thus improve the accuracy of facial occlusion repair under dynamic conditions.

[0085] Optionally, when calculating the displacement vector of the occluded object based on the continuous frame sequence and the occlusion area information, the method includes inputting the continuous frame sequence and the occlusion area information into a preset occlusion trajectory prediction network to output the displacement vector of the occluded object.

[0086] The preset occlusion trajectory prediction network can include 3 layers of convolutional and gated recurrent unit modules.

[0087] Optionally, when determining the corresponding continuous frame sequence based on the video frames to be detected, the process includes: determining the corresponding continuous frame sequence based on a preset number of video frames and the video frames to be detected.

[0088] Optionally, the formula for calculating the displacement vector of the obstruction is:

[0089]

[0090] In the formula, denoted by , A represents the displacement vector of the occluder; A represents the gradient matrix of the occluded region of the video frame to be detected; b represents the pixel-by-pixel grayscale difference vector between two adjacent video frames.

[0091] The formula for the gradient matrix of the occluded region of the video frame to be detected is:

[0092]

[0093] In the formula, I x (p i ) represents pixel p i The gray-level gradient along the x-axis, where i can take values ​​of 1, 2, 3, ..., n; y (p i ) represents pixel p iThe gray-level gradient along the y-axis, where i can take values ​​of 1, 2, 3, ..., n; x (p i ), I y (p i All of these can be calculated using gradient extraction algorithms.

[0094] The formula for the pixel-by-pixel grayscale difference vector between two adjacent video frames is:

[0095]

[0096] In the formula, I t (p i ) represents the gradient change between adjacent video frames, where i can take values ​​of 1, 2, 3, ..., n.

[0097] Optionally, when calculating the differential gradient of the occlusion region information, the process includes: obtaining an occlusion mask of the occlusion region information; and calculating the differential gradient of the occlusion mask to obtain the differential gradient of the occlusion region information.

[0098] Alternatively, the formula for the displacement gradient field is expressed as:

[0099]

[0100] In the formula, G disp M represents the displacement gradient field; occ Indicates a masking layer; The differential gradient representing the information of the occluded region; K sobel This represents the gradient extraction algorithm, specifically the Sobel operator. This represents the convolution operation.

[0101] Optionally, determining the first and second weights for gradient optical flow fusion includes: determining the inter-frame luminance variance based on a continuous frame sequence; determining the blurriness based on the video frame to be detected; and determining the first and second weights for gradient optical flow fusion based on the inter-frame luminance variance and the blurriness.

[0102] By dynamically assigning weights to inter-frame brightness variance and ambiguity, the accuracy of occlusion trajectory prediction can be improved, thereby enhancing the accuracy of facial occlusion repair under dynamic conditions.

[0103] Optionally, the formula for calculating the first weight is:

[0104]

[0105] Where α represents the first weight; σ lum σ represents the inter-frame luminance variance; blur Represented as ambiguity; σ lum and σ blurThe variance can be calculated using the Laplace operator.

[0106] Optionally, the formula for calculating the second weight is:

[0107] β=1-α

[0108] In the formula, β represents the second weight.

[0109] Optionally, the formula for calculating the fused vector information is:

[0110]

[0111] In the formula, V occ This represents the fused vector information; Represents the image gradient field; α·||G disp || represents the displacement gradient; This represents gradient optical flow.

[0112] It should be noted that when the illumination is stable, the displacement gradient is emphasized, and the first weight can be initially set to 0.7; when the motion is blurred, the gradient optical flow is emphasized, and the second weight can be initially set to 0.3.

[0113] Optionally, when generating a vector field heatmap based on the fused vector information, the process includes: performing facial key point topology tracking and sparse optical flow chain tracking based on the fused vector information to obtain the vector field heatmap.

[0114] By using topological tracing of facial key points and sparse optical flow chain tracing, the trajectory of each key point under the action of the vector field is calculated, which improves the accuracy of facial occlusion trajectory in the vector field heatmap, thereby improving the accuracy of facial occlusion repair under dynamic conditions.

[0115] Optionally, based on the fused vector information, face key point topology tracking is performed to obtain initial key point coordinates; based on the initial key point coordinates, sparse optical flow chain tracking is performed to obtain key point coordinate trajectories; and a vector field heatmap is generated based on the key point coordinate trajectories.

[0116] It should be noted that the initial keypoint coordinates can be represented as P0 = {p1, p2, ..., p...} n The value of n can be 10⁶ or 68, and this application does not impose any restrictions; it can be adjusted according to the actual situation.

[0117] Optionally, the formula for calculating the coordinate trajectory of key points is:

[0118]

[0119] In the formula, Indicates the coordinates of the i-th key point at the next moment; ω represents the coordinates of the i-th key point at the current moment; k δ represents the Gaussian weight coefficients of the neighborhood in the k-th direction. k This represents the neighborhood offset in the k-th direction; N represents the number of directions in the neighborhood.

[0120] It should be noted that the mesh for facial landmark topology tracking processing can be a 3×3 structure, that is, it has a neighborhood in 8 directions.

[0121] Optionally, the formula for the Gaussian weighting coefficients is:

[0122]

[0123] In the formula, σ represents the standard deviation of the Gaussian distribution, which can be 1.

[0124] It should be noted that in the vector field heatmap, the intensity of each point is the magnitude of the fused vector information, and the direction is the argument of the fused vector information, which prepares for subsequent image restoration processing.

[0125] S204: Obtain a preset geometric and texture dual-constraint matching model; wherein, the preset geometric and texture dual-constraint matching model is trained based on a preset spatiotemporal memory bank, which is used to store historical frames and facial features in historical frames.

[0126] Optionally, before step S204, the method further includes: acquiring historical video data including complete facial information; extracting multiple historical frames based on the historical video data; performing feature extraction processing on the historical frames to obtain facial features; wherein the facial features include multiple of timestamp index, key point coordinates, texture histogram, expression encoding, and time decay weight; and storing the facial features and the corresponding historical frames in a preset spatiotemporal memory.

[0127] Optionally, when performing feature extraction processing on historical frames to obtain facial features, the process includes: performing multimodal feature extraction processing and expression encoding processing on historical frames to obtain facial features.

[0128] By storing historical frames and facial features in a pre-defined spatiotemporal memory, a data foundation is provided for training a pre-defined geometric and texture dual-constraint matching model.

[0129] S205: Based on the preset geometric and texture dual-constraint matching model, the vector field heatmap is matched to obtain face patch features.

[0130] Optionally, based on a preset geometric and texture dual-constraint matching model, the vector field heatmap is subjected to geometric consistency screening in a preset spatiotemporal memory to obtain a first matching history frame; based on the preset geometric and texture dual-constraint matching model, the vector field heatmap is subjected to texture similarity screening in a preset spatiotemporal memory to obtain a second matching history frame; based on the first matching history frame and the second matching history frame, the face patch features are determined.

[0131] By performing geometric consistency screening and texture similarity screening on the vector field heatmap, the accuracy of face patch features can be improved, thereby enhancing the accuracy of facial occlusion repair under dynamic conditions. At the same time, it can reduce the amount of computation, save computing resources, and improve matching efficiency.

[0132] Optionally, the vector field heatmap is subjected to geometric consistency screening in a preset spatiotemporal memory to obtain the first matching history frame, including: performing geometric consistency screening on the vector field heatmap in a preset spatiotemporal memory, and selecting video frames whose average Euclidean distance between the facial key points in the video frame to be matched and the facial key points in the vector field heatmap is less than a preset matching distance as the first matching history frame.

[0133] The preset matching distance can be 5 pixels.

[0134] Optionally, the vector field heatmap is subjected to texture similarity filtering in a preset spatiotemporal memory to obtain a second matching history frame, including: obtaining the local binary pattern histogram of the video frame to be matched; calculating the local binary pattern histogram of the occlusion region of the video frame to be detected based on the vector field heatmap; calculating the intersection of the local binary pattern histogram of the video frame to be matched and the local binary pattern histogram of the occlusion region to determine the histogram similarity; retaining video frames whose histogram similarity exceeds a preset similarity threshold to obtain the second matching history frame.

[0135] The preset similarity threshold can be 85%.

[0136] Optionally, when performing matching processing on the vector field heatmap to obtain face patch features, the process includes: obtaining the attention weights of multiple video frames to be matched and the vector field heatmap in a preset spatiotemporal memory; selecting the video frame to be matched with the highest attention weight as the matching history frame; and performing weighted fusion on the matching history frames according to the attention weights corresponding to the matching history frames to obtain face patch features.

[0137] S206: Based on the face patch features and vector field heatmap, perform image inpainting processing to obtain the target inpainted image corresponding to the occluded area information.

[0138] Optionally, based on the face patch features, the vector field heatmap, and the preset cyclic consistency constraint loss function, the cyclic consistency constraint loss parameters for image inpainting are determined; the vector field heatmap is then processed for image inpainting based on the cyclic consistency constraint loss parameters to obtain the target inpainted image corresponding to the occlusion area information.

[0139] By using a preset cycle consistency constraint loss function to determine the cycle consistency constraint loss parameters, the efficiency and accuracy of image restoration processing can be improved.

[0140] Optionally, a preset image fusion technique is used to perform pixel fusion based on the face patch features and the occlusion area information of the video frame to be detected in the vector field heatmap, so as to obtain an initial repair image.

[0141] The preset image fusion technique can be Poisson fusion.

[0142] Optionally, the initial restoration image is determined as the current frame restoration image; the next frame restoration image is obtained; the forward deformed image of the current frame is obtained based on the current frame restoration image, the forward optical flow vector information, and the optical flow deformation function; the reverse deformed image of the next frame is obtained based on the next frame restoration image, the forward optical flow vector information, and the optical flow deformation function; the cyclic consistency constraint loss parameters used for image restoration processing are determined based on the forward deformed image of the current frame, the reverse deformed image of the next frame, and the preset cyclic consistency constraint loss function; and the target restoration image corresponding to the occlusion area information is obtained based on the cyclic consistency constraint loss parameters.

[0143] Among them, the forward optical flow vector information can be calculated using the calculation formula of the fused vector information.

[0144] Optionally, the optical flow deformation function can be obtained by bilinear interpolation through surrounding neighboring pixels. The surrounding neighborhood can consist of four pixels.

[0145] Optionally, when obtaining the reverse deformed image of the next frame based on the next frame repaired image, the forward optical flow vector information, and the optical flow deformation function, the process includes: obtaining the reverse optical flow vector information based on the forward optical flow vector information; and inputting the next frame repaired image and the reverse optical flow vector information into the optical flow deformation function to obtain the reverse deformed image of the next frame.

[0146] Optionally, the formula for the optical flow deformation function is:

[0147]

[0148] In the formula, The output after the optical flow deformation operation is represented; x and y represent the coordinates of key points in the input restored image; x′ and y′ represent the coordinates of key points in the deformed restored image; I represents the input restored image; V represents the input vector information, which can be forward optical flow vector information or reverse optical flow vector information; i and j represent the coordinates of neighboring pixels around the key point coordinates in the deformed restored image, and the number of neighbors can be 4; I(i,j) represents the pixel value of the neighboring pixel (i,j); This indicates rounding down x′ and y′. This indicates that x′ and y′ are rounded up.

[0149] Optionally, the formula for the preset cycle consistency constraint loss function is:

[0150]

[0151] In the formula, This represents the loss parameter for cycle consistency constraints; Indicates the image to be repaired in the current frame; Indicates the reverse-distorted image of the next frame; Indicates the next frame of the repaired image; This represents the forward-distorted image of the current frame; This represents the reverse consistency loss; This represents the forward consistency loss.

[0152] The formula for calculating the forward deformed image of the current frame is:

[0153]

[0154] In the formula, V t→t+1 This represents the forward optical flow vector information.

[0155] The formula for calculating the inversely deformed image of the next frame is as follows:

[0156]

[0157] In the formula, V t+1→t This represents the reverse optical flow vector information.

[0158] Optionally, when performing image inpainting processing on the vector field heatmap based on the cyclic consistency constraint loss parameter to obtain the target inpainted image corresponding to the occlusion region information, the process includes: setting the current frame inpainted image and the next frame inpainted image as optimizable variables; obtaining a preset optimizer; performing image deformation processing on the current frame inpainted image and the next frame inpainted image based on the forward optical flow vector information and the reverse optical flow vector information, respectively, to obtain the forward deformed image of the current frame and the reverse deformed image of the next frame; calculating the forward consistency loss and the reverse consistency loss based on the current frame inpainted image, the next frame inpainted image, the forward deformed image of the current frame, the reverse deformed image of the next frame, and the preset cyclic consistency constraint loss function to obtain the cyclic consistency constraint loss parameter; using the preset optimizer to calculate the first optimization gradient and the second optimization gradient of the cyclic consistency constraint loss parameter on the current frame inpainted image and the next frame inpainted image; using the preset optimizer to optimize the current frame inpainted image and the next frame inpainted image based on the preset learning rate, the first optimization gradient, and the second optimization gradient; and performing iterative optimization based on the preset number of iterations to determine the target inpainted image corresponding to the current frame inpainted image and the target inpainted image corresponding to the next frame inpainted image.

[0159] Optionally, the first optimization gradient can be the derivative of the cycle consistency constraint loss parameter with respect to the repaired image in the current frame; the second optimization gradient can be the derivative of the cycle consistency constraint loss parameter with respect to the repaired image in the next frame.

[0160] Optionally, the formula for optimizing the repaired image of the current frame is: Optimized image of the current frame = Repaired image of the current frame - Preset learning rate * First optimization gradient.

[0161] Optionally, the formula for optimizing the next frame of the repaired image is: Optimized image of the next frame = Repaired image of the next frame - Preset learning rate * Second optimization gradient.

[0162] The facial image restoration method provided in this application involves training a geometric and texture dual-constraint matching model based on a spatiotemporal memory. Using this model, combined with the spatiotemporal memory, a vector field heatmap is matched to obtain facial patch features. Based on these features and the heatmap, image restoration processing is performed to obtain the target restored image corresponding to the occlusion area information. This method improves the accuracy of facial occlusion restoration under dynamic conditions.

[0163] This application provides a facial image restoration system, including: occlusion propagation vector field modeling, a spatiotemporal memory module, and a cross-frame adaptive restoration engine. The occlusion propagation vector field modeling is used to analyze the motion trajectory of occluders and generate a vector field heatmap. The spatiotemporal memory module includes a preset spatiotemporal memory for storing complete facial features from historical frames. The cross-frame adaptive restoration engine matches facial patch features from the spatiotemporal memory module to perform image restoration processing, obtaining a target restored image corresponding to the occluded region information. The cross-frame adaptive restoration engine includes a face correlation analysis decision module, used to analyze whether the target restored image meets preset restoration requirements and generate a face correlation analysis decision, which includes the target restored image and the restored image confidence level.

[0164] Figure 3 A schematic diagram of another facial image restoration method provided in this application embodiment, as shown below. Figure 3 As shown, the method includes: acquiring a video frame to be detected; extracting key points from the video frame to be detected to obtain facial key points; and detecting occlusion areas based on the facial key points to determine whether occlusion has occurred.

[0165] If occlusion is detected, occlusion region information is generated. Based on the video frame to be detected and the occlusion region information, a vector field heatmap is generated. A pre-defined geometric and texture dual-constraint matching model is used, combined with the vector field heatmap, to match in a pre-defined spatiotemporal memory to obtain face patch features. Poisson fusion technology is used to perform image inpainting based on the face patch features to obtain an initial inpainted image. The initial inpainted image is iteratively optimized based on a pre-defined cyclic consistency constraint loss function to obtain the target inpainted image corresponding to the occlusion region information. Face correlation analysis decision processing is performed on the target inpainted image to obtain the face correlation analysis decision.

[0166] Optionally, when generating a vector field heatmap based on the video frame to be detected and the occlusion area information, the process includes: analyzing the displacement trend of the occlusion area between adjacent frames based on the video frame to be detected and the occlusion area information, and generating a vector field heatmap with directional arrows and confidence color temperature.

[0167] If no occlusion is detected, facial features of the video frame to be detected are extracted and stored in a preset spatiotemporal memory. The preset spatiotemporal memory continuously updates dynamic decay weights to ensure that the newest and most relevant features are called first.

[0168] The facial image restoration method provided in this application embodiment achieves closed-loop operation on the end-side device, while saving computing resources and improving detection and restoration efficiency.

[0169] Figure 4A schematic diagram of a method for generating a vector field heat map provided in an embodiment of this application is shown below. Figure 4 As shown, the method includes: determining the corresponding continuous frame sequence based on the video frame to be detected; using the sparse iterative optical flow method, calculating the displacement vector of the occlusion object based on the continuous frame sequence and occlusion region information; calculating the differential gradient of the occlusion region information, and constructing a displacement gradient field based on the differential gradient; determining the first weight and the second weight for gradient optical flow fusion; performing gradient optical flow fusion calculation processing based on the first weight, the occlusion object displacement vector, the second weight, and the displacement gradient field to obtain the fused vector information; and performing face keypoint topology tracking processing and sparse optical flow chain tracking processing based on the fused vector information to obtain a vector field heatmap.

[0170] It should be noted that when the ambient lighting is stable, the first weight is set to 70%, which depends on the displacement gradient and focuses more on the movement of the occluder itself; while when the motion blur is intensified, the second weight is set to 30%, which depends on the gradient optical flow and uses the texture changes of the overall scene to help correct the trajectory.

[0171] The method for generating a vector field heatmap in this embodiment calculates the displacement vector of the occluder using a sparse iterative optical flow method; constructs a displacement gradient field using the differential gradient of the occluded region information; and obtains fused vector information based on the first weight, the displacement vector of the occluder, the second weight, and the displacement gradient field; and performs topological tracking of facial key points and sparse optical flow chain tracking based on the fused vector information to generate a vector field heatmap. This method improves the accuracy of the vector field heatmap, thereby improving the accuracy of facial occlusion repair under dynamic conditions.

[0172] Figure 5 This is a schematic diagram of the facial image restoration processing device provided in the embodiments of this application, as shown below. Figure 5 As shown, the facial image restoration processing device provided in this embodiment includes: a first acquisition module 501, a detection module 502, a generation module 503, a second acquisition module 504, a first processing module 505, and a second processing module 506.

[0173] The first acquisition module 501 is used to acquire the video frame to be detected;

[0174] The detection module 502 is used to perform occlusion detection processing on the video frame to be detected in order to obtain occlusion detection results; wherein, the occlusion detection results include occlusion area information;

[0175] The generation module 503 is used to generate a vector field heat map based on the video frame to be detected and the occlusion area information, using a preset fusion algorithm of optical flow displacement field and image displacement gradient.

[0176] The second acquisition module 504 is used to acquire a preset geometric and texture dual-constraint matching model; wherein, the preset geometric and texture dual-constraint matching model is trained based on a preset spatiotemporal memory bank, and the preset spatiotemporal memory bank is used to store historical frames and facial features in historical frames;

[0177] The first processing module 505 is used to perform matching processing on the vector field heat map according to the preset geometric and texture dual-constraint matching model to obtain face patch features;

[0178] The second processing module 506 is used to perform image restoration processing based on face patch features and vector field heatmap to obtain the target restored image corresponding to the occluded area information.

[0179] In one possible implementation, the generation module 503 is specifically used for: determining the corresponding continuous frame sequence based on the video frame to be detected; calculating the displacement vector of the occlusion object using the sparse iterative optical flow method based on the continuous frame sequence and occlusion region information; calculating the differential gradient of the occlusion region information and constructing a displacement gradient field based on the differential gradient; determining the first weight and the second weight for gradient optical flow fusion; performing gradient optical flow fusion calculation processing based on the first weight, the displacement vector of the occlusion object, the second weight, and the displacement gradient field to obtain the fused vector information; and generating a vector field heatmap based on the fused vector information.

[0180] In one possible implementation, the generation module 503, when determining the first and second weights for gradient optical flow fusion, is specifically used to: determine the inter-frame luminance variance based on a continuous frame sequence; determine the blurriness based on the video frame to be detected; and determine the first and second weights for gradient optical flow fusion based on the inter-frame luminance variance and the blurriness.

[0181] In one possible implementation, the generation module 503, when generating a vector field heatmap based on the fused vector information, is specifically used to: perform facial key point topology tracking processing and sparse optical flow chain tracking processing based on the fused vector information to obtain the vector field heatmap.

[0182] In one possible implementation, the occlusion detection result also includes information on the non-occluded area; correspondingly, the facial image restoration processing device further includes: an update module, used to perform multimodal feature extraction processing and expression encoding processing based on the video frame to be detected and the non-occluded area information, so as to obtain multimodal feature data and expression encoding feature data; and to update a preset spatiotemporal memory bank based on the multimodal feature data and expression encoding feature data.

[0183] In one possible implementation, the update module is further configured to: acquire historical video data including complete facial information; extract multiple historical frames based on the historical video data; perform feature extraction processing on the historical frames to obtain facial features; wherein the facial features include multiple of timestamp index, key point coordinates, texture histogram, expression encoding and time decay weight; and store the facial features and the corresponding historical frames in a preset spatiotemporal memory.

[0184] In one possible implementation, the second acquisition module 504 is specifically used to: perform geometric consistency screening on the vector field heatmap in a preset spatiotemporal memory based on a preset geometric and texture dual-constraint matching model to obtain a first matching history frame; perform texture similarity screening on the vector field heatmap in a preset spatiotemporal memory based on the preset geometric and texture dual-constraint matching model to obtain a second matching history frame; and determine the face patch features based on the first matching history frame and the second matching history frame.

[0185] In one possible implementation, the second processing module 506 is specifically used to: determine the cyclic consistency constraint loss parameters for image inpainting based on the face patch features, the vector field heatmap, and the preset cyclic consistency constraint loss function; and perform image inpainting processing on the vector field heatmap based on the cyclic consistency constraint loss parameters to obtain the target inpainted image corresponding to the occlusion area information.

[0186] The facial image restoration processing device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0187] Figure 6 This is a schematic diagram of the structure of the facial image restoration processing device provided in an embodiment of this application. Figure 6 As shown, the facial image restoration processing device provided in this embodiment includes at least one processor 601 and a memory 602. Optionally, the device further includes a communication component 603. The processor 601, memory 602, and communication component 603 are connected via a bus 604.

[0188] In a specific implementation, at least one processor 601 executes computer execution instructions stored in memory 602, causing at least one processor 601 to perform the above-described method.

[0189] The specific implementation process of processor 601 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0190] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0191] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0192] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0193] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0194] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0195] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0196] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0197] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0198] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0199] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0200] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0201] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0202] Finally, it should be noted that other embodiments of this application will readily conceive of by those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and alterations may be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for restoring facial images, characterized in that, include: Acquire the video frames to be detected; The video frame to be detected is subjected to occlusion detection processing to obtain occlusion detection results; wherein, the occlusion detection results include occlusion area information; Based on the video frame to be detected and the occlusion area information, a vector field heat map is generated using a preset fusion algorithm of optical flow displacement field and image displacement gradient. Obtain a preset geometric and texture dual-constraint matching model; wherein the preset geometric and texture dual-constraint matching model is trained based on a preset spatiotemporal memory bank, and the preset spatiotemporal memory bank is used to store historical frames and facial features in the historical frames; According to the preset geometric and texture dual-constraint matching model, the vector field heatmap is matched to obtain face patch features; Based on the face patch features and the vector field heatmap, image restoration processing is performed to obtain the target restored image corresponding to the occluded area information.

2. The method according to claim 1, characterized in that, The step of generating a vector field heatmap based on the video frame to be detected and the occlusion area information, using a preset fusion algorithm of optical flow displacement field and image displacement gradient, includes: Based on the video frames to be detected, determine the corresponding consecutive frame sequence; The sparse iterative optical flow method is used to calculate the displacement vector of the occlusion object based on the continuous frame sequence and the occlusion region information; Calculate the differential gradient of the occlusion region information, and construct a displacement gradient field based on the differential gradient; Determine the first and second weights for gradient optical flow fusion; Gradient optical flow fusion calculation is performed based on the first weight, the displacement vector of the occluder, the second weight, and the displacement gradient field to obtain the fused vector information. Based on the fused vector information, a vector field heat map is generated.

3. The method according to claim 2, characterized in that, The determination of the first and second weights for gradient optical flow fusion includes: Based on the continuous frame sequence, determine the inter-frame luminance variance; Determine the blurriness based on the video frame to be detected; Based on the inter-frame luminance variance and the blurriness, the first and second weights for gradient optical flow fusion are determined.

4. The method according to claim 2, characterized in that, The step of generating a vector field heatmap based on the fused vector information includes: Based on the fused vector information, topological tracking of facial key points and sparse optical flow chain tracking are performed to obtain a vector field heatmap.

5. The method according to any one of claims 1 to 4, characterized in that, The occlusion detection results also include information on unoccluded areas; Accordingly, after performing occlusion detection processing on the video frame to be detected to obtain the occlusion detection result, the method further includes: Based on the video frame to be detected and the information of the non-occluded area, multimodal feature extraction and expression encoding are performed to obtain multimodal feature data and expression encoding feature data. The preset spatiotemporal memory is updated based on the multimodal feature data and the facial expression encoding feature data.

6. The method according to any one of claims 1 to 4, characterized in that, Before obtaining the preset geometric and texture dual-constraint matching model, the following steps are also included: Obtain historical video data including complete facial information; Based on the historical video data, extract multiple historical frames; Feature extraction processing is performed on the historical frames to obtain facial features; wherein, the facial features include multiple of the following: timestamp index, key point coordinates, texture histogram, expression encoding, and time decay weight; The facial features and corresponding historical frames are stored in the preset spatiotemporal memory.

7. The method according to claim 6, characterized in that, The step of matching the vector field heatmap according to the preset geometric and texture dual-constraint matching model to obtain face patch features includes: According to the preset geometric and texture dual-constraint matching model, the vector field heatmap is subjected to geometric consistency screening in the preset spatiotemporal memory to obtain the first matching history frame; Based on the preset geometric and texture dual-constraint matching model, the vector field heatmap is subjected to texture similarity screening in the preset spatiotemporal memory to obtain the second matching history frame; Based on the first and second matching history frames, the face patch features are determined.

8. The method according to claim 7, characterized in that, The step of performing image inpainting processing based on the face patch features and the vector field heatmap to obtain the target inpainted image corresponding to the occluded area information includes: Based on the face patch features, the vector field heatmap, and the preset cycle consistency constraint loss function, the cycle consistency constraint loss parameters used for image inpainting processing are determined. The vector field heatmap is inpainted using the cycle consistency constraint loss parameters to obtain the target repaired image corresponding to the occlusion area information.

9. A facial image restoration processing device, characterized in that, include: The first acquisition module is used to acquire the video frames to be detected; The detection module is used to perform occlusion detection processing on the video frame to be detected in order to obtain occlusion detection results; wherein, the occlusion detection results include occlusion area information; The generation module is used to generate a vector field heatmap based on the video frame to be detected and the occlusion area information, using a preset fusion algorithm of optical flow displacement field and image displacement gradient. The second acquisition module is used to acquire a preset geometric and texture dual-constraint matching model; wherein, the preset geometric and texture dual-constraint matching model is trained based on a preset spatiotemporal memory bank, and the preset spatiotemporal memory bank is used to store historical frames and facial features in the historical frames; The first processing module is used to perform matching processing on the vector field heatmap according to the preset geometric and texture dual-constraint matching model to obtain face patch features; The second processing module is used to perform image restoration processing based on the face patch features and the vector field heatmap to obtain the target restored image corresponding to the occlusion area information.

10. A facial image restoration processing device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-8.

12. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-8.

Images