Face occlusion image redrawing network method

Abstract

The invention relates to the technical field of image analysis, in particular to a facial occlusion image redrawing network method, which comprises the following steps: migrating the time sequence modeling advantage of RWKV to the field of image redrawing to realize semantic perception and identity consistency reconstruction of an occlusion area. Reconstructing a calculation normal form of a WKV attention mechanism and a token conversion layer, and decoupling recursive calculation based on time steps in RWKV into feature interaction of spatial dimensions, so that the feature interaction adapts to structural characteristics of a two-dimensional image; a window circulation WKV attention module is designed, multi-direction feature propagation is simulated through sub-window scanning and a bidirectional circulation mechanism, and local fine-grained texture and global structure constraint modeling are synchronously captured in a single-layer network; according to the method, a multi-scale token transfer layer is constructed, and the context dependency relationship of fine structures such as interbrow wrinkles and lip lines is re-calibrated and strengthened through cross-scale features, so that the problem of detail homogenization caused by occlusion is avoided, and the redrawing effect of a facial occlusion image is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image analysis, and in particular to a face occlusion image redrawing network method. BACKGROUND

[0002] With the development of deep learning, face redrawing technology has evolved from early methods based on traditional image processing to end-to-end reconstruction schemes based on deep neural networks. The current mainstream technology needs to cope with two major challenges of global degradation (low light, noise) and local occlusion. The loss of key identity features (eye shape, nose contour, lip line, etc.) caused by local occlusion is the core bottleneck that restricts the quality of redrawing. If the semantic association between the occluded area and the non-occluded area cannot be effectively established, it is easy to cause the reconstruction result to have problems such as structure rupture, identity deviation, or detail homogenization.

[0003] The existing technology for redrawing face occlusion images can be mainly divided into three categories, and the core structure and working mechanism of each type of technology are as follows: (1) Traditional convolutional network method, this method takes convolution operation as the core feature extraction unit, typical representatives include repair models based on gated convolution (Gated Convolution) (such as CMT, DRFNet), reconstruction networks based on multi-scale feature fusion (such as PFAN) and the like. The core structure follows the "encoder - decoder" architecture: The encoder extracts image features of different scales by stacking convolutional layers (including gated convolution, dilated convolution, etc.), capturing local texture and basic structure information; The decoder restores the image resolution through upsampling operations (such as pixel recombination, transposed convolution), combines the multi-scale features of the encoder through jump connection, and realizes the filling of the occluded area; Some models introduce feature aggregation modules (such as FSAB) or attention mechanisms (such as channel attention, spatial attention) to strengthen the weight distribution of key features.

[0004] (2) Transformer method, to solve the limitation of local receptive field of traditional convolutional network, the mainstream Transformer method (such as Spa-former, HAN) introduces self-attention mechanism into face redrawing task, the core structure characteristics are as follows: Adopt tokenization operation to convert image features into one-dimensional sequence, model global feature dependency relationship through self-attention mechanism, break through the limitation of local modeling of convolutional network; Combined with multi-head attention, position encoding and other modules, enhance the ability to capture long-distance semantic association; Some models adopt a U-Net-derived architecture, which balances the reconstruction accuracy of global structure and local details through encoder-decoder scale transformation and feature fusion.

[0005] (3) RWKV derivative methods: RWKV (accepting weighted key value mechanism) is an efficient linear attention model originally used for 1D time series processing (such as natural language processing). In recent years, some studies have attempted to transfer it to the image domain, forming two types of derivative schemes: Vision-RWKV: By adjusting the token arrangement, the spatial dimension of the adapted image is recursively calculated using the time steps of RWKV, and a four-way token movement mechanism is used to capture contextual information; Restore-RWKV: For image restoration tasks, a recurrent WKV (Re-WKV) attention mechanism is proposed, which expands the receptive field through bidirectional scanning and enhances the ability to model global dependencies.

[0006] In addition, existing technologies also include redrawing models based on generative adversarial networks (GANs) (such as MAT), which optimize the reconstruction details through adversarial training between the generator and the discriminator, but the core still relies on the feature modeling logic of the aforementioned convolution or attention mechanisms.

[0007] However, when using the above method, the following technical problems often arise: (1) Traditional convolutional network methods: insufficient long-range semantic association modeling.

[0008] The core issues are that the reconstruction results are prone to structural breaks at occlusion boundaries and homogenization of details. For example, the CMT model has difficulty distinguishing the identity features of different individuals in scenarios with central or large-scale occlusion, leading to a convergence of reconstructed image styles; DRFNet suffers from pixel blurring and artifacts, and its texture filling lacks realism.

[0009] The root cause is that the local receptive field characteristics and static weight mechanism of convolutional operations limit the capture of long-range semantic relationships between occluded and unoccluded regions. As a highly structured target, the human face exhibits strong correlations in the topological relationships of its organs such as the eyes, nose, and mouth. However, convolutional networks can only progressively transfer local features and cannot directly establish global constraints across occluded regions, resulting in the inability to effectively transfer key identity features to occluded areas.

[0010] (2) Transformer class methods: Imbalance between computational complexity and local fine modeling.

[0011] The core issues are low redrawing efficiency and insufficient refinement of occlusion boundaries in large-scale occlusion scenarios. For example, Spa-former has a PSNR of only 25.13dB when there is 40%-60% occlusion, and the facial key points are significantly offset (with a high NRMSE index); the HAN model is prone to facial structure distortion, which can have irreversible effects on identity information.

[0012] The root cause is that the Transformer's self-attention mechanism has a quadratic computational complexity (proportional to the square of the number of tokens), which limits the model's processing efficiency for high-resolution images and makes it difficult to simultaneously achieve global structure modeling and local detail capture in a single-layer network. At the same time, the unidirectional attention mechanism cannot fully adapt to the two-dimensional spatial characteristics of face images, resulting in unnatural feature transitions at occlusion boundaries and limited fine-grained reconstruction capabilities.

[0013] (3) RWKV derivative methods: Image domain adaptability and local dependency modeling defects.

[0014] The core problem is the inability to balance global structure and local details, resulting in insufficient completeness of facial topology reconstruction. For example, Vision-RWKV's four-way token movement mechanism only considers feature interactions in a limited number of directions, failing to fully utilize the spatial attributes of the face; while Restore-RWKV's recurrent WKV attention enhances global dependency modeling, it treats input features as a whole, ignoring the local correlations between different facial regions (such as between the eyebrows and around the lips), leading to insufficient accuracy in detail reconstruction.

[0015] The root cause is that the original RWKV was designed to process 1D time series, and the existing derivative models have not been deeply adapted to the structural characteristics of face images. On the one hand, the problem of adapting RWKV recursive computation to the spatial structure of 2D images has not been solved; on the other hand, the token moving mechanism and attention scanning method have not considered the heterogeneity of face regions (the contour and texture features of different regions are significantly different), resulting in an imbalance between global and local modeling.

[0016] (4) Common problems: poor adaptability to identity consistency and large-scale occlusion.

[0017] Existing models (such as MAT) are prone to identity shifts during redrawing, and may even alter background information outside the face region, leading to a significant drop in evaluation metrics (FID, SSIM). When faced with large-scale occlusion of 40%-60%, most models (such as PFAN and HAN) are unable to effectively extract key identity features from the unoccluded areas and transfer them to the occluded areas, resulting in structural distortion or loss of details.

[0018] The root cause is that existing technologies do not have a dedicated feature modeling module designed for the unique topological structure of the human face. They only use general feature extraction and fusion strategies, which cannot accurately capture the constraints of key organs such as the eyes, nose, and mouth, resulting in a lack of effective guidance for feature generation in occluded areas.

[0019] In summary, the technical problems of existing facial occlusion image redrawing technologies can be categorized into three points: First, the challenge of balancing long-range semantic association and local detail modeling; second, the insufficient adaptation of efficient attention mechanisms such as RWKV to the image domain; and third, the lack of dedicated modeling modules for structural facial features. These issues make it difficult for existing technologies to simultaneously meet the three core requirements of "structural integrity, identity consistency, and texture realism" in medium- to large-scale occlusion scenarios, limiting their deployment and application in real-world scenarios, thus resulting in poor facial occlusion image redrawing performance. Summary of the Invention

[0020] To address the technical problem of poor image redrawing results for facial occlusion images, this invention proposes a network method for redrawing facial occlusion images.

[0021] This invention provides a network method for redrawing facial occlusion images, the method comprising: S1. Select an unobstructed real RGB image and generate a random binary mask; multiply the unobstructed real RGB image and the random binary mask point by point to obtain the occluded image, and then concatenate the occluded image and the random binary mask to obtain the input tensor; S2, input the input tensor obtained in step S1 into the encoder, and extract 256×256, 128×128 and 64×64 resolution feature maps in sequence through 5 cascaded gated convolutional layers; output the final feature map of the encoder, and temporarily store the multi-scale feature maps of the first 4 gated convolutional layers. S3. The final feature map obtained in step S2 is input into the structure estimation module based on the U-Net architecture. Cross-scale feature transfer is achieved through downsampling, upsampling, and skip connections. The core completes the structure modeling through a 7-layer W-RWKV module. Within the module, the MT-Shift mechanism is first used to perform multi-scale token interaction and fusion on the features. Then, the WR-WKV attention mechanism is used to simultaneously capture global structural constraints and local texture dependencies. Finally, a coarsely repaired facial structure feature map is output. S4, input the roughly repaired facial structure feature map and the multi-scale feature map temporarily stored in step S2 into the texture refinement module, and output the initial redrawn image by alternating through 3-5 sets of gated convolutional layers and feature aggregation and synthesis modules; S5 uses a multi-loss function combination constraint model for training; the AdamW optimizer is used with a learning rate of 1e-5, and the training is conducted for 200-300 epochs to output the final redrawn image.

[0022] Optionally, the random binary mask includes three types: center occlusion, 20%-40% random occlusion, and 40%-60% random occlusion.

[0023] Alternatively, the formula corresponding to the combination of multiple loss functions is: ; in, It is the total loss; It is a loss of perception in the high receptive field; It is an absolute loss; It's a loss of style; It is about combating losses; , , and All of these are hyperparameters.

[0024] Optionally, the step of inputting the input tensor obtained in step S1 into the encoder and extracting 256×256, 128×128, and 64×64 resolution feature maps sequentially through five cascaded gated convolutional layers includes: The input tensor obtained in step S1 is input into the encoder. When passing through the first layer, the input tensor is processed and a 256×256 resolution feature map is output. When passing through the second to fourth layers, the resolution is gradually downsampled, and the resolution is reduced to 128×128 and 64×64 respectively. When passing through the fifth layer, the resolution is maintained at 64×64, and the encoder main feature map is output. The output encoder main feature map is the output encoder final feature map of step S2.

[0025] Optionally, the module first performs multi-scale token interaction and fusion on the features using the MT-Shift mechanism, including: MT-Shift employs a multi-branch structure, with each branch responsible for moving tokens within a specific scale range; input feature map The result is passed through three convolutional layers with different kernel sizes, and then passed through an SCConv layer for each layer. This process can be represented as follows: ; in, These are the learnable parameters in each branch. This represents the output of each branch. , This indicates the size of the convolution kernel; MT-Shift selects two feature fusions, which include: element-wise addition of the results from branches with kernel sizes of 1 and 3, respectively, to obtain the final result. The results of branches with kernel sizes of 3 and 5 are fused together to obtain Finally, an element-by-element addition is performed to obtain the fusion result. The corresponding formula is: ; ; ; The fusion result and original input The final result is obtained by fusion. The corresponding formula is: ; in, This represents the Sigmoid activation function.

[0026] Optionally, the simultaneous capture of global structural constraints and local texture dependencies through the WR-WKV attention mechanism includes: Given a one-dimensional sequence The input features, where This represents the total number of tokens. Through a normalization layer and an MT-Shift layer, the corresponding formula is: ; Will The receptive matrix is ​​obtained through three parallel linear projection layers. Key matrix and value matrix The corresponding formula is: ; ; ; in, , and These represent three parallel linear projection layers; then... and Global attention results are obtained through the window loop WR-WKV mechanism. The corresponding formula is: ; Will and Element-by-element multiplication yields the final result of Spatial Mix, and the corresponding formula is: ; The purpose of the channel fusion module is to extract and fuse features along the channel dimension; for an input feature sequence After passing it through a normalization layer and an MT-Shift layer, the corresponding formula is: ; The acceptance matrix is ​​obtained using the following method. Key matrix and value matrix The corresponding formula is: ; ; ; in, , and These represent three linear projection layers similar to those in spatial mixing; This represents the Squared ReLU activation function, used to enhance the ability of channel mixing modules to capture nonlinear features in the spatial dimension; Will go through Activation function and Element-wise multiplication is performed to fuse the results, yielding the output of the W-RWKV module. The corresponding formula is: .

[0027] This invention provides an efficient redrawing scheme that balances global structural integrity, local detail realism, and identity feature consistency, and has the following beneficial effects: This approach addresses the challenge of adapting RWKV (which accepts weighted key-value pairs) from 1D time-series processing to the 2D image domain. Existing RWKV derivative models (such as Vision-RWKV and Restore-RWKV) lack deep optimization for facial occlusion redrawing scenarios, failing to fully utilize the spatial structural characteristics of faces and leading to an imbalance in global and local dependency modeling. It also overcomes the shortcomings of traditional convolutional networks in long-range semantic association modeling. Existing convolutional methods (such as CMT and DRFNet) are limited by their local receptive field, making it difficult to establish effective semantic associations between occluded and unoccluded regions, easily resulting in structural breaks at occlusion boundaries and homogenization of details. Furthermore, it alleviates the contradiction between computational complexity and refined local modeling in Transformer-like methods. Existing Transformer derivative models (such as Spa-former and HAN) have quadratic computational complexity in their self-attention mechanisms, and their unidirectional attention design cannot adapt to the 2D spatial characteristics of faces, leading to low redrawing efficiency and structural distortion under large-scale occlusion. Finally, it overcomes the limitations of existing token-based methods. The transfer mechanism is not well adapted to the heterogeneity of face regions. Existing token movement schemes (such as RWKV unidirectional movement and Restore-RWKV omnidirectional movement) do not consider the structural differences of different regions of the face (eyes, nose, lips), and cannot accurately capture the local dependencies of subtle structures such as lip lines and frown lines. The identity offset problem under medium to large-scale occlusion (20%-60% occlusion ratio) has been solved. Existing models (such as MAT and PFAN) are prone to problems such as distortion of face topology, loss of identity features, or incorrect modification of background areas when the occlusion area is large, making it difficult to maintain the identity consistency between the redrawn image and the real image.

[0028] In summary, the core objective of this invention is to provide a face occlusion image redrawing network based on RWKV. Through an innovative attention mechanism and token transfer layer design, it achieves pixel-level structural reconstruction, high-fidelity texture filling, and identity feature preservation of the occluded area while ensuring computational efficiency. This meets the robustness requirements for face redrawing with different degrees of occlusion in real-world scenarios, thereby improving the effect of face occlusion image redrawing. Attached Figure Description

[0029] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1This is a flowchart of a facial occlusion image redrawing network method according to the present invention; Figure 2 This is a network structure diagram of the method of the present invention; Figure 3 Here are the structural diagrams for WR-WKV and MT-Shift; Figure 4 A comparison chart of NRMSE metrics for redrawing facial key points in images using different methods; Figure 5 A comparison chart showing the visual effects of redrawing images using different methods; Figure 6 A visual comparison of facial key points redrawn using different methods. Detailed Implementation

[0031] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the specific implementation methods, structures, features, and effects of the technical solution proposed according to the present invention are described in detail below with reference to the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0033] For input unobstructed RGB images , randomized mask and Perform point-by-point multiplication to obtain the occluded image. Then, apply the mask. and occluded images By concatenating the components, we obtain the input tensor of the network. The dimension of the input tensor is ,in, It involves concatenating multiple feature vectors. , , These represent the height, width, and number of channels of the input image, respectively. Represents a binary mask image. The input tensor of the network. The redrawn image is obtained by passing through an encoder, a structure estimation module, and a texture thinning module. .

[0034] refer to Figure 1The flowchart illustrates some embodiments of a facial occlusion image redrawing network method according to the present invention. This facial occlusion image redrawing network method includes the following steps: Step S1: Select an unobstructed real RGB image and generate a random binary mask; multiply the unobstructed real RGB image and the random binary mask point by point to obtain the occluded image, and then concatenate the occluded image and the random binary mask to obtain the input tensor.

[0035] The random binary mask includes three types: center occlusion, 20%-40% random occlusion, and 40%-60% random occlusion.

[0036] In some embodiments, input data preparation and tensor preparation can specifically involve selecting an unobstructed, real RGB image. Generate a random binary mask (Including three categories: central occlusion, 20%-40% random occlusion, and 40%-60% random occlusion); and The occlusion image is obtained by point-by-point multiplication. Then splice it together and , obtain the input tensor ,in, , , These represent the height, width, and number of channels of the input image, respectively. , .

[0037] As an example, regarding data loading and mask generation, specifically, firstly, high-resolution (256×256 pixels) unmasked RGB images of faces are loaded from a training dataset (such as CelebA-HQ or FFHQ). Subsequently, a random binary mask image is generated. Mask types include central region occlusion (covering the central 1 / 4 of the image) or random region occlusion (occlusion ratio 20%-40% or 40%-60%). The mask generation strategy is randomly selected during training to enhance the model's generalization ability. The occluded image is generated and concatenated with the input; specifically, this can be achieved through point-by-point multiplication. Generate the occluded image. Then With mask The input tensor of the network is formed by concatenating the data along the channel dimension. The dimension of the input tensor is This tensor serves as the input source for the entire network and is directly fed into the encoder module.

[0038] It should be noted that step S1 simulates facial occlusion in a real-world scenario, providing standardized input to the network and ensuring the uniformity and repeatability of subsequent processing.

[0039] Step S2: Input the input tensor obtained in step S1 into the encoder, and extract 256×256, 128×128 and 64×64 resolution feature maps sequentially through 5 cascaded gated convolutional layers; output the final feature map of the encoder, and temporarily store the multi-scale feature maps of the first 4 gated convolutional layers.

[0040] In some embodiments, multi-scale feature extraction, specifically, The input encoder extracts feature maps at resolutions of 256×256, 128×128, and 64×64 sequentially through five cascaded gated convolutional layers; the output encoder features the final feature map. Simultaneously, the multi-scale feature maps of the first four gated convolutional layers are temporarily stored for subsequent texture refinement. Here, C represents the number of feature channels.

[0041] As an example, inputting the input tensor obtained in step S1 into the encoder and extracting 256×256, 128×128, and 64×64 resolution feature maps sequentially through five cascaded gated convolutional layers can include: inputting the input tensor obtained in step S1 into the encoder... Input encoder, input tensor Feature extraction is performed sequentially through five gated convolutional layers. The specific process is as follows: the input tensor is processed in the first layer, and a 256×256 resolution feature map is output; in the second to fourth layers, the resolution is progressively downsampled, decreasing to 128×128 and 64×64 respectively; in the fifth layer, the resolution is maintained at 64×64, and the encoder's main feature map is output. Where C is the number of feature channels, and the main feature map of the output encoder is the final feature map of the output encoder in step S2. Multi-scale feature caching: Specifically, to improve feature utilization, the intermediate feature maps output by the first four gated convolutional layers are cached, and these features will be directly fed into the subsequent texture refinement module.

[0042] It should be noted that the encoder extracts multi-scale hierarchical features, preserving detailed information and providing rich context for the structure estimation module. The design of caching multi-scale features is one of the improvements of this invention, avoiding detail loss caused by information compression at the encoder's end and significantly improving the quality of subsequent texture refinement.

[0043] Step S3: Input the final feature map obtained in step S2 into the structure estimation module based on the U-Net architecture. Cross-scale feature transfer is achieved through downsampling, upsampling and skip connections. The core completes the structure modeling through a 7-layer W-RWKV module. In this module, the features are first interacted and fused with multi-scale tokens through the MT-Shift mechanism. Then, the global structure constraints and local texture dependencies are captured simultaneously through the WR-WKV attention mechanism. Finally, a coarsely repaired facial structure feature map is output.

[0044] In some embodiments, the structure estimation module is executed. Specifically, the structure estimation module adopts the U-Net architecture, with the W-RWKV module at its core, and its overall structure is as follows: Figure 2 As shown, facial structure reconstruction (the core innovative step) specifically involves... The input is a U-Net-based structural estimation module, which achieves cross-scale feature transfer through downsampling (convolution with stride of 2), upsampling (pixel reconstruction), and skip connections. The core performs structural modeling through a 7-layer W-RWKV module. Within this module, the MT-Shift mechanism is used to perform multi-scale token interaction and fusion of features. Then, the WR-WKV attention mechanism (partial window bidirectional cyclic scanning) simultaneously captures global structural constraints and local texture dependencies. The final output is a coarsely repaired facial structural feature map FSE∈ .

[0045] It should be noted that step S3 is the core innovation of this invention. The W-RWKV module, MT-Shift mechanism, and WR-WKV attention mechanism work together to solve the core pain points of RWKV adapting to 2D images and global-local modeling imbalance.

[0046] Feature transfer and resolution adjustment, specifically, encoder output features The input structure estimation module is processed through 7 layers of W-RWKV modules. Each layer performs upsampling or downsampling through convolutional layers and pixel shuffle. The U-Net's skip connections concatenate or add the feature maps of the same resolution in the encoder path with the corresponding feature maps in the decoder path to achieve multi-scale feature fusion.

[0047] Based on the unique facial structural features of face images, an innovative multi-scale token shifting mechanism is proposed. MT-Shift can aggregate context at different scales in the spatial dimension, achieving good establishment of local dependencies.

[0048] As an example, performing multi-scale token interaction and fusion of features within a module using the MT-Shift mechanism may include the following steps: Considering the different facial contours in different regions of a face image, MT-Shift employs a multi-branch structure, with each branch responsible for moving the token within a specific scale. Input feature map The data is processed through three convolutional layers with different kernel sizes, and then the results are each processed through an SCConv layer. This improves the utilization of feature information in spatial and channel dimensions at different scales and reduces the weights of redundant features. This process is represented as: ; in, These are the learnable parameters in each branch. This represents the output of each branch. , This indicates the size of the convolution kernel. Then, to avoid the loss of effective facial feature information, MT-Shift performs feature fusion twice. Specifically, as shown... Figure 3 As shown in (d), the branch results with kernel sizes of 1 and 3 are obtained by element-wise addition and fusion. The results of branches with kernel sizes of 3 and 5 are fused together to obtain Finally, an element-by-element addition is performed to obtain the fusion result. The corresponding formula is: ; ; ; Finally, the fusion results and original input The final result is obtained by fusion. The corresponding formula is: ; in, This represents the Sigmoid activation function.

[0049] As another example, the W-RWKV module internally processes data. Specifically, the W-RWKV module comprises two cascaded parts: Spatial Mix and Channel Mix. Its detailed structure is as follows: Figure 3 As shown, the spatial hybridization module aims to construct long-range dependencies in a spatial dimension. Given a one-dimensional sequence... The input features, where This represents the total number of tokens. Through a normalization layer (LayerNorm Layer) and an MT-Shift layer, the corresponding formula is: ; The normalization layer serves to accelerate model convergence. MT-Shift is specifically designed to capture multi-scale contextual dependencies. MT-Shift can aggregate context at different scales in spatial dimensions, achieving a good establishment of local dependencies. A detailed explanation follows.

[0050] Will The receptive matrix is ​​obtained through three parallel linear projection layers. Key matrix and value matrix The corresponding formula is: ; ; ; in, , and These represent three parallel linear projection layers. Then... and Global attention results are obtained through the window loop WR-WKV mechanism. The corresponding formula is: ; To leverage the unique facial structural features of human face images, this invention's window-based recurrent WR-WKV mechanism incorporates a divide-and-conquer approach into traditional Re-WKV. Specifically, unlike Re-WKV which directly scans the entire feature map, this invention divides the feature map into multiple windows, performing Re-WKV attention calculations on each window. This approach aims to effectively improve the capture of facial contour features in local regions while simultaneously scanning the entire map from the first to the last token. The core principle of the proposed window-based WKV attention mechanism is as follows: Figure 3 As shown in (c) and (e), its core can be expressed as: ; Where T represents the number of times Re-WKV is executed in each window.

[0051] Compared to Re-WKV, the WR-WKV proposed in this invention achieves token interaction in local regions by performing horizontal and vertical scanning in divided small windows. This can significantly improve the model's ability to capture facial structural information and lay a good foundation for subsequent reconstruction of facial texture details.

[0052] Finally, and Element-by-element multiplication yields the final result of Spatial Mix, and the corresponding formula is: ; The purpose of the channel blending module is to extract and fuse features along the channel dimension. For an input feature sequence... First, it is passed through a normalization layer and an MT-Shift layer, and the corresponding formula is: ; The acceptance matrix is ​​then obtained using the following method. Key matrix and value matrix The corresponding formula is: ; ; ; in, , and These represent three linear projection layers similar to those in spatial mixing. This represents the Squared ReLU activation function, used to enhance the ability of the channel blending module to capture nonlinear features in the spatial dimension. It's important to note that the method used to compute the VC is not exactly the same as in spatial blending. In fact, a Multilayer Perceptron (MLP) is incorporated into this data flow process. This is done to improve the utilization of feature information in the channel dimension, facilitating the extraction and fusion of channel features.

[0053] Finally, go through Activation function and Element-wise multiplication is performed to fuse the results, yielding the output of the W-RWKV module. The corresponding formula is: .

[0054] It's important to note that W-RWKV aims to retain the advantages of the original RWKV in efficiently processing 1D sequences while improving the model's ability to model key facial structural features against an image background. Therefore, W-RWKV improves upon Vision-RWKV and Restore-RWKV by innovating the attention and token transformation layers in spatial and channel blending, while keeping other operations unchanged. This improvement allows the model to effectively capture global and local dependencies in 2D images. Effects: The structure estimation module outputs coarsely repaired facial structural features. Through the collaboration of WR-WKV and MT-Shift, it achieves topological continuity modeling at occlusion boundaries, effectively preventing distortion of identity features.

[0055] Step S4: Input the roughly repaired facial structure feature map and the multi-scale feature map temporarily stored in step S2 into the texture refinement module. Iterate through 3-5 sets of gated convolutional layers and feature aggregation and synthesis modules to output the initial redrawn image.

[0056] In some embodiments, texture refinement repair specifically involves inputting the FSE and the multi-scale feature map temporarily stored in step S2 into the texture refinement module. Through 3-5 sets of "gated convolutional layers + feature aggregation and synthesis modules (FSAB)" alternating iterations, fine structures such as lip lines and skin textures are gradually filled in, and the initial redrawn image is output.

[0057] As an example, the texture thinning module may perform the following steps: feature aggregation and thinning, and final image generation. Specifically, the feature aggregation and thinning module outputs features from the structure estimation module. The texture refinement module aggregates multi-scale feature maps cached in the encoder stage. It alternates between gated convolutional layers and a Feature Aggregation and Synthesis (FSAB) module, which fuses features through channel attention and spatial convolution, progressively upsampling to a resolution of 256×256. Finally, after multiple rounds of refinement, the module outputs the final reconstructed image. Effects: By utilizing multi-scale contextual information, texture details are refined, artifacts are eliminated, and high-fidelity face images are generated.

[0058] Step S5: Use a combination of multiple loss functions to constrain the model training; use the AdamW optimizer, set the learning rate to 1e-5, train for 200-300 epochs, and output the final redrawn image.

[0059] In some embodiments, model training and optimization can specifically employ a combination of multiple loss functions to constrain model training; the AdamW optimizer (β1=0.5, β2=0.9) is used, with a learning rate of 1e-5, to train for 200-300 epochs, and the final redrawn image Iout is output.

[0060] It should be noted that the combination of multiple loss functions in step S5, through the four-dimensional constraints of "structure-pixel-style-detail", ensures the structural integrity and identity consistency of the redrawn image, which is a key auxiliary innovation for performance improvement.

[0061] As an example, the formula for a combination of multiple loss functions can be: ; in, That is the total loss. It is a loss of perception in the high receptive field. It is an absolute loss. It's a loss of style. It is about combating losses. , , and All are hyperparameters. During the experiment, they were set to... , , , .

[0062] It should be noted that the total loss in this invention includes the High Receptive Field Perception Loss (HRFPL). and absolute loss function In addition, style loss was employed to maintain stylistic consistency between the redrawn image and the real image, and to encourage the model to generate more realistic details. And adversarial loss .

[0063] The optimizer and training settings, specifically, used the AdamW optimizer (parameters β1=0.5, β2=0.9) with a learning rate of 1×10−5, and were trained on a single NVIDIA RTX 3090 GPU. Effect: Multi-loss collaborative optimization ensures that the reconstructed images are close to reality in structure, texture, and vision.

[0064] Key performance validation was implemented, specifically with quantitative evaluation metrics: tests were conducted on the CelebA-HQ, FFHQ, and PFHQ datasets, using metrics such as PSNR, SSIM, and FID. As shown in Table 1, FMI-RWKV outperformed the model in most metrics, especially when the occlusion area was large (40%-60%), with a PSNR improvement of 0.09-0.26 dB compared to the suboptimal model. Topological visualization validation was performed: the normalized root mean square error (NRMSE) of key points between the redrawn image and the real image was calculated using a face alignment algorithm (such as the Bulat model). Figure 5 and Figure 6 As shown, FMI-RWKV has the lowest NRMSE index and its key point topology is closest to reality, verifying identity consistency. Effect: Through objective indicators and visual comparisons, the outstanding effects of WR-WKV and MT-Shift mechanisms in maintaining the integrity of face topology and identity features are confirmed, providing a reliable foundation for downstream tasks.

[0065] Through the above steps, the FMI-RWKV network has fully realized the efficient redrawing of facial occlusion images. Its core innovation lies in transferring the time series modeling advantages of RWKV to the image domain, and solving the problem of insufficient long-range dependency modeling in traditional methods through WR-WKV attention and MT-Shift mechanism. It is suitable for real-world scenarios such as security and virtual reality.

[0066] In summary, the core objective of this invention is to provide a face occlusion image redrawing network based on RWKV. Through an innovative attention mechanism and token transfer layer design, it achieves pixel-level structural reconstruction, high-fidelity texture filling, and identity feature preservation of the occluded area while ensuring computational efficiency. This meets the robustness requirements for face redrawing with different degrees of occlusion in real-world scenarios, thereby improving the effect of face occlusion image redrawing.

[0067] The effective effects of this invention are as follows: Leading in structural and identity fidelity: Under large-scale occlusion of 40%-60%, the PSNR on the FFHQ dataset reaches 24.08dB (0.31dB improvement over the second-best model), and the NRMSE index for facial landmarks is the lowest across all scenarios, completely solving the problems of structural breakage and identity misalignment. Excellent texture and robustness: The SSIM index is the best in all test scenarios (highest 0.9470), and the FID is 38.93 under 20%-40% occlusion on the PFHQ dataset (10.1% lower than PFAN), with natural texture details and no artifacts, adapting to different occlusion ratios. Controllable computational efficiency: Single-GPU training per epoch ≤ 30 minutes, 256×256 image inference ≤ 50ms, linear complexity avoids the quadratic computational cost of Transformer, and low deployment cost.

[0068] The experiment selected FID, PSNR, and SSIM to evaluate the model's performance. PSNR and SSIM are widely used image quality metrics for evaluating visual perception and pixel structural similarity. FID quantifies the difference in feature space distribution between the reconstructed image and the real image; a smaller FID value indicates that the feature distribution of the reconstructed image is closer to that of the real image, meaning the reconstructed image is of higher quality.

[0069] Six advanced models were selected for quantitative and qualitative comparisons, including CMT, MAT, HAN, PFAN, Spa-former, and DRFNet. To ensure fairness in the comparative experiments, all experiments were trained and tested on the same experimental environment and dataset.

[0070] The superior performance of the proposed FMI-RWKV model on face redrawing tasks with varying degrees of occlusion was verified through quantitative and qualitative comparisons. As shown in Table 1, FMI-RWKV achieved the best or second-best results on all three datasets with different occlusion levels, indicating that the proposed model can effectively model facial topology in face redrawing. Furthermore, even with large occlusion areas (40%-60%), FMI-RWKV also achieved better performance metrics than other models, leading the second-best PSNR by 0.09 dB, 0.26 dB, and 0.14 dB on the three datasets, respectively. It is noteworthy that when testing the metrics of each model, MAT's metrics differed significantly from other models. This is because the MAT model modifies the background region outside the face region during image reconstruction, thus affecting the evaluation metric values. Furthermore, as can be observed from the table, FMI-RWKV achieved the best results on all SSIM metrics, fully demonstrating the excellent modeling ability of the W-RWKV module proposed in this invention for the topological structure of faces in occluded regions.

[0071] Table 1 To more fully demonstrate the superiority of FMI-RWKV in the task of reconstructing occluded face images, this invention provides a visual comparison of the reconstructed images from FMI-RWKV and other comparative models. Qualitative results are as follows: Figure 5 As shown, FMI-RWKV not only reconstructs a good facial topology in face redrawing tasks with varying degrees of occlusion, but also achieves good texture detail filling. In contrast, HAN and Spa-former suffer from facial structure distortion when reconstructing the central region mask, thus causing irreversible damage to identity information. While MAT reconstructs a believable facial structure, it inevitably alters identity information, such as... Figure 5In the first row, it's clearly visible that the reconstructed face image differs significantly from the real image. Furthermore, the visualization also shows that MAT alters the background region during reconstruction, affecting the evaluation metrics. CMT, when reconstructing central and large-scale masks, exhibits a clear similarity in style, failing to distinguish distinct identity features. PFAN also suffers from facial structure distortion when reconstructing large-scale mask face images, resulting in a significant deviation from the real image. DRFNet exhibits pixel blurring and noticeable artifacts during facial texture filling, reducing the quality of the reconstructed image. Overall, the proposed FMI-RWKV demonstrates superior performance in reconstructing face images with varying degrees of occlusion, outperforming other comparative models.

[0072] Specifically, the Bulat model and the NRMSE metric are used to quantify the differences between the facial landmarks redrawn by each model and the landmarks in the real facial image. The NRMSE metric results of different models redrawing facial landmarks with different occlusion levels in the CelebA-HQ, FFHQ, and PFHQ datasets are shown below. Figure 4 As shown in the figure, (a) represents the Center type mask; (b) represents the 20%-40% type mask; and (c) represents the 40%-60% type mask. It can be observed from the figure that FMI-RWKV achieved the lowest NRMSE index under different occlusion levels in all three datasets, verifying its ability to reconstruct the topological integrity and continuity of faces under different mask conditions.

[0073] also, Figure 6 A visual comparison of the keypoint coordinates of faces redrawn using different methods was also provided. The red line in the figure highlights the keypoints and topological structure of the face in the real image, while the blue line highlights the keypoints and topological structure of the face images reconstructed by each model. The visualization clearly shows that the image reconstructed by FMI-RWKV has the closest topological structure to the real image. Other methods, such as HAN and Spa-former, redrawn the face keypoints significantly differently from the real image, clearly affecting the integrity and realism of the facial structure. This is also related to… Figure 6 This is corroborated by the comparison results of the NRMSE index.

[0074] This invention utilizes the linear attention mechanism of RWKV, avoiding the quadratic computational complexity of Transformers. Furthermore, it optimizes local computation through windowed processing, achieving efficient training and inference on a single NVIDIA RTX 3090 GPU, balancing performance and efficiency. Training efficiency: single epoch training time ≤ 30 minutes, 20% shorter than Spa-former (a Transformer-like model); Inference efficiency: single 256×256 image inference time ≤ 50ms, meeting the requirements of real-time applications such as security monitoring and facial recognition.

[0075] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for redrawing facial occlusion images using a network, characterized in that, Includes the following steps: S1. Select an unobstructed real RGB image and generate a random binary mask; multiply the unobstructed real RGB image and the random binary mask point by point to obtain the occluded image, and then concatenate the occluded image and the random binary mask to obtain the input tensor; S2, input the input tensor obtained in step S1 into the encoder, and extract 256×256, 128×128 and 64×64 resolution feature maps in sequence through 5 cascaded gated convolutional layers; output the final feature map of the encoder, and temporarily store the multi-scale feature maps of the first 4 gated convolutional layers. S3. The final feature map obtained in step S2 is input into the structure estimation module based on the U-Net architecture. Cross-scale feature transfer is achieved through downsampling, upsampling, and skip connections. The core completes the structure modeling through a 7-layer W-RWKV module. Within the module, the MT-Shift mechanism is first used to perform multi-scale token interaction and fusion on the features. Then, the WR-WKV attention mechanism is used to simultaneously capture global structural constraints and local texture dependencies. Finally, a coarsely repaired facial structure feature map is output. S4, input the roughly repaired facial structure feature map and the multi-scale feature map temporarily stored in step S2 into the texture refinement module, and output the initial redrawn image by alternating through 3-5 sets of gated convolutional layers and feature aggregation and synthesis modules; S5 uses a multi-loss function combination constraint model for training; the AdamW optimizer is used with a learning rate of 1e-5, and the training is conducted for 200-300 epochs to output the final redrawn image.

2. The facial occlusion image redrawing network method according to claim 1, characterized in that, Random binary masks include three types: center occlusion, 20%-40% random occlusion, and 40%-60% random occlusion.

3. The facial occlusion image redrawing network method according to claim 1, characterized in that, The formula for combining multiple loss functions is: ； in, It is the total loss; It is a loss of perception in the high receptive field; It is an absolute loss; It's a loss of style; It is about combating losses; , , and All of these are hyperparameters.

4. The facial occlusion image redrawing network method according to claim 1, characterized in that, The input tensor obtained in step S1 is input to the encoder, and 256×256, 128×128, and 64×64 resolution feature maps are extracted sequentially through five cascaded gated convolutional layers, including: The input tensor obtained in step S1 is input into the encoder. When passing through the first layer, the input tensor is processed and a 256×256 resolution feature map is output. When passing through the second to fourth layers, the resolution is gradually downsampled, and the resolution is reduced to 128×128 and 64×64 respectively. When passing through the fifth layer, the resolution is maintained at 64×64, and the encoder main feature map is output. The output encoder main feature map is the output encoder final feature map of step S2.

5. The facial occlusion image redrawing network method according to claim 1, characterized in that, Within the module, features are first processed and fused using a multi-scale token interaction mechanism via MT-Shift, including: MT-Shift employs a multi-branch structure, with each branch responsible for moving tokens within a specific scale range; input feature map The result is passed through three convolutional layers with different kernel sizes, and then passed through an SCConv layer for each layer. This process can be represented as follows: ； in, These are the learnable parameters in each branch. This represents the output of each branch. , This indicates the size of the convolution kernel; MT-Shift selects two feature fusions, which include: element-wise addition of the results from branches with kernel sizes of 1 and 3, respectively, to obtain the final result. The results of branches with kernel sizes of 3 and 5 are fused together to obtain Finally, an element-by-element addition is performed to obtain the fusion result. The corresponding formula is: ； ； ； The fusion result and original input The final result is obtained by fusion. The corresponding formula is: ； in, This represents the Sigmoid activation function.

6. The facial occlusion image redrawing network method according to claim 5, characterized in that, The method of simultaneously capturing global structural constraints and local texture dependencies through the WR-WKV attention mechanism includes: Given a one-dimensional sequence The input features, where This represents the total number of tokens. Through a normalization layer and an MT-Shift layer, the corresponding formula is: ； Will The receptive matrix is ​​obtained through three parallel linear projection layers. Key matrix and value matrix The corresponding formula is: ； ； ； in, , and These represent three parallel linear projection layers; then... and Global attention results are obtained through the window loop WR-WKV mechanism. The corresponding formula is: ； Will and Element-by-element multiplication yields the final result of Spatial Mix, and the corresponding formula is: ； The purpose of the channel fusion module is to extract and fuse features along the channel dimension; for an input feature sequence After passing it through a normalization layer and an MT-Shift layer, the corresponding formula is: ； The acceptance matrix is ​​obtained using the following method. Key matrix and value matrix The corresponding formula is: ； ； ； in, , and These represent three linear projection layers similar to those in spatial mixing; This represents the Squared ReLU activation function, used to enhance the ability of channel mixing modules to capture nonlinear features in the spatial dimension; Will go through Activation function and Element-wise multiplication is performed to fuse the results, yielding the output of the W-RWKV module. The corresponding formula is: 。

Images