Face image super-resolution reconstruction method and system

Abstract

The present application relates to a kind of face image super-resolution reconstruction method and system.The method includes obtaining original low-resolution face image, inputting original low-resolution face image into shallow feature extraction network, obtaining shallow feature map, and extracting image feature token from shallow feature map;Image feature token is input into deep feature extraction network, multiple rounds of iteration optimization are executed, and deep global feature is obtained;Deep global feature is input into image reconstruction network, and high-resolution face image is reconstructed and output.The method improves the structural accuracy, semantic consistency and visual fidelity of the reconstructed image by explicitly introducing structural edge and semantic segmentation information and guiding the token clustering process in a progressive manner.Meanwhile, by clustering tokens into multiple groups and performing attention calculation in parallel within each group, the global modeling capability can be maintained while meeting the lightweight needs of resource-constrained devices and real-time scenarios.

Description

Technical Field

[0001] This invention relates to the field of automotive electronics and intelligent driving assistance technology, and in particular to a method and system for super-resolution reconstruction of facial images. Background Technology

[0002] Super-resolution reconstruction of facial images aims to recover high-resolution images with clear textures from low-resolution facial images, and has wide applications in facial recognition, security monitoring, portrait restoration, and mobile photography. Traditional methods based on convolutional neural networks are limited by the local receptive field of the convolutional kernel, making it difficult to effectively capture long-range dependencies in facial images. While Transformer-based methods, which have emerged in recent years, improve global modeling capabilities through self-attention mechanisms, their computational complexity increases quadratically with image resolution, making them difficult to deploy on resource-constrained devices and in real-time scenarios.

[0003] To reduce computational overhead, researchers have proposed various localized sparse attention strategies. However, these methods typically employ fixed region partitioning independent of content, making it difficult to effectively aggregate texture information that is distant but similar in content, thus affecting the stability of reconstruction quality. Token interaction methods based on content clustering can promote long-distance information propagation through clustering, but currently they mainly rely on the similarity of low-level image features for clustering, lacking effective utilization of high-level structural and semantic information. Moreover, attention is often limited to a single scale, making it difficult to consider both coarse-grained contours and fine-grained textures, and failing to utilize the significant left-right symmetry prior of face images, resulting in low information interaction efficiency and limited consistency of reconstructed structures. Summary of the Invention

[0004] This invention provides a method and system for super-resolution reconstruction of facial images to overcome at least one of the aforementioned technical problems in the prior art.

[0005] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions: In a first aspect, the present invention provides a method for super-resolution reconstruction of facial images, comprising: Obtain the original low-resolution face image, input the original low-resolution face image into the shallow feature extraction network to obtain the shallow feature map, and extract the image feature token from the shallow feature map; The image feature token is input into a deep feature extraction network, and multiple rounds of iterative optimization are performed to obtain deep global features; wherein each round of iterative optimization includes: Extract structural and semantic features from the current image feature tokens; The current image feature token is weighted and fused with the structural feature and the semantic feature to obtain the guiding feature, wherein the weight of the weighted fusion is progressively adjusted according to the level depth of the current iteration round; Based on the guiding features, the current image feature tokens are clustered into multiple related token groups; For each of the token groups, intra-group self-attention computation and inter-group cross-attention computation are performed in parallel to obtain aggregated features; The deep global features are input into the image reconstruction network to reconstruct and output a high-resolution face image.

[0006] In one possible implementation of the first aspect, clustering the current image feature tokens into multiple related token groups includes: Calculate the cosine similarity between each current image feature token and a pre-constructed and maintained set of global token centers; Each current image feature token is assigned to the group corresponding to the global token center that has the maximum cosine similarity to the current image feature token, to obtain multiple related token groups.

[0007] In one possible implementation of the first aspect, the parallel execution of intra-group self-attention computation and inter-group cross-attention computation includes: The current image feature token within each token group is evenly divided into several token subgroups; Each of the aforementioned token subgroups performs intra-group self-attention computation. Perform inter-group cross-attention calculations between each of the token subgroups and the global token center.

[0008] In one possible implementation of the first aspect, each of the token subgroups performs intra-group self-attention computation, including: Calculate the initial attention score among the current image feature tokens within the token subgroup; A symmetry bias term is superimposed on the initial attention score to obtain a corrected attention score, wherein the value of the symmetry bias term is determined based on the degree of symmetry of the coordinates of the current image feature token in the image space about the vertical midline of the face. The attention weight is calculated based on the corrected attention score.

[0009] In one possible implementation of the first aspect, performing inter-group cross-attention calculation between each of the token subgroups and the global token center includes: Calculate the correlation score between the current token subgroup features and each of the global token centers; Based on the correlation score, the top K global token centers are selected as the center subset for participating in the interaction, where K is a preset positive integer; Using the current token subgroup features as the query and the central subset as the key and value, perform inter-group cross-attention calculation.

[0010] In one possible implementation of the first aspect, after obtaining the aggregated features in each round of the iterative optimization, the method further includes: The aggregated features are downsampled at multiple scales to obtain feature maps at multiple different scales; For each scale of the feature map, the window size of the local attention is dynamically adjusted according to the scale, and local region self-attention calculation is performed to obtain the feature map after scale calculation. The feature maps calculated at all the aforementioned scales are upsampled to the same resolution and fused to obtain multi-scale fused features.

[0011] In one possible implementation of the first aspect, each iteration of the deep feature extraction network consists of a residual set, and after obtaining the multi-scale fused features, it further includes: The multi-scale fused features are convolved, and the convolved multi-scale fused features are residually connected with the input features of the current residual group to obtain the final output features of the current residual group, which are then used as the input of the next residual group.

[0012] In one possible implementation of the first aspect, before clustering the current image feature tokens into multiple related token groups, performing intra-group self-attention calculation and inter-group cross-attention calculation, and dynamically adjusting the window size of local attention according to the scale and performing local region self-attention calculation, a layer normalization operation is performed first; after clustering the current image feature tokens into multiple related token groups, performing intra-group self-attention calculation and inter-group cross-attention calculation, and dynamically adjusting the window size of local attention according to the scale and performing local region self-attention calculation, a convolutional feedforward network is connected, and residual shortcut connections are added respectively.

[0013] In one possible implementation of the first aspect, the step of inputting the deep global features into the image reconstruction network to reconstruct a high-resolution face image includes: Convolution and pixel shuffling are performed on the deep global features to obtain global residual features in the pixel space; The original low-resolution face image is upsampled and added pixel by pixel to the global residual feature to obtain a high-resolution face image.

[0014] Compared with the prior art, the present invention has at least the following beneficial effects: The super-resolution reconstruction method for face images provided by this invention introduces both structural features (edges) and semantic features (facial regions) for dual guidance, and uses progressive weight adjustment for the clustering process. This solves the problem that existing methods, which rely solely on the underlying texture similarity for clustering, are prone to incorrectly aggregating semantically unrelated but texture-similar tokens. This significantly improves the structural accuracy, semantic consistency, and visual fidelity of the reconstructed image.

[0015] Furthermore, the face image super-resolution reconstruction method provided by this invention clusters tokens into multiple groups and performs attention calculations in parallel within each group. The actual attention calculations only occur within smaller token groups, rather than among all tokens in the entire image. This significantly reduces computational complexity, enabling the maintenance of global modeling capabilities while meeting the lightweight requirements of resource-constrained devices and real-time scenarios.

[0016] Secondly, the present invention provides a face image super-resolution reconstruction system, comprising: The shallow feature extraction module is used to acquire the original low-resolution face image, input the original low-resolution face image into the shallow feature extraction network to obtain the shallow feature map, and extract the image feature token from the shallow feature map. A deep global feature extraction module is used to input the image feature token into a deep feature extraction network, perform multiple rounds of iterative optimization, and obtain deep global features; wherein each round of iterative optimization includes: Extract structural and semantic features from the current image feature tokens; The current image feature token is weighted and fused with the structural feature and the semantic feature to obtain the guiding feature, wherein the weight of the weighted fusion is progressively adjusted according to the level depth of the current iteration round; Based on the guiding features, the current image feature tokens are clustered into multiple related token groups; For each of the token groups, intra-group self-attention computation and inter-group cross-attention computation are performed in parallel to obtain aggregated features; The image reconstruction and output module is used to input the deep global features into the image reconstruction network, reconstruct and output a high-resolution face image.

[0017] Thirdly, the present invention provides an electronic device comprising: at least one processor and at least one memory, wherein the memory stores computer-readable instructions; the computer-readable instructions are executed by one or more of the processors to cause the electronic device to implement the face image super-resolution reconstruction method as described in any implementation of the first aspect.

[0018] Fourthly, the present invention provides a storage medium having a computer-executable program stored thereon, the computer-executable program being used to cause a computer to perform a face image super-resolution reconstruction method as described in any implementation of the first aspect.

[0019] Understandably, the beneficial effects of the system of the second aspect, the electronic device of the third aspect, and the storage medium of the fourth aspect provided above can be referred to in light of the beneficial effects of the first aspect and any of its possible design embodiments, which will not be repeated here. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention; Figure 2 This is a flowchart of a face image super-resolution reconstruction method provided in an embodiment of the present invention; Figure 3 This is a structural block diagram of a face image super-resolution reconstruction system provided in an embodiment of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be described below with reference to the accompanying drawings. In the description of the present invention, unless otherwise stated, " / " indicates that the objects before and after are in an "or" relationship. For example, A / B can represent A or B. The "or" in the present invention is merely a description of the relationship between the related objects, indicating that three relationships can exist. For example, A or B can represent: A alone, A and B simultaneously, and B alone. A and B can be singular or plural. Furthermore, in the description of the present invention, unless otherwise stated, "multiple" refers to two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items.

[0023] Furthermore, to facilitate a clear description of the technical solutions of the embodiments of the present invention, the terms "first" and "second" are used in the embodiments of the present invention to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" are not necessarily different.

[0024] In this embodiment of the invention, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in this embodiment of the invention should not be construed as superior or more advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner for ease of understanding.

[0025] Super-resolution reconstruction of facial images aims to recover high-resolution images with clear textures from low-resolution facial images, and has wide applications in facial recognition, security monitoring, portrait restoration, and mobile photography. Traditional methods based on convolutional neural networks are limited by the local receptive field of the convolutional kernel, making it difficult to effectively capture long-range dependencies in facial images. While Transformer-based methods, which have emerged in recent years, improve global modeling capabilities through self-attention mechanisms, their computational complexity increases quadratically with image resolution, making them difficult to deploy on resource-constrained devices and in real-time scenarios.

[0026] To reduce computational overhead, researchers have proposed various localized sparse attention strategies. However, these methods typically employ fixed region partitioning independent of content, making it difficult to effectively aggregate texture information that is distant but similar in content, thus affecting the stability of reconstruction quality. Token interaction methods based on content clustering can promote long-distance information propagation through clustering, but currently they mainly rely on the similarity of low-level image features for clustering, lacking effective utilization of high-level structural and semantic information. Moreover, attention is often limited to a single scale, making it difficult to consider both coarse-grained contours and fine-grained textures, and failing to utilize the significant left-right symmetry prior of face images, resulting in low information interaction efficiency and limited consistency of reconstructed structures.

[0027] In view of this, on the one hand, embodiments of the present invention provide a method for super-resolution reconstruction of face images, comprising: acquiring an original low-resolution face image; inputting the original low-resolution face image into a shallow feature extraction network to obtain a shallow feature map, and extracting image feature tokens from the shallow feature map; inputting the image feature tokens into a deep feature extraction network, performing multiple rounds of iterative optimization to obtain deep global features; wherein each round of iterative optimization includes: extracting structural features and semantic features from the current image feature tokens; performing weighted fusion of the current image feature tokens with the structural features and the semantic features to obtain guiding features, wherein the weights of the weighted fusion are progressively adjusted according to the layer depth of the current iteration round; clustering the current image feature tokens into multiple related token groups according to the guiding features; performing intra-group self-attention calculation and inter-group cross-attention calculation in parallel for each token group to obtain aggregated features; inputting the deep global features into an image reconstruction network to reconstruct and output a high-resolution face image.

[0028] This invention provides a face image super-resolution reconstruction method that introduces both structural features (edges) and semantic features (facial regions) for dual guidance, and uses progressive weighting to adjust the clustering process. This solves the problem in existing methods that rely solely on low-level texture similarity for clustering, which easily leads to the erroneous aggregation of semantically unrelated but texture-similar tokens. This significantly improves the structural accuracy, semantic consistency, and visual fidelity of the reconstructed image. By clustering tokens into multiple groups and performing attention calculations in parallel within each group, the actual attention calculation only occurs within smaller token groups, rather than among all tokens in the entire image. This greatly reduces computational complexity, enabling lightweight design for resource-constrained devices and real-time scenarios while maintaining global modeling capabilities.

[0029] In some embodiments, the face image super-resolution reconstruction method provided by the present invention can be executed by any electronic device 20 with data processing capabilities, such as a general-purpose computer, personal computer, laptop computer, switch, or tablet computer, etc. The specific implementation of the electronic device 20 is not limited here.

[0030] Figure 1 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention is shown. The electronic device 20 includes a processor 210, a memory 220, and a communication interface 230.

[0031] Processor 210 may include one or more processing cores. Processor 210 connects to various parts within electronic device 200 using various interfaces and lines, and performs various functions and processes data of electronic device 200 by running or executing instructions, programs, code sets, or instruction sets stored in memory 220, and by calling data stored in memory 220. Optionally, processor 210 may be implemented using at least one of the following hardware forms: Central Processing Unit (CPU), Graphics Processing Unit (GPU), Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA).

[0032] The memory 220 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 220 may include a non-transitory computer-readable storage medium. The memory 220 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 220 may include a program storage area. This program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the various method embodiments described above, etc.

[0033] Communication interface 230 is used to communicate with other devices, equipment or communication networks, such as data storage devices, image processing devices or Ethernet, wireless access network (RAN), wireless local area network (WLAN), etc.

[0034] In terms of physical implementation, the aforementioned devices (such as processor 210, memory 220, and communication interface 230) can each be devices within the same device (such as a laptop computer). Alternatively, at least two of these devices can be located within the same device, i.e., as different devices within the same device, similar to the deployment of devices or components in a distributed system.

[0035] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 20. In other embodiments of the present invention, the electronic device 20 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0036] The following description, in conjunction with the accompanying drawings, illustrates a face image super-resolution reconstruction method provided by an embodiment of the present invention.

[0037] like Figure 2 As shown, this embodiment of the invention provides a face image super-resolution reconstruction method, applied to software-defined networks, which may include, but is not limited to: S1: Obtain the original low-resolution face image, input the original low-resolution face image into the shallow feature extraction network to obtain the shallow feature map, and extract the image feature token from the shallow feature map.

[0038] In the specific implementation process, the original low-resolution face image I in the embodiments of the present invention LR There are various ways to obtain images, such as direct real-time capture by a low-resolution camera; reading existing low-resolution face images from a local disk, database, or cloud; receiving low-resolution face images from other devices via a network; and obtaining simulated low-resolution images by applying downsampling, blurring, noise, or other degradation processing to high-resolution face images, etc., without any limitation here.

[0039] In specific implementation, the shallow feature extraction network in this embodiment of the invention may, but is not limited to, consist of a single convolutional layer (such as a 3×3 convolutional layer), and its function is to extract the original low-resolution face image I. LR Mapping from the original pixel input space to a high-dimensional feature space breaks the spatial correlation of the original pixels, and outputs a shallow feature map. Its mathematical expression is shown below: ; In the formula: F s () represents the shallow feature extraction function, corresponding to a 3×3 convolution operation.

[0040] In specific implementation, the image feature token in this embodiment of the invention is a shallow feature map. An equivalent representation after being flattened in a spatial dimension can be understood as follows: The input low-resolution face image is processed through a single convolutional layer, outputting a spatial feature map of shape C×H×W, i.e., a shallow feature map. .

[0041] Then the shallow feature map In spatial dimension, it is flattened into N = H × W vectors of length C, and each vector corresponds to an image feature token.

[0042] S2: Input the image feature token into the deep feature extraction network, perform multiple rounds of iterative optimization, and obtain deep global features.

[0043] In the specific implementation process, the deep feature extraction network in the embodiments of the present invention is composed of N residual groups stacked in series, and each round of iterative optimization is performed by one residual group.

[0044] Specifically, shallow feature maps Entering the deep feature extraction network, for the i-th residual group, let its input feature be... Within the token aggregation block of the i-th residual group, perform the following specific operations: Extract structural and semantic features from the current image feature token.

[0045] In specific implementation, embodiments of the present invention utilize edge extraction techniques (such as the Sobel operator, Canny operator, or Laplacian operator) to extract the structural features of the face at the current level. (Such as the facial features, facial edges, hairline, eyebrow shape, and other linear information, which constitute the spatial skeleton of the face), its functional expression is shown in the following formula: ; In the formula, Sobel() is a fixed operator used to extract structural features.

[0046] This invention utilizes a pre-trained lightweight semantic segmentation network to extract semantic features of the face at the current layer. (Such as the left eye region, nose region, mouth region, background region, etc.), its function expression is shown in the following formula: ; In the formula, TinyNet() is a lightweight semantic segmentation network used to extract semantic features.

[0047] After obtaining the structural features and the semantic features, the current image feature token is weighted and fused with the structural features and the semantic features to obtain the guiding features. The weight of the weighted fusion is progressively adjusted according to the hierarchical depth of the current iteration round.

[0048] In specific implementation, this embodiment of the invention performs weighted fusion of the current image feature token with the structural feature and the semantic feature to obtain the guiding feature. Its function expression is shown in the following formula: ; In the formula, φ() and ψ() are Convolution projection function, and For progressive weights that change dynamically with network layers, set to... , In the shallow layer (where i is smaller), Larger values If the value is small or even zero, token clustering is mainly dominated by structural information such as edges; as the number of layers increases, Gradually decrease and As the number of cells increases, the clustering process gradually transitions to being dominated by information from high-level semantic regions such as eyes, nose, and mouth.

[0049] After obtaining the guiding features, the current image feature tokens are clustered into multiple related token groups based on the guiding features.

[0050] In one feasible implementation, the clustering of the current image feature tokens into multiple related token groups in this embodiment of the invention may include, but is not limited to: Calculate the cosine similarity between each current image feature token and a pre-constructed and maintained set of global token centers; Each current image feature token is assigned to the group corresponding to the global token center that has the maximum cosine similarity to the current image feature token, to obtain multiple related token groups.

[0051] In specific implementation, embodiments of the present invention construct and maintain a set of M global token centers shared across the training set during the training phase. An exponential moving average strategy is adopted, and the global token center is slowly updated based on the temporary value calculated in each iteration. Its functional expression is shown in the following equation: ; In the formula, λ is the attenuation parameter. This is the temporary value of the token center updated after a single iteration.

[0052] It should be noted that, in this embodiment of the invention, the global token center gradually learns and stores representative feature patterns related to structure and semantics in the entire dataset during the training process. During the inference phase, these learned global token centers are directly and permanently used, participating only in clustering and grouping, and are no longer updated.

[0053] During clustering, the guiding features are calculated. Each current image feature token With all M global token centers Cosine similarity, construct a similarity matrix , The total number of feature tokens is given. The current image feature tokens are assigned to the group corresponding to the center with the highest similarity, thus dividing all image feature tokens into M token groups related to structural semantics. : .

[0054] After obtaining M token groups, for each token group, perform intra-group self-attention calculation and inter-group cross-attention calculation in parallel to obtain aggregated features.

[0055] In one feasible implementation, the parallel execution of intra-group self-attention calculation and inter-group cross-attention calculation in this embodiment of the invention may include, but is not limited to: Each of the token groups The internal current image feature token is evenly divided into several fixed-size... token subgroup Its function expression is shown in the following formula: ; ; Each of the aforementioned token subgroups performs in-group self-attention computation; Perform inter-group cross-attention calculations between each of the token subgroups and the global token center.

[0056] The embodiments of the present invention effectively avoid the problem of low parallel computing efficiency caused by the uneven number of tokens in different groups by uniformly dividing the current image feature tokens within each token group into several token subgroups and then performing intra-group self-attention calculation and inter-group cross-attention calculation in parallel.

[0057] In one feasible implementation, the intra-group self-attention calculation performed within each token subgroup in this embodiment of the invention may include, but is not limited to: Calculate the initial attention score among the current image feature tokens within the token subgroup; A symmetry bias term is superimposed on the initial attention score to obtain a corrected attention score, wherein the value of the symmetry bias term is determined based on the degree of symmetry of the coordinates of the current image feature token in the image space about the vertical midline of the face. The attention weight is calculated based on the corrected attention score.

[0058] In the specific implementation process, the embodiments of the present invention perform linear projection on the features of each token subgroup to obtain the query matrix. Key matrix Sum matrix Its function expression is shown in the following formula: ; In the formula, Let be the projection weight matrix. For feature dimensions.

[0059] To leverage the prior knowledge of facial symmetry, a symmetry bias term is introduced when calculating the attention weights. The result of the dot product of the query matrix and the key matrix is ​​shown in the following formula: ; .

[0060] In the formula, This is a relative position encoding matrix. Perform a token position restoration operation.

[0061] In this embodiment of the invention, The value of is calculated based on the degree of symmetry of the token space coordinates with respect to the vertical central axis of the image; the stronger the symmetry, the larger the bias. This mechanism guides each image feature token to focus on its features at a symmetrical position in image space, thereby leveraging the natural left-right symmetry of the face to enhance the modeling ability of local structures.

[0062] In one feasible implementation, the method of performing inter-group cross-attention calculation between each token subgroup and the global token center in this embodiment of the invention may include, but is not limited to: Calculate the correlation score between the current token subgroup features and each of the global token centers; Based on the correlation score, the top K global token centers are selected as the center subset for participating in the interaction, where K is a preset positive integer; Using the current token subgroup features as the query and the central subset as the key and value, perform inter-group cross-attention calculation.

[0063] In specific implementation, embodiments of the present invention may, but are not limited to, using the calculation of the cosine similarity between the features of the current token subgroup and each of the global token centers to obtain the relevance score. The specific process may be as follows: First, average pooling is used to aggregate the features of all image feature tokens within the current token subgroup, resulting in a feature vector representing the current token subgroup. Then, the feature vector of the current token subgroup is calculated and its relationship to all M global token centers is established. The cosine similarity is used to obtain M relevance scores; based on the relevance scores, the top K most relevant global token centers are selected to form an interaction subset. Then, using the characteristics of the current token subset as the query, and the interaction subset... As keys and values, multi-head cross-attention computation is performed, and its function expression is shown below: ; ; ; In the formula, The cross-attention projection weight matrix, This is for multi-head self-attention calculation.

[0064] The embodiments of the present invention introduce a Top-K interaction mechanism, which can efficiently integrate global information and reduce computational redundancy.

[0065] Finally, the output features obtained by calculating intra-group self-attention and inter-group cross-attention are combined. and Element-wise addition and fusion are performed, and feature integration is carried out through a convolutional layer (1×1 convolutional layer) to obtain the aggregated features of this token aggregation block. Its function expression is shown in the following formula: ; In the formula, F Conv () is the feature fusion function of a 1×1 convolution.

[0066] In one feasible implementation, after obtaining the aggregated features, the embodiments of the present invention may, but are not limited to, further include: The aggregated features are downsampled at multiple scales to obtain feature maps at multiple different scales; For each scale of the feature map, the window size of the local attention is dynamically adjusted according to the scale, and local region self-attention calculation is performed to obtain the feature map after scale calculation. The feature maps calculated at all the aforementioned scales are upsampled to the same resolution and fused to obtain multi-scale fused features.

[0067] In specific implementation, this embodiment of the invention provides a multi-scale pyramid local region self-attention module. This module constructs multiple attention branches at different scales, such as the original scale, half-scale, and quarter-scale attention branches. These branches perform operations on the input aggregated features at the original scale, half-scale, and quarter-scale, respectively. At different scale branches, multi-head self-attention calculations are performed within the local window with an adapted attention window size. The calculation formula is shown below: ; ; ; In the formula, For average pooling downsampling operation, This is a bilinear interpolation upsampling operation. In the window Multi-head self-attention calculation at different sizes.

[0068] At the original scale branch, the window covers a smaller receptive field, focusing on capturing fine-grained texture details, such as skin pores and the number of eyebrow hairs. At the half-scale branch, the window's effective receptive field on the original image doubles, capturing medium-scale structural information. At the quarter-scale branch, the window covers the largest area of ​​the original image, capturing coarse-grained global structure and the overall layout of facial features. This multi-scale parallel modeling approach allows the model to simultaneously consider information at different granularities.

[0069] Finally, the feature maps calculated at the aforementioned scales are upsampled to the same resolution and concatenated along the channel dimension. A 1×1 convolutional layer is then used for multi-scale feature fusion to obtain the multi-scale fused features. Its function expression is shown in the following formula: ; In the formula, F Conv ( ) represents the feature fusion function of a 1×1 convolution. This is for channel splicing operations.

[0070] This invention enables parallel capture and fusion of information of different granularities and receptive field ranges in an image without significantly increasing computational overhead. It performs multi-scale downsampling on the aggregated features, then performs multi-head self-attention calculation within a local window with an appropriate attention window size on different scale branches, and finally upsamples all the feature maps calculated at the scales to the same resolution and fuses them. This enhances the ability to express structural information and fine-grained texture information at different scales.

[0071] In specific implementation, within a residual group, both the token aggregation block and the multi-scale pyramid local region self-attention module of this invention follow a unified standardized processing flow: Before performing token clustering, intra-group self-attention calculation, inter-group cross-attention calculation, and dynamically adjusting the window size of local attention according to the scale, and performing local region self-attention calculation, the input features are first normalized to effectively eliminate feature scale differences and stabilize the training process.

[0072] After all the above calculations are completed, each is sequentially connected to a convolutional feedforward network consisting of two 1×1 convolutional layers and a nonlinear activation function, which realizes the information interaction and nonlinear transformation of the output features in the channel dimension, thereby enhancing the expressive power of the features.

[0073] Meanwhile, residual shortcut connections are added to the token aggregation block and the multi-scale pyramid local region self-attention module, respectively, and the initial input of the module is added element-wise to the output of the convolutional feedforward network to optimize the gradient propagation path and alleviate the gradient vanishing problem of deep networks.

[0074] In one feasible implementation, after obtaining the multi-scale fusion features, the embodiments of the present invention may, but are not limited to, further include: The multi-scale fused features are convolved, and the convolved multi-scale fused features are residually connected with the input features of the current residual group to obtain the final output features of the current residual group, which are then used as the input of the next residual group.

[0075] In specific implementation, embodiments of the present invention may, but are not limited to, multi-scale fusion features. A 3×3 convolution operation is performed to further refine local features, and the local spatial correlation of convolution is used to implicitly learn position embeddings, compensating for the lack of position awareness in Transformer-type models. The output features of the 3×3 convolution are then compared with the input features of the current residual group. Performing residual connections stabilizes the model training process, alleviates the gradient vanishing problem in deep networks, and yields the output features of the i-th residual group. Output features of the i-th residual group As the input for the next round of residual sets, Deep global features are obtained after rounds of iteration. The characteristic transformation formula for the i-th residual group is shown in the following equation: ; In the formula, F Conv ( ) is the feature fusion function of a 3×3 convolution.

[0076] S3: Input the deep global features into the image reconstruction network, reconstruct and output a high-resolution face image.

[0077] In one feasible implementation, the method of inputting the deep global features into the image reconstruction network to reconstruct a high-resolution face image, as described in this embodiment of the invention, may include, but is not limited to: Convolution and pixel shuffling are performed on the deep global features to obtain global residual features in the pixel space; The original low-resolution face image is upsampled and added pixel by pixel to the global residual feature to obtain a high-resolution face image.

[0078] In the specific implementation process, this embodiment of the invention employs a global residual learning strategy for final reconstruction. On one hand, the input original low-resolution face image is directly upsampled to the target high-resolution size; on the other hand, deep global features are... The input image reconstruction function, consisting of convolutional layers (such as 3×3 convolutional layers) and pixel shuffling operations, is mapped to high-frequency residual detail information in the image pixel space. Adding these two values ​​pixel-by-pixel yields the final reconstructed high-resolution face image. Its reconstruction function expression is shown in the following equation: ; In the formula, For the final high-resolution image output; F Up () is the upsampling operation function, which scales a low-resolution image to the target high-resolution size; F IR () is the image reconstruction function.

[0079] The embodiments of the present invention simplify the model's fitting target from complete image pixel values ​​to only predicting the high-frequency residual difference between high-resolution images and low-resolution upsampled images through the above-described global residual reconstruction method. This significantly reduces the learning difficulty of the model, enabling the model to focus its expressive power on the recovery of key details, thereby improving the clarity and fidelity of the reconstructed image.

[0080] To verify the technical effectiveness of the face image super-resolution reconstruction method provided in this embodiment of the invention, experiments were conducted on the CelebA face image public dataset, as detailed below: Based on the CelebA dataset, existing mainstream face super-resolution methods (including Bicubic, SPARNet, SISN, SFMNet, UFSRNet, FreqFormer, and CATANet) are tested against this solution in a ×4 super-resolution image reconstruction test.

[0081] Peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and feature similarity (FSIM) were used as objective evaluation metrics. The experimental results are shown in Table 1 below. Experiments show that, compared with existing mainstream face super-resolution methods, the face image super-resolution reconstruction method provided in this embodiment of the invention has significantly improved performance. Specifically, under the ×4 magnification of the CelebA dataset, this method improves PSNR by 0.70dB, SSIM by 0.0098, and FSIM by 0.0067 compared with the baseline model CATANet; and all indicators are superior to other mainstream comparison methods.

[0082] Based on the face image super-resolution reconstruction method provided in the first aspect, embodiments of the present invention provide a face image super-resolution reconstruction system, such as... Figure 3 As shown, the face image super-resolution reconstruction system includes: The shallow feature extraction module 110 is used to acquire the original low-resolution face image, input the original low-resolution face image into the shallow feature extraction network to obtain a shallow feature map, and extract image feature tokens from the shallow feature map. The deep global feature extraction module 120 is used to input the image feature token into the deep feature extraction network, perform multiple rounds of iterative optimization, and obtain deep global features; wherein each round of iterative optimization includes: Extract structural and semantic features from the current image feature tokens; The current image feature token is weighted and fused with the structural feature and the semantic feature to obtain the guiding feature, wherein the weight of the weighted fusion is progressively adjusted according to the level depth of the current iteration round; Based on the guiding features, the current image feature tokens are clustered into multiple related token groups; For each of the token groups, intra-group self-attention computation and inter-group cross-attention computation are performed in parallel to obtain aggregated features; The image reconstruction and output module 130 is used to input the deep global features into the image reconstruction network, reconstruct and output a high-resolution face image.

[0083] Based on the face image super-resolution reconstruction method provided in the first aspect, this embodiment of the invention also provides a storage medium storing a computer-executable program. The computer-executable program is used to cause a computer to execute the face image super-resolution reconstruction method as described in any implementation of the first aspect. Explanations of the relevant content and descriptions of the beneficial effects of any of the computer-readable storage media provided above can be found in the corresponding embodiments described above, and will not be repeated here.

[0084] Those skilled in the art will understand that the program for implementing all or part of the steps of the above embodiments, which can be executed by a program instructing related hardware, can be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a random access memory, etc. The processing unit or processor mentioned above can be a central processing unit, a general-purpose processor, an application-specific integrated circuit (ASIC), a microprocessor (DSP), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof.

[0085] This invention also provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform any of the methods described in the above embodiments. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this invention is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., SSD), etc.

[0086] It should be noted that the devices for storing computer instructions or computer programs provided in the embodiments of the present invention, such as, but not limited to, the aforementioned memory, computer-readable storage medium, and communication chip, are all non-transitory. Those skilled in the art should recognize that the functions described in the embodiments of the present invention in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable storage medium or transmitted as one or more instructions or code on a computer-readable storage medium. Computer-readable storage media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of computer programs from one place to another. Storage media can be any available medium accessible to general-purpose or special-purpose computers.

[0087] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for super-resolution reconstruction of facial images, characterized in that, include: Obtain the original low-resolution face image, input the original low-resolution face image into the shallow feature extraction network to obtain the shallow feature map, and extract the image feature token from the shallow feature map; The image feature token is input into a deep feature extraction network, and multiple rounds of iterative optimization are performed to obtain deep global features; wherein each round of iterative optimization includes: Extract structural and semantic features from the current image feature tokens; The current image feature token is weighted and fused with the structural feature and the semantic feature to obtain the guiding feature, wherein the weight of the weighted fusion is progressively adjusted according to the level depth of the current iteration round; Based on the guiding features, the current image feature tokens are clustered into multiple related token groups; For each of the token groups, intra-group self-attention computation and inter-group cross-attention computation are performed in parallel to obtain aggregated features; The deep global features are input into the image reconstruction network to reconstruct and output a high-resolution face image.

2. The method for super-resolution reconstruction of face images according to claim 1, characterized in that, The step of clustering the current image feature tokens into multiple related token groups includes: Calculate the cosine similarity between each current image feature token and a pre-constructed and maintained set of global token centers; Each current image feature token is assigned to the group corresponding to the global token center that has the maximum cosine similarity to the current image feature token, to obtain multiple related token groups.

3. The method for super-resolution reconstruction of face images according to claim 2, characterized in that, The parallel execution of intra-group self-attention calculation and inter-group cross-attention calculation includes: The current image feature token within each token group is evenly divided into several token subgroups; Each of the aforementioned token subgroups performs in-group self-attention computation; Perform inter-group cross-attention calculations between each of the token subgroups and the global token center.

4. The method for super-resolution reconstruction of face images according to claim 3, characterized in that, Each of the token subgroups performs intra-group self-attention calculation, including: Calculate the initial attention score among the current image feature tokens within the token subgroup; A symmetry bias term is superimposed on the initial attention score to obtain a corrected attention score, wherein the value of the symmetry bias term is determined based on the degree of symmetry of the coordinates of the current image feature token in the image space about the vertical midline of the face. The attention weight is calculated based on the corrected attention score.

5. The method for super-resolution reconstruction of face images according to claim 3, characterized in that, The step of performing inter-group cross-attention calculation between each of the token subgroups and the global token center includes: Calculate the correlation score between the current token subgroup features and each of the global token centers; Based on the correlation score, the top K global token centers are selected as the center subset for participating in the interaction, where K is a preset positive integer; Using the current token subgroup features as the query and the central subset as the key and value, perform inter-group cross-attention calculation.

6. The method for super-resolution reconstruction of face images according to claim 1, characterized in that, In each round of the iterative optimization, after obtaining the aggregated features, the following steps are also included: The aggregated features are downsampled at multiple scales to obtain feature maps at multiple different scales; For each scale of the feature map, the window size of the local attention is dynamically adjusted according to the scale, and local region self-attention calculation is performed to obtain the feature map after scale calculation. The feature maps calculated at all the aforementioned scales are upsampled to the same resolution and fused to obtain multi-scale fused features.

7. The method for super-resolution reconstruction of face images according to claim 6, characterized in that, Each iteration of the deep feature extraction network consists of a residual set. After obtaining the multi-scale fused features, it also includes: The multi-scale fused features are convolved, and the convolved multi-scale fused features are residually connected with the input features of the current residual group to obtain the final output features of the current residual group, which are then used as the input of the next residual group.

8. The method for super-resolution reconstruction of face images according to claim 7, characterized in that, Before clustering the current image feature tokens into multiple related token groups, performing intra-group self-attention calculation and inter-group cross-attention calculation, and dynamically adjusting the window size of local attention according to the scale and performing local region self-attention calculation, layer normalization is performed first. After clustering the current image feature tokens into multiple related token groups, performing intra-group self-attention calculation and inter-group cross-attention calculation, and dynamically adjusting the window size of local attention according to the scale and performing local region self-attention calculation, a convolutional feedforward network is connected, and residual shortcut connections are added respectively.

9. The method for super-resolution reconstruction of face images according to claim 1, characterized in that, The step of inputting the deep global features into the image reconstruction network to reconstruct a high-resolution face image includes: Convolution and pixel shuffling are performed on the deep global features to obtain global residual features in the pixel space; The original low-resolution face image is upsampled and added pixel by pixel to the global residual feature to obtain a high-resolution face image.

10. A face image super-resolution reconstruction system, characterized in that, include: The shallow feature extraction module is used to acquire the original low-resolution face image, input the original low-resolution face image into the shallow feature extraction network to obtain the shallow feature map, and extract the image feature token from the shallow feature map. A deep global feature extraction module is used to input the image feature token into a deep feature extraction network, perform multiple rounds of iterative optimization, and obtain deep global features; wherein each round of iterative optimization includes: Extract structural and semantic features from the current image feature tokens; The current image feature token is weighted and fused with the structural feature and the semantic feature to obtain the guiding feature, wherein the weight of the weighted fusion is progressively adjusted according to the level depth of the current iteration round; Based on the guiding features, the current image feature tokens are clustered into multiple related token groups; For each of the token groups, intra-group self-attention computation and inter-group cross-attention computation are performed in parallel to obtain aggregated features; The image reconstruction and output module is used to input the deep global features into the image reconstruction network, reconstruct and output a high-resolution face image.

Images