A high-resolution face image reconstruction method based on sparse refocusing

Abstract

The application discloses a high-resolution face image reconstruction method based on sparse refocusing, and relates to the technical field of image processing, and comprises the following steps: a first convolutional layer with a convolution kernel of 3*3 is used to extract shallow features of an input low-resolution face image; component features are extracted based on three sequentially connected component feature extraction networks; the input of the first component feature extraction network is connected with the output of the first convolutional layer; high-resolution component images are generated based on three sequentially connected component generation modules; the input of each component generation module is connected with the output of a component feature extraction network; the high-resolution component images are combined by pixel-by-pixel addition to generate a reconstructed high-resolution face image. The application can effectively suppress redundant features and emphasize key features, realizes efficient feature extraction, reconstructs components and combines according to the complexity of the image components, reduces the difficulty of high-frequency detail reconstruction, and restores the high-resolution face image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a high-resolution face image reconstruction method based on sparse refocusing. Background Technology

[0002] Surveillance systems, primarily consisting of front-end cameras, transmission cables, and video monitoring platforms, are widely used in public places such as urban roads, airports, train stations, shopping malls, banks, and schools, as well as civilian locations like factories, warehouses, and residential communities, due to their advantages of being intuitive, accurate, timely, and providing rich information. However, limitations imposed by factors such as imaging distance, motion blur, and lighting obstruction sometimes prevent cameras from capturing facial images with sufficient resolution and recognition accuracy to meet the demands of smart security applications. As a key task in computer vision, Face Super-Resolution (FSR) technology can reconstruct one or more low-resolution facial images into corresponding high-resolution facial images while maintaining existing hardware. This is achieved by comprehensively utilizing signal processing, statistical learning, and optimization theory. Therefore, FSR technology is a low-cost and practical solution that can significantly improve the imaging quality of video surveillance systems, especially facial resolution and recognition accuracy, without requiring the replacement of existing cameras.

[0003] FSR (Facial Resolution Super-Resolution) generates a corresponding high-resolution (HR) face image from an existing low-resolution (LR) face image through image processing algorithms. With the development of deep learning, Transformer-based methods have become dominant in the field of FSR and have become the main method for single-image super-resolution research at present.

[0004] However, existing Transformers have the following limitations that hinder high-quality FSR restoration. On the one hand, the underlying self-attention mechanism of Transformers cannot avoid redundant features participating in image reconstruction, thus interfering with high-resolution face image restoration. On the other hand, different face image components (e.g., nose, wrinkles, facial contours, skin, etc.) have different reconstruction difficulties, and directly reconstructing a complete face image may result in significant blurring of facial details.

[0005] Therefore, how to provide a super-resolution method for facial images to achieve fine facial image detail reconstruction has become an urgent problem to be solved. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and propose a high-resolution face image reconstruction method based on sparse refocusing, which can effectively aggregate valuable features to participate in image reconstruction while avoiding interference from redundant features; in addition, it can restore different image components in a divide-and-conquer manner, significantly reducing the difficulty of reconstructing complex facial details.

[0007] To achieve the above objectives, the technical solution of the present invention is a high-resolution face image reconstruction method based on sparse refocusing, comprising:

[0008] On the one hand, a high-resolution face image reconstruction method based on sparse refocusing includes:

[0009] The shallow features of the input low-resolution face image are extracted using the first convolutional layer with a 3×3 kernel.

[0010] The low-resolution face image is extracted using three sequentially connected component feature extraction networks; wherein the input of the first component feature extraction network is connected to the output of the first convolutional layer; the component features include contour, corner points, and planar features;

[0011] A high-resolution component map is generated based on three sequentially connected component generation modules; wherein the input of each component generation module is connected to the output of a component feature extraction network.

[0012] A reconstructed high-resolution face image is generated by merging high-resolution component images pixel by pixel.

[0013] Preferably, each component feature extraction network consists of a second convolutional layer with a 3×3 kernel and several Transformer modules based on sparse refocusing connected in series;

[0014] The Transformer module based on sparse refocusing includes a first normalization layer, a self-attention unit based on sparse refocusing, a second normalization layer, and a hybrid dimension feedforward unit connected in sequence.

[0015] For the input feature map The output of the sparse refocusing-based Transformer module is C, H, and W represent the number of channels, length, and width of the feature map, respectively. The processing procedure based on the sparse refocusing Transformer module is as follows:

[0016] X′=SRSA(LN(X))+X

[0017] Y = MDFN(LN(X′)) + X′

[0018] Where X′ represents the intermediate calculation result; LN(X) represents the first normalization layer processing function; SRSA(·) represents the self-attention unit processing function based on sparse refocusing; MDFN(·) represents the mixed-dimensional feedforward unit processing function; and LN(X′) represents the second normalization layer processing function.

[0019] Preferably, for the input feature map The processing procedure of the self-attention unit based on sparse refocusing is as follows:

[0020] First, using three linear layers to respectively Convert to query ,key Sum ;

[0021] Next, by order With transposed Perform matrix multiplication to obtain the incidence matrix. Its expression is as follows:

[0022]

[0023] in, Indicates the transpose operation;

[0024] Secondly, Input the correlation refocusing module to obtain the enhanced focus correlation matrix. Its expression is as follows:

[0025]

[0026] Next, a sparse feature selection operation is used to mask the masking. The lowest contribution ranking in each row Each attention weight is assigned a value of negative infinity (-inf), while the remaining attention weights remain unchanged; the sparse feature selection operation is expressed as:

[0027]

[0028] in, This represents the sparse feature selection operation function; Indicates the location of The The result corresponding to the sparse feature selection operation performed on the attention weight set at the row position; Indicates row index; Indicates column index; This represents the sparse feature selection operation; Indicates that it is located at The line, number Attention weights on column positions; Indicates that it is located at The The set of attention weights at row positions;

[0029] Finally, for sparse matrices Perform a softmax calculation and multiply it with the V matrix. Then, input the result into a linear layer to generate enhanced features. Its expression is as follows:

[0030]

[0031] in, Functions representing linear layers; Indicates to The sparse matrix obtained after performing sparse feature selection row by row.

[0032] Preferably, for the input correlation matrix The processing procedure of the associated refocusing module is as follows:

[0033] Will Input layer normalization layer, obtain values ;

[0034] Will The query is transformed sequentially through a transpose operation, a linear layer, a normalization layer, and two parallel linear layers. s and keys ;in, Indicates the number of channels; This represents the floor function;

[0035] right and transposed Perform matrix multiplication and Softmax calculations, and then multiply the results by... Pixel-by-pixel multiplication to obtain the focus correlation matrix This is used as the output of the module, and its expression is as follows:

[0036]

[0037] in, This indicates a pixel-by-pixel multiplication operation.

[0038] Preferably, for the input feature map The processing procedure for the hybrid-dimensional feedforward unit is as follows:

[0039] Channel-level feature associations are established by sequentially connecting linear layers, GELU activation functions, and linear layers;

[0040] Multi-scale features are extracted and pixel-level accumulation is performed using a parallel structure constructed from three depthwise convolutions;

[0041] The calculation results are input into a linear layer for adaptive correction. .

[0042] Preferably, for the input correlation matrix The processing steps for each component generation module are as follows:

[0043] First, a third convolutional layer with a 3×3 kernel and a sub-pixel convolutional layer are used to... Size upsampling to The result is denoted as ;

[0044] Secondly, construct a parallel double-branch structure and... As input to the structure, the upper branch of the structure utilizes a fourth convolutional layer with a 3×3 kernel. Convert to intermediate high resolution face image The next branch uses the mask generation module to... Convert to component mask diagram The mask generation module consists of a fifth convolutional layer with a 3×3 kernel, a GELU activation function, a sixth convolutional layer with a 3×3 kernel, and a Sigmoid activation function connected in series.

[0045] Finally, let and Perform pixel-by-pixel multiplication to separate specific components from the intermediate high-resolution face image and obtain a high-resolution component map. .

[0046] Preferably, the high-resolution face image reconstruction method based on sparse refocusing further includes: constructing a joint loss function for face reconstruction to supervise the generation of high-resolution face images by a divide-and-conquer face image super-resolution network based on sparse refocusing; wherein, the divide-and-conquer face image super-resolution network based on sparse refocusing consists of a first convolutional layer with a kernel of 3×3 and three component sub-networks connected in series; each component sub-network includes a connected component feature extraction network and the component generation module; the component feature extraction network consists of a second convolutional layer with a kernel of 3×3 and five Transformer modules based on sparse refocusing connected in series.

[0047] Preferred, reconstructed joint loss function ,as follows:

[0048]

[0049] in, Represents the L1 loss function; Represents high-resolution face images from the training set; Represents a reconstructed high-resolution face image; A function representing the Harris corner detection algorithm; These represent the application of the Harris corner detection algorithm from... The extracted high-resolution contour map, high-resolution corner map, and high-resolution planar map; These represent the high-resolution contour map, high-resolution corner map, and high-resolution planar map predicted by the divide-and-conquer face image super-resolution network based on sparse refocusing, respectively. These represent the intermediate high-resolution face images generated by the three component sub-networks, respectively.

[0050] As can be seen from the above description of the present invention, compared with the prior art, the present invention has the following beneficial effects:

[0051] This invention extracts component features based on a three-component feature extraction network in series, which can effectively suppress redundant features and emphasize key features, thus achieving efficient feature extraction. Based on the output of the self-testing generation module, it reconstructs and merges the three necessary components of plane, edge and corner according to the complexity of image components, which can reduce the difficulty of reconstructing high-frequency details and restore high-resolution face images.

[0052] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments, but the high-resolution face image reconstruction method based on sparse refocusing of the present invention is not limited to the embodiments. Attached Figure Description

[0053] Figure 1 This is a flowchart of a high-resolution face image reconstruction method based on sparse refocusing according to an embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram of a divide-and-conquer face image super-resolution network based on sparse refocusing, according to an embodiment of the present invention.

[0055] Figure 3 This is a schematic diagram of a Transformer module based on sparse refocusing according to an embodiment of the present invention;

[0056] Figure 4 This is a schematic diagram of a self-attention unit based on sparse refocusing according to an embodiment of the present invention;

[0057] Figure 5 This is a schematic diagram of a hybrid-dimensional feedforward unit according to an embodiment of the present invention;

[0058] Figure 6 This is a schematic diagram of the component generation module in an embodiment of the present invention. Detailed Implementation

[0059] The technical solutions in the embodiments of the present invention will be described and discussed in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention.

[0060] See Figure 1 As shown in the figure, this embodiment of a high-resolution face image reconstruction method based on sparse refocusing includes the following steps:

[0061] S101, using the first convolutional layer with a kernel of 3×3 to extract shallow features of the input low-resolution face image;

[0062] S102, component features are extracted based on three sequentially connected component feature extraction networks; wherein, the input of the first component feature extraction network is connected to the output of the first convolutional layer;

[0063] S103, a high-resolution component map is generated based on three sequentially connected component generation modules; wherein, the input of each component generation module is connected to the output of a component feature extraction network;

[0064] S104 uses pixel-by-pixel addition to merge high-resolution component images to generate a reconstructed high-resolution face image.

[0065] In this embodiment, see Figure 2 As shown, a first convolutional layer with a 3×3 kernel and three component sub-networks are connected in series to form a divide-and-conquer face image super-resolution network based on sparse refocusing. Each component sub-network includes a connected component feature extraction network and the component generation module. The component feature extraction network consists of a second convolutional layer with a 3×3 kernel and five Transformer modules based on sparse refocusing connected in series.

[0066] In this embodiment, see Figure 3 As shown, the Transformer module based on sparse refocusing includes a first normalization layer, a self-attention unit based on sparse refocusing, a second normalization layer, and a hybrid dimension feedforward unit connected in sequence.

[0067] Given feature map As the input to the sparse refocusing-based Transformer module, its corresponding output The calculation process is as follows:

[0068]

[0069]

[0070] in, This indicates intermediate calculation results; This represents the first normalization layer processing function; This represents the self-attention unit processing function based on sparse refocusing; This represents the processing function of the mixed-dimensional feedforward unit; This represents the second-level normalization layer processing function; These represent the number of channels, length, and width of the feature map, respectively.

[0071] For details, see Figure 4 The diagram shown is a schematic of the structure of the self-attention unit based on sparse refocusing in this embodiment, and its processing is as follows.

[0072] First, using three linear layers to respectively Convert to query ,key Sum .

[0073] Next, by order With transposed Perform matrix multiplication to obtain the incidence matrix. Its expression is as follows:

[0074]

[0075] in, This indicates the transpose operation.

[0076] Secondly, Input the correlation refocusing module to obtain the enhanced focus correlation matrix. Its expression is as follows:

[0077]

[0078] Next, a sparse feature selection operation is used to mask the masking. Each line The lowest-ranked contributor One attention weight is assigned a value of negative infinity (-inf), while the remaining attention weights remain unchanged; the sparse feature selection operation is represented as:

[0079]

[0080] in, This represents the sparse feature selection operation function; Indicates the location of The The result corresponding to the sparse feature selection operation performed on the attention weight set at the row position; Indicates row index; Indicates column index; This represents the sparse feature selection operation; Indicates that it is located at The line, number Attention weights on column positions; Indicates that it is located at The The set of attention weights at row positions;

[0081] Finally, for sparse matrices Perform a softmax calculation and multiply it with the V matrix. Then, input the result into a linear layer to generate enhanced features. Its expression is as follows:

[0082]

[0083] in, Functions representing linear layers; Indicates to The sparse matrix obtained after performing sparse feature selection row by row.

[0084] Specifically, for the input correlation matrix The processing procedure of the associated refocusing module is as follows:

[0085] Will Input layer normalization layer, obtain values ;

[0086] Will The query is transformed sequentially through a transpose operation, a linear layer, a normalization layer, and two parallel linear layers. s and keys ;in, Indicates the number of channels; This represents the floor function;

[0087] right and transposed Perform matrix multiplication and Softmax calculations, and then multiply the results by... Pixel-by-pixel multiplication to obtain the focus correlation matrix This is used as the output of the module, and its expression is as follows:

[0088]

[0089] in, This indicates a pixel-by-pixel multiplication operation.

[0090] For details, see Figure 5 The diagram shown is a schematic representation of the hybrid-dimensional feedforward unit in this embodiment. For the input feature map... The processing procedure for the hybrid dimension feedforward unit is as follows.

[0091] First, a linear layer-GELU activation function-linear layer model is used to model channel-level feature associations. Next, a parallel structure constructed from three depthwise convolutions (kernels of 3×3, 5×5, and 7×7) is used to extract multi-scale features and perform pixel-level accumulation. Finally, the computation results are input into a linear layer for adaptive correction and the result is output. .

[0092] See Figure 6 The diagram shown is a schematic of the component generation module according to an embodiment of the present invention. For the input correlation matrix... The processing procedure for each component generation module is as follows.

[0093] Given an input feature map scaling factor First, using a convolutional layer with a third convolutional kernel of 3×3 and a sub-pixel convolutional layer to... Size upsampling to The result is denoted as Secondly, construct a parallel double-branch structure and... As input to the structure, the upper branch of the structure utilizes a fourth convolutional layer with a 3×3 kernel. Convert to intermediate high resolution face image The next branch uses the mask generation module to... Convert to component mask diagram The mask generation module consists of a fifth convolutional layer with a 3×3 kernel, a GELU activation function, a sixth convolutional layer with a 3×3 kernel, and a Sigmoid activation function connected in series; finally, let... The mask performs pixel-by-pixel multiplication to separate specific components from the intermediate high-resolution face image and obtain a high-resolution component map. .

[0094] based on Figures 3-5 The structure of the self-attention unit, hybrid-dimensional feedforward unit, and component generation module based on sparse refocusing, and the above processing procedure, are described. Given an input low-resolution face image... scaling factor The execution process of the divide-and-conquer face image super-resolution network based on sparse refocusing is as follows.

[0095] First, shallow features are extracted using a convolutional layer with a 3×3 kernel. ;

[0096] Secondly, component features are extracted using the component feature extraction network within the component sub-network. , where i = 1, 2, 3. On the one hand, A high-resolution component diagram will be generated through the corresponding component generation module. ;on the other hand, It is passed to the (i+1)th component sub-network to extract high-resolution features corresponding to other types of components.

[0097] Finally, high-resolution component images are merged using pixel-by-pixel addition. To generate reconstructed high-resolution face images .

[0098] Furthermore, to supervise the generation of high-resolution face images by the sparse refocusing-based divide-and-conquer face image super-resolution network, a joint loss function for face reconstruction is constructed. Its expression is as follows:

[0099]

[0100] in, Represents the L1 loss function; Represents high-resolution face images from the training set; Represents a reconstructed high-resolution face image; A function representing the Harris corner detection algorithm; These represent the application of the Harris corner detection algorithm from... The extracted high-resolution contour map, high-resolution corner map, and high-resolution planar map; These represent the high-resolution contour map, high-resolution corner map, and high-resolution planar map predicted by the divide-and-conquer face image super-resolution network based on sparse refocusing, respectively. These represent the intermediate high-resolution face images generated by the three component sub-networks, respectively. The generation of the high-resolution component graph, the intermediate high-resolution face image, and the reconstructed high-resolution face image is supervised using the loss function described above to ensure that the divide-and-conquer face image super-resolution network based on sparse refocusing can generate high-quality face images.

[0101] The above is merely a preferred embodiment of the present invention. However, the present invention is not limited to the above embodiment. Any equivalent changes and modifications made according to the present invention, provided that the resulting functional effects do not exceed the scope of the present invention, shall fall within the protection scope of the present invention.

Claims

1. A high-resolution face image reconstruction method based on sparse refocusing, characterized in that, include: The shallow features of the input low-resolution face image are extracted using the first convolutional layer with a 3×3 kernel. Component features of low-resolution face images are extracted using three sequentially connected component feature extraction networks. The input of the first component feature extraction network is connected to the output of the first convolutional layer. The component features include contours, corner points, and planar features. Each component feature extraction network consists of a second convolutional layer with a 3×3 kernel and several Transformer modules based on sparse refocusing connected in series. The Transformer module based on sparse refocusing includes a first normalization layer, a self-attention unit based on sparse refocusing, a second normalization layer, and a mixed-dimensional feedforward unit connected in sequence. For the input feature map The output of the sparse refocusing-based Transformer module is C, H, and W represent the number of channels, length, and width of the feature map, respectively. The processing procedure based on the sparse refocusing Transformer module is as follows: ； ； in, Indicates intermediate calculation results; This represents the first normalization layer processing function; This represents the self-attention unit processing function based on sparse refocusing; This represents the processing function of the mixed-dimensional feedforward unit; This represents the second-level normalization layer processing function; For the input feature map The processing procedure of the self-attention unit based on sparse refocusing is as follows: First, using three linear layers to respectively Convert to query ,key Sum ; Next, the incidence matrix is ​​obtained by performing matrix multiplication between Q and the transpose of K. Its expression is as follows: ； Where T represents the transpose operation; Secondly, input M into the correlation and refocusing module to obtain the enhanced focusing correlation matrix. Its expression is as follows: ； Next, a sparse feature selection operation is used to mask the masking. In each row, the K attention weights with the lowest contribution ranking are assigned the value of negative infinity - inf, while the remaining attention weights remain unchanged; the sparse feature selection operation is expressed as: ； in, This represents the sparse feature selection operation function; Indicates the location of The result corresponding to the sparse feature selection operation performed on the attention weight set at the i-th row position; Indicates row index; Indicates column index; This represents the sparse feature selection operation; Indicates that it is located at The attention weights at the i-th row and j-th column positions; Indicates that it is located at The set of attention weights at the i-th row position; Finally, for sparse matrices Perform a softmax calculation and multiply it with the V matrix. Then, input the result into a linear layer to generate enhanced features. Its expression is as follows: ； in, Functions representing linear layers; Indicates to The sparse matrix obtained after performing sparse feature selection row by row; For the input correlation matrix The processing procedure of the associated refocusing module is as follows: The input layer M is normalized to obtain the value. ; Will The query is transformed sequentially through a transpose operation, a linear layer, a normalization layer, and two parallel linear layers. s and keys ;in, Indicates the number of channels; This represents the floor function; right and transposed Perform matrix multiplication and Softmax calculations, and then multiply the results by... Pixel-by-pixel multiplication to obtain the focus correlation matrix This is used as the output of the module, and its expression is as follows: ； in, This indicates a pixel-by-pixel multiplication operation; A high-resolution component map is generated based on three sequentially connected component generation modules; wherein the input of each component generation module is connected to the output of a component feature extraction network. A reconstructed high-resolution face image is generated by merging high-resolution component images pixel by pixel.

2. The high-resolution face image reconstruction method based on sparse refocusing as described in claim 1, characterized in that, For the input feature map The processing procedure for the hybrid-dimensional feedforward unit is as follows: Channel-level feature associations are established by sequentially connecting linear layers, GELU activation functions, and linear layers; Multi-scale features are extracted and pixel-level accumulation is performed using a parallel structure constructed from three depthwise convolutions; The calculation results are input into a linear layer for adaptive correction. .

3. The high-resolution face image reconstruction method based on sparse refocusing as described in claim 1, characterized in that, For the input correlation matrix The processing steps for each component generation module are as follows: First, a third convolutional layer with a 3×3 kernel and a sub-pixel convolutional layer are used to... The size is upsampled to C×(H×s)×(H×s), and the result is denoted as ; Secondly, construct a parallel double-branch structure and... As input to the structure, the upper branch of the structure utilizes a fourth convolutional layer with a 3×3 kernel. Convert to intermediate high resolution face image The next branch uses the mask generation module to... Convert to component mask diagram The mask generation module consists of a fifth convolutional layer with a 3×3 kernel, a GELU activation function, a sixth convolutional layer with a 3×3 kernel, and a Sigmoid activation function connected in series. Finally, let The mask performs pixel-by-pixel multiplication to separate specific components from the intermediate high-resolution face image and obtain a high-resolution component map. .

4. The high-resolution face image reconstruction method based on sparse refocusing as described in claim 1, characterized in that, Also includes: A joint loss function for face reconstruction is constructed to supervise the generation of high-resolution face images by a divide-and-conquer face image super-resolution network based on sparse refocusing. The divide-and-conquer face image super-resolution network based on sparse refocusing consists of a first convolutional layer with a 3×3 kernel and three component sub-networks connected in series. Each component sub-network includes a connected component feature extraction network and the component generation module. The component feature extraction network consists of a second convolutional layer with a 3×3 kernel and five Transformer modules based on sparse refocusing connected in series.

5. The high-resolution face image reconstruction method based on sparse refocusing as described in claim 4, characterized in that, Joint loss function for reconstruction ,as follows: ； in, Represents the L1 loss function; Represents high-resolution face images from the training set; Represents a reconstructed high-resolution face image; A function representing the Harris corner detection algorithm; These represent the application of the Harris corner detection algorithm from... The extracted high-resolution contour map, high-resolution corner map, and high-resolution planar map; These represent the high-resolution contour map, high-resolution corner map, and high-resolution planar map predicted by the divide-and-conquer face image super-resolution network based on sparse refocusing, respectively. These represent the intermediate high-resolution face images generated by the three component sub-networks, respectively.

Images