Relevant method, device and equipment of face super-division network, chip and storage medium

Abstract

The embodiment of the invention discloses a human face super-division network related method, device and equipment, a chip and a storage medium, and the method comprises the steps: carrying out the multi-layer coding of a low-quality human face image in a training set through employing a coding sub-network, and obtaining the multi-layer image features of the low-quality human face image; adopting a global attention sub-network to extract global semantic information of the last layer of image features to obtain global features; and performing multi-layer decoding processing on the global features by adopting a decoding sub-network in combination with the multi-layer image features to obtain a prediction result of the high-quality face image corresponding to the low-quality face image. According to the method, the global attention module is used for modeling global semantic information, so that the problem of insufficient global semantic information capture caused by local link characteristics of a traditional convolution structure is solved, generation of pseudo details in the face super-division process is effectively inhibited, and the face super-division effect is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device, equipment, chip and storage medium related to a face super-resolution network. Background Art

[0002] Portrait photography is a key application scenario for camera functions, where the clarity and naturalness of the face directly affects the success rate and user experience. Face super-resolution technology can break through the limitations of hardware imaging resolution and further improve the clarity of the finished image, and is widely used in current mobile phone camera functions.

[0003] Due to the local connection characteristics of traditional convolutional structures, the receptive field of existing face super-resolution models based on convolutional neural networks usually has local characteristics, that is, it can only focus on some local information and cannot focus on global semantic information. Therefore, in the task of face super-resolution, the input noise may be treated as details and enhanced, which may lead to serious pseudo details in the super-resolution results. Summary of the invention

[0004] The embodiments of the present application at least provide a method, device, equipment, chip and storage medium related to a face super-resolution network.

[0005] The technical solution of this application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a training method for a face super-resolution network, wherein the face super-resolution network to be trained includes at least: an encoding sub-network to be trained, a global attention sub-network to be trained, and a decoding sub-network to be trained; the method includes: using the encoding sub-network to perform multi-layer encoding processing on low-quality face images in a training set to obtain multi-layer image features of the low-quality face images; using the global attention sub-network to extract the global semantic information of the last layer of image features to obtain global features; using the decoding sub-network to perform multi-layer decoding processing on the global features in combination with the multi-layer image features to obtain a prediction result of a high-quality face image corresponding to the low-quality face image; based on the prediction result and the real high-quality face image corresponding to the low-quality face image, determining a first loss; based on the first loss, adjusting the network parameters of the face super-resolution network to be trained so that the loss output by the adjusted face super-resolution network meets the convergence condition to obtain the target face super-resolution network.

[0007] In the second aspect, an embodiment of the present application provides a training method for a face super-resolution network, the method comprising: using an encoding sub-network to perform multi-layer encoding processing on a target low-quality face image to obtain multi-layer image features of the target low-quality face image; using a global attention sub-network to extract the global semantic information of the last layer of image features to obtain global features; using a decoding sub-network to perform multi-layer decoding processing on the global features in combination with the multi-layer image features to obtain a prediction result of a high-quality face image corresponding to the target low-quality face image.

[0008] In a third aspect, an embodiment of the present application provides a training device for a face super-resolution network, wherein the face super-resolution network to be trained includes at least: an encoding sub-network to be trained, a global attention sub-network to be trained, and a decoding sub-network to be trained; the device includes: an encoding unit, which uses the encoding sub-network to perform multi-layer encoding processing on low-quality face images in a training set to obtain multi-layer image features of the low-quality face images; an extraction unit, which uses the global attention sub-network to extract the global semantic information of the last layer of image features to obtain global features; a decoding unit, which uses the decoding sub-network to perform multi-layer decoding processing on the global features in combination with the multi-layer image features to obtain a prediction result of a high-quality face image corresponding to the low-quality face image; a determination unit, which is used to determine a first loss based on the prediction result and the real high-quality face image corresponding to the low-quality face image; an adjustment unit, which is used to adjust the network parameters of the face super-resolution network to be trained based on the first loss, so that the loss output by the adjusted face super-resolution network meets the convergence condition to obtain the target face super-resolution network.

[0009] In a fourth aspect, an embodiment of the present application provides a face super-resolution device, which includes: an encoding unit, which uses an encoding sub-network to perform multi-layer encoding processing on a target low-quality face image to obtain multi-layer image features of the target low-quality face image; an extraction unit, which uses a global attention sub-network to extract the global semantic information of the last layer of image features to obtain global features; and a decoding unit, which uses a decoding sub-network to perform multi-layer decoding processing on the global features in combination with multi-layer image features to obtain a prediction result of a high-quality face image corresponding to the target low-quality face image.

[0010] In the fifth aspect, an embodiment of the present application provides a face super-resolution device, which includes a memory and a processor; wherein the memory is used to store computer-executable instructions; the processor is connected to the memory, and is used to implement the method described in the first aspect or the second aspect by executing the computer-executable instructions.

[0011] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by at least one processor, it implements the method described in the first aspect or the second aspect.

[0012] In a seventh aspect, an embodiment of the present application provides a chip, which includes: a processor, configured to call and run a computer program from a memory, so that a device equipped with the chip executes the method described in the first aspect or the second aspect.

[0013] In an embodiment of the present application, a coding sub-network can be used to perform multi-layer coding processing on low-quality face images, and then a global attention sub-network can be used to process the coding results to extract global semantic information and obtain global features. Finally, a decoding sub-network is used to perform multi-layer decoding processing on the obtained global features, thereby obtaining a predicted high-quality face image. This method uses a global attention module to model global semantic information, thereby compensating for the problem of insufficient global semantic information capture caused by the local connection characteristics of the traditional convolutional structure, suppressing the generation of pseudo details in the face super-resolution process, and thus improving the face super-resolution effect. In addition, this method is conducive to multi-scale analysis of low-quality face images by performing multi-layer coding processing on low-quality face images in the training set, so that the information contained in the obtained image features can be richer.

[0014] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0016] Figure 1 Schematic diagram of the traditional U-net architecture;

[0017] Figure 2 This is a schematic diagram of the super-resolution results obtained by using a super-resolution network trained using the traditional U-net architecture;

[0018] Figure 3 A flowchart of a method for training a face super-resolution network provided in an embodiment of the present application;

[0019] Figure 4 Schematic diagram of the face super-resolution network provided in the embodiment of the present application Figure 1 ;

[0020] Figure 5 Schematic diagram of the face super-resolution network provided in the embodiment of the present application Figure 2 ;

[0021] Figure 6 A schematic diagram of a process for extracting global semantic information of the last layer of image features using a global attention sub-network provided in an embodiment of the present application;

[0022] Figure 7 A flowchart of a face super-resolution method provided in an embodiment of the present application;

[0023] Figure 8 A schematic diagram of an example of using a face super-resolution network to perform face super-resolution on a low-quality face image provided in an embodiment of the present application;

[0024] Fig. 9 Schematic diagram of face super-resolution results provided in the embodiment of the present application Figure 1 ;

[0025] Fig.10 Schematic diagram of face super-resolution results provided in the embodiment of the present application Figure 2 ;

[0026] Fig.11 A schematic diagram of the composition structure of a training device for a face super-resolution network provided in an embodiment of the present application;

[0027] Fig.12 A schematic diagram of the structure of a face super-resolution device provided in an embodiment of the present application;

[0028] Fig.13 A schematic diagram of a hardware entity of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0030] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meaning as those commonly understood by those skilled in the art of the present application. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0031] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. It should also be noted that the terms "first\second\third" involved in the embodiments of the present application are only used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first\second\third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0032] It should be understood that the term "and / or" in the embodiments of the present application is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0033] Portrait photography is a key application scenario for the camera function, in which the clarity and naturalness of the face will directly affect the success rate and user experience. Face super-resolution technology can break through the limitations of hardware imaging resolution and further improve the clarity of the finished image, and is widely used in current mobile phone camera functions. However, due to the local connection characteristics of the traditional convolutional structure, the receptive field of the existing convolutional neural network-based face super-resolution model usually has local characteristics, that is, it can only focus on some local information and cannot focus on global semantic information. Therefore, in the task of face super-resolution, the input noise may be treated as details and enhanced, which in turn leads to serious pseudo-details in the super-resolution results.

[0034] For example, U-net is a commonly used network architecture that is widely used in segmentation, image restoration and other fields. The main architecture of traditional U-net is as follows: Figure 1 As shown. Figure 1 In the figure, the input of the U-net network architecture is a low-quality face image, and the output is a predicted high-quality face image. The network architecture adopts an encoding-decoding structure, in which the left half is the encoding process and the right half is the decoding process. Since the traditional U-net architecture is a deep neural network built based on local convolutional components, its receptive field has local characteristics, which can easily lead to serious pseudo details in the super-resolution image.

[0035] Figure 2 This is the super-resolution result obtained by using the super-resolution network trained with the traditional U-net architecture. Figure 2It can be observed that the network enhances some pseudo details on the face below the eyes. This is because local noise appears to be the same color as hair. If only local information is considered, the noise is easily mistaken for hair and enhanced.

[0036] In view of this, the embodiment of the present application provides a method, device, equipment, chip and storage medium related to a face super-resolution network, which can be executed by a processor of a computer device. Among them, the computer device may refer to a device with data processing capabilities such as a server, a laptop, a tablet computer, and a desktop computer. In this method, a coding sub-network can be used to perform multi-layer coding processing on low-quality face images, and then a global attention sub-network is used to process the coding results to extract global semantic information and obtain global features. Finally, a decoding sub-network is used to perform multi-layer decoding processing on the obtained global features, thereby obtaining a predicted high-quality face image. The method uses a global attention module to model global semantic information, which makes up for the problem of insufficient global semantic information capture caused by the local connection characteristics of the traditional convolutional structure, suppresses the generation of pseudo details in the face super-resolution process, and thus improves the face super-resolution effect. In addition, the method realizes multi-scale analysis of low-quality face images by performing multi-layer coding processing on low-quality face images in the training set, so that the information contained in the obtained image features can be richer.

[0037] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0038] Figure 3 A training method for a face super-resolution network provided in an embodiment of the present application is shown, wherein the face super-resolution network to be trained comprises at least: an encoding sub-network to be trained, a global attention sub-network to be trained, and a decoding sub-network to be trained.

[0039] like Figure 3 As shown, the method may include:

[0040] S301, using a coding sub-network to perform multi-layer coding processing on low-quality face images in a training set to obtain multi-layer image features of the low-quality face images.

[0041] Exemplarily, the training set may include, for example, a plurality of image pairs consisting of low-quality face images and corresponding real high-quality face images.

[0042] In this step, a coding subnetwork (or encoder) can be used to perform multi-layer coding processing on the low-quality face images in the training set, so as to obtain multi-layer image features of the low-quality face images. The purpose of using the coding subnetwork to perform multi-layer coding processing on the low-quality face images is to map the low-quality face images from the image space to the feature space. From a cognitive perspective, the role of the coding subnetwork is to abstract the image information of the input low-quality face images.

[0043] Exemplarily, the encoding subnetwork may include, for example, a plurality of encoding modules connected in series, for implementing multi-layer encoding processing of low-quality facial images.

[0044] Figure 4 Schematic diagram of the face super-resolution network provided in the embodiment of the present application Figure 1 .like Figure 4 As shown, the encoding subnetwork in the face super-resolution network may include, for example, four encoding modules connected in series, namely encoding module #1 to encoding module #4.

[0045] In some embodiments, the step of using a coding subnetwork to perform multi-layer coding processing on low-quality facial images in a training set may include: 1) using a first coding module to perform coding processing on the convolution result of the low-quality facial images after convolution processing; 2) using each coding module except the first coding module to perform coding processing on the image features output by the previous coding module. The convolution processing of the low-quality facial images may be implemented, for example, using a 3×3 convolution kernel.

[0046] by Figure 4 For example, encoding module #1 (i.e., the first encoding module) can be used to encode the convolution result of the low-quality face image after convolution processing to obtain the corresponding image features; encoding module #2, encoding module #3 and encoding module #4 (i.e., each encoding module except the first encoding module) can be used to encode the image features output by the previous encoding module respectively.

[0047] For example, for low-quality facial images in the training set, convolution processing can be performed on the low-quality facial images first, and then, encoding module #1 can be used to encode the convolution results of the low-quality facial images to obtain corresponding image features; similarly, encoding module #2 can be used to encode the image features output by encoding module #1, encoding module #3 can be used to encode the image features output by encoding module #2, and encoding module #4 can be used to encode the image features output by encoding module #3.

[0048] In some embodiments, each encoding module except the last encoding module may include: a first residual connection submodule and a downsampling submodule; and the last encoding module may include: a first residual connection submodule and an output submodule.

[0049] Figure 5 Schematic diagram of the face super-resolution network provided in the embodiment of the present application Figure 2 .like Figure 5 As shown, each encoding module (encoding module #1 to encoding module #3) except the last encoding module (encoding module #4) may include: a first residual connection submodule and a downsampling submodule; the last encoding module (encoding module #4) may include a first residual connection submodule and an output submodule.

[0050] In some embodiments, the step of using the first encoding module to encode the convolution result of the low-quality facial image after convolution processing may include: 1) using the first first residual connection sub-module to perform residual connection processing on the convolution result of the low-quality facial image after convolution processing; 2) using the first downsampling sub-module to downsample the residual connection result output by the first first residual connection sub-module.

[0051] by Figure 5 For example, the steps of encoding the convolution result of the low-quality face image after convolution processing by using encoding module #1 (i.e., the first encoding module) include: using the first residual connection submodule (i.e., the first residual connection submodule) in encoding module #1 to perform residual connection processing on the convolution result of the low-quality face image after convolution processing, and outputting the corresponding residual connection result; using the downsampling submodule (i.e., the first downsampling submodule) in encoding module #1 to perform downsampling processing on the residual connection result, and outputting the corresponding downsampling result. In encoding module #1, the image features (residual connection results) output by the first residual connection submodule can be used for the subsequent decoding process, and the image features (downsampling results) output by the downsampling submodule can be used as the input image features of encoding module #2.

[0052] In some embodiments, using each encoding module except the first encoding module, the step of encoding the image features output by the previous encoding module may include the following steps 11) to 13):

[0053] 11) Using each first residual connection submodule except the first first residual connection submodule, perform residual connection processing on the downsampling result output by the downsampling submodule in the previous encoding module.

[0054] by Figure 5For example, the first residual connection submodules in encoding modules #2 to #4 (i.e., each first residual connection submodule except the first first residual connection submodule) can be used to perform residual connection processing on the downsampling results output by the downsampling submodule in the previous encoding module. For example, the first residual connection submodule in encoding module #2 can be used to perform residual connection processing on the downsampling results output by the downsampling submodule in encoding module #1, the first residual connection submodule in encoding module #3 can be used to perform residual connection processing on the downsampling results output by the downsampling submodule in encoding module #2, and the first residual connection submodule in encoding module #4 can be used to perform residual connection processing on the downsampling results output by the downsampling submodule in encoding module #3.

[0055] 12) Using each downsampling submodule except the first downsampling submodule, downsampling the residual connection result output by the first residual connection submodule in the current encoding module.

[0056] by Figure 5 For example, the downsampling submodules in encoding module #2 and encoding module #3 (i.e., each downsampling submodule except the first downsampling submodule) can be used to downsample the residual connection result output by the first residual connection submodule in the current encoding module. For example, the downsampling submodule in encoding module #2 can be used to downsample the residual connection result output by the first residual connection submodule in the current encoding module #2, and the downsampling submodule in encoding module #3 can be used to downsample the residual connection result output by the first residual connection submodule in the current encoding module #3. Since encoding module #4 (the last encoding module) does not include a downsampling submodule, encoding module #4 does not need to perform this step.

[0057] 13) Using the output submodule, the residual connection result output by the last residual connection submodule is output as the last layer of image features of the low-quality face image.

[0058] by Figure 5 For example, the output submodule in the encoding module #4 can be used to output the residual connection result output by the first residual connection submodule (i.e., the last first residual connection submodule) in the encoding module #4 as the last layer of image features of the low-quality face image. The last layer of image features can be used as the input image features of the global attention subnetwork.

[0059] It should be understood that Figure 5In the encoding process performed by encoding module #N (N=1, 2, 3 or 4), it is the Nth layer encoding process performed on the low-quality face image. The image features output by encoding module #N can be used as the Nth layer image features of the low-quality face image. Exemplarily, for encoding modules #1 to #3, the image features output by the encoding modules include: the image features output by the first residual connection submodule and the image features output by the downsampling submodule. The image features output by each first residual connection submodule can be used for the subsequent decoding process, and the image features output by each downsampling submodule can be used as the input image features of the next encoding module; for encoding module #4, the output image features are: the image features output by the output submodule, that is, the image features output by the first residual connection submodule (residual connection results). The image features output by encoding module #4 (that is, the last layer of image features of the low-quality face image) can be used as the input image features of the global attention subnetwork.

[0060] In some embodiments, each first residual connection submodule may be composed of M (M is a positive integer) residual blocks (Resblock). As an example, each first residual connection submodule may be composed of two residual blocks connected in series. In this case, the result of processing the input feature using the first residual connection submodule is the result of processing the input feature using the two residual blocks connected in series.

[0061] In some embodiments, the downsampling process performed by each downsampling submodule may be implemented, for example, by using a convolution kernel with a step size of S. As an example, the value of S may be 2, that is, the input feature may be downsampled by using a convolution kernel with a step size of 2. In some embodiments, after each downsampling process is performed, the downsampling result may be further processed by using, for example, a ReLU function.

[0062] It should be noted that Figure 4 and Figure 5 The number of encoding modules in is only an exemplary description for ease of understanding and should not constitute any limitation on the implementation process of the embodiments of the present application. For example, in some other scenarios, the number of encoding modules can be determined according to the complexity of the current face super-resolution task and / or actual needs. In one example, if the complexity of the current task is high, the number of encoding modules can be appropriately increased; if the complexity of the current task is low, the number of encoding modules can be appropriately reduced. In another example, if the current task has high requirements for the real-time performance of the algorithm, a relatively small number of encoding modules can be used.

[0063] According to the method of this embodiment, by performing multi-layer encoding processing on low-quality facial images in the training set, it is beneficial to perform multi-scale analysis on the low-quality facial images, so that the information contained in the obtained image features is richer.

[0064] S302, using the global attention sub-network to extract the global semantic information of the last layer of image features to obtain global features.

[0065] After obtaining the multi-layer image features of low-quality face images, the global attention sub-network in the face super-resolution network can be used to extract the global semantic information of the last layer of image features to obtain global features.

[0066] For example, Figure 5 In the figure, the last layer of image features is the image features output by encoding module #4, which can be used as the input image features of the global attention sub-network. The global attention sub-network can then be used to extract global semantic information from the image features to obtain the corresponding global features.

[0067] In some embodiments, the process of extracting the global semantic information of the last layer of image features using the global attention sub-network is as follows: Figure 6 As shown in FIG. 1 , the global attention subnetwork may include three convolution units, a first operation unit, and a second operation unit. The three convolution units may be represented by three different mapping functions f(), g(), and h(). As an example, the three mapping functions may be composed of three 1x1 convolutions, such as Figure 6 shown.

[0068] Using the global attention sub-network, the process of extracting the global semantic information of the last layer of image features may include the following steps 21) to 23):

[0069] 21) Three convolution units are respectively used to perform convolution processing on the last layer of image features to obtain a first processing result, a second processing result and a third processing result.

[0070] As an example, the last layer of image features of the low-quality face image obtained in S301 is recorded as X. In this step, f(), g() and h() can be used to perform convolution processing on X, so as to map the input image feature X to three different spaces. The obtained results are recorded as key=f(X) (first processing result), query=g(X) (second processing result) and value=h(X) (third processing result).

[0071] 22) Using the first operation unit, perform a matrix multiplication operation on the first processing result (key) and the second processing result (query), and then use the activation function (such as Figure 6 The softmax function in is used to perform nonlinear transformation to obtain the attention map.

[0072] In one implementation, in order to make the dimensions of key and query meet the conditions for matrix multiplication, the key can be transposed, and then the query and the transposed key can be matrix multiplied. In one implementation, after the query and the transposed key are matrix multiplied, the result of the matrix multiplication can also be normalized. As an example, the calculation formula of the attention map can be expressed as: attention map = softmax(g(X)×f(X) T / sqrt(d)). Where d is the dimension of the image feature X; sqrt() is a square root function used to calculate the square root of a non-negative real number; the function of sqrt(d) is to perform normalization.

[0073] Through this step, the global relationship between each pixel in the image and other pixels can be established, thereby capturing the global semantic information in the image.

[0074] 23) Using the second operation unit, perform matrix multiplication operation on the attention map and the third processing result (value) to obtain the global feature.

[0075] That is, the input features are re-weighted using the attention map. out The calculation formula can be expressed as: out =attention map×h(x)=(softmax(g(X)×f(X) T / sqrt(d)))×h(x).

[0076] In one implementation, the result of the matrix multiplication operation on the attention map and the third processing result may be transformed in dimension so that the output global feature matches the required output dimension. Figure 6 In the above example, 1×1 convolution can be used to transform the dimension, so as to obtain the final global feature X out .

[0077] The global attention module in the embodiment of the present application may correspond to the information link in cognitive science, that is, to re-summarize and aggregate the local information in the image. According to the method of this embodiment, by using the global attention module to model the global semantic information, the problem of insufficient global semantic information capture caused by the local link characteristics of the traditional convolutional structure is compensated, and the generation of pseudo details in the face super-resolution process is effectively suppressed, thereby improving the face super-resolution effect. For example, the method can model the global skin information, thereby effectively suppressing the generation of pseudo textures (such as pseudo hair) on the face skin; for another example, the global hair can be modeled to make the repaired hair more natural.

[0078] S303, using a decoding sub-network, combined with multi-layer image features, multi-layer decoding processing is performed on the global features to obtain a prediction result of a high-quality face image corresponding to the low-quality face image.

[0079] In this step, a decoding subnetwork (or decoder) can be used to perform multi-layer decoding processing on the global features output by the global attention subnetwork in S302 in combination with the multi-layer image features obtained by the encoding subnetwork, thereby obtaining a prediction result of a high-quality face image corresponding to the low-quality face image, or in other words, obtaining a repaired high-quality face image.

[0080] Exemplarily, the decoding subnetwork may include a plurality of decoding modules connected in series, and each decoding module may correspond to one encoding module. For example, the last decoding module may correspond to the first encoding module, the second-to-last decoding module may correspond to the second encoding module, the third-to-last decoding module may correspond to the third encoding module, and so on. In one possible manner, the number of decoding modules is the number of encoding modules minus 1, and in this case, the last encoding module has no corresponding decoding module.

[0081] like Figure 4 As shown, the decoding subnetwork in the face super-resolution network may include, for example, three decoding modules connected in series, namely, decoding module #1 to decoding module #3. Each decoding module may correspond to an encoding module. For example, decoding module #1 may correspond to encoding module #3, decoding module #2 may correspond to encoding module #2, and decoding module #3 may correspond to encoding module #1. Figure 4 In the example, the number of decoding modules is the number of encoding modules minus 1, and encoding module #4 has no corresponding decoding module.

[0082] In some embodiments, the steps of using a decoding subnetwork to perform multi-layer decoding processing on global features in combination with multi-layer image features may include: 1) using the first decoding module to decode the global features in combination with the image features output by the encoding module corresponding to the current decoding module; 2) using each decoding module except the first decoding module to decode the decoding result output by the previous decoding module in combination with the image features output by the encoding module corresponding to the current decoding module.

[0083] by Figure 4 For example, decoding module #1 (i.e., the first decoding module) can be used to encode the global features output by the global attention subnetwork in combination with the image features output by encoding module #3 to obtain the corresponding decoding results; decoding module #2 and decoding module #3 (i.e., each decoding module except the first decoding module) can be used to respectively decode the decoding results output by the previous decoding module in combination with the image features output by the encoding module corresponding to the current decoding module.

[0084] For example, for the global features output by the global attention sub-network, decoding module #1 can be first used to decode the global features in combination with the image features output by encoding module #3 corresponding to the current decoding module #1; similarly, decoding module #2 can be used to decode the decoding results output by decoding module #1 in combination with the image features output by encoding module #2 corresponding to the current decoding module #2, and then decoding module #3 can be used to decode the decoding results output by decoding module #2 in combination with the image features output by encoding module #1 corresponding to the current decoding module #3.

[0085] In some embodiments, each decoding module may include an upsampling submodule and a second residual connection submodule, such as Figure 5 As shown. The upsampling process performed by each upsampling submodule can be implemented, for example, using the nearest neighbor (NN) interpolation algorithm, and the upsampling scaling factor is the same as the downsampling scaling factor in the encoding module corresponding to the current decoding module. For example, assuming that the downsampling factor of the downsampling submodule in encoding module #3 is 2 times, then the upsampling factor of the upsampling submodule in decoding module #1 is also 2 times. In some embodiments, after each upsampling process is performed, the upsampling result can be further processed using, for example, a ReLU function.

[0086] In some embodiments, each second residual connection submodule may be composed of M (M is a positive integer) residual blocks (Resblock). As an example, each second residual connection submodule may be composed of 2 residual blocks connected in series,

[0087] In some embodiments, the step of decoding the global features using the first decoding module in combination with the image features output by the encoding module corresponding to the current decoding module may include: 1) using the first upsampling submodule to upsample the global features output by the global attention subnetwork; 2) using the first second residual connection submodule to perform residual connection processing on the result obtained by adding the upsampling result output by the first upsampling submodule and the image features output by the encoding module corresponding to the current decoding module.

[0088] by Figure 5For example, using decoding module #1 (the first decoding module) and combining with the image features output by encoding module #3 (the encoding module corresponding to the current decoding module), the step of decoding the global features may include: using the upsampling submodule (the first upsampling submodule) in decoding module #1 to upsample the global features output by the global attention subnetwork, and outputting the corresponding upsampling results; using the second residual connection submodule (the first second residual connection submodule) in decoding module #1 to perform residual connection processing on the result obtained by adding the upsampling result and the image features output by encoding module #3, and outputting the corresponding residual connection result (i.e., the decoding result of decoding module #1).

[0089] As an implementation, when the second residual connection submodule in the decoding module #1 is used to perform residual connection processing, the image features output by the first residual connection submodule in the encoding module #3 may be combined. For example, before the residual connection processing is performed, the upsampling result output by the upsampling submodule in the decoding module #1 may be added to the image features (residual connection results) output by the first residual connection submodule in the encoding module #3, and then the second residual connection submodule in the decoding module #1 is used to perform residual connection processing on the added result.

[0090] In some embodiments, the step of decoding the decoding result output by the previous decoding module by using each decoding module except the first decoding module and combining the image features output by the encoding module corresponding to the current decoding module may include the following steps 31) and 32):

[0091] 31) Using each upsampling submodule except the first upsampling submodule, upsampling the residual connection result output by the second residual connection submodule in the previous decoding module.

[0092] by Figure 5 For example, the upsampling submodules in decoding module #2 and decoding module #3 (i.e., each upsampling submodule except the first upsampling submodule) can be used to perform upsampling processing on the residual connection result output by the second residual connection submodule in the previous decoding module. For example, the upsampling submodule in decoding module #2 can be used to perform upsampling processing on the residual connection result output by the second residual connection submodule in decoding module #1, and the upsampling submodule in decoding module #3 can be used to perform upsampling processing on the residual connection result output by the second residual connection submodule in decoding module #2.

[0093] 32) Using each second residual connection submodule except the first second residual connection submodule, a residual connection process is performed on a result obtained by adding an upsampling result output by the upsampling submodule in the current decoding module to an image feature output by the encoding module corresponding to the current decoding module.

[0094] by Figure 5 For example, the second residual connection submodules in decoding module #2 and decoding module #3 (i.e., each second residual connection submodule except the first second residual connection submodule) can be used to perform residual connection processing on the result obtained by adding the upsampling result output by the upsampling submodule in the current decoding module to the image feature output by the encoding module corresponding to the current decoding module. In this step, the image feature output by the encoding module corresponding to the current decoding module can be, for example, the image feature output by the first residual connection submodule in the encoding module. For example, the second residual connection submodule in decoding module #2 can be used to perform residual connection processing on the result obtained by adding the upsampling result output by the upsampling submodule in decoding module #2 to the image feature output by the first residual connection submodule in encoding module #2; similarly, the second residual connection submodule in decoding module #3 can be used to perform residual connection processing on the result obtained by adding the upsampling result output by the upsampling submodule in decoding module #3 to the image feature output by the first residual connection submodule in encoding module #1.

[0095] Among them, the residual connection result output by the last second residual connection sub-module after convolution processing is the target decoding result. The result obtained by adding the target decoding result to the low-quality face image is the prediction result of the high-quality face image corresponding to the low-quality face image, or in other words, it is the high-quality face image obtained by super-resolution (restoration) of the low-quality face image.

[0096] by Figure 5 For example, after the second residual connection submodule (the last second residual connection submodule) in decoding module #3 outputs the residual connection result, the residual connection result can be convolved (for example, using a 3×3 convolution kernel), and the convolution result can be used as the target decoding result. Further, by adding the target decoding result to the low-quality face image, the prediction result of the high-quality face image corresponding to the low-quality face image can be obtained.

[0097] In some embodiments, the number of channels of the multi-layer image features of the low-quality facial images shows an increasing trend, that is, the number of channels of the N+1th layer (N is a positive integer) of image features is greater than or equal to the number of channels of the Nth layer of image features, and in the multi-layer image features, the number of channels of the first layer of image features is the smallest, and the number of channels of the last layer of image features is the largest.

[0098] In some embodiments, the number of channels of the decoding results output by the multiple decoding modules is in a decreasing trend. That is, the number of channels of the decoding result output by the N+1th (N is a positive integer)th decoding module is less than or equal to the number of channels of the decoding result output by the Nth decoding module, and among the multiple decoding modules, the number of channels of the decoding result output by the first decoding module is the largest, and the number of channels of the decoding result output by the last decoding module is the smallest.

[0099] S304, determining a first loss based on the prediction result and the actual high-quality face image corresponding to the low-quality face image.

[0100] The first loss is: the loss of the prediction result (the prediction result of the high-quality face image corresponding to the low-quality face image obtained in S303) relative to the real high-quality face image. The first loss can be used to adjust the network parameters of the face super-resolution network to be trained to obtain the target face super-resolution network.

[0101] S305, based on the first loss, adjusting the network parameters of the face super-resolution network to be trained, so that the loss output by the adjusted face super-resolution network meets the convergence condition to obtain the target face super-resolution network.

[0102] Among them, the face super-resolution network can be used to perform face super-resolution on a target low-quality face image to be subjected to face super-resolution, so as to obtain a prediction result of a high-quality face image corresponding to the target low-quality face image.

[0103] Figure 7 FIG. 2 shows a face super-resolution method provided by an embodiment of the present application. Figure 7 As shown, the method may include:

[0104] S701, using a coding sub-network to perform multi-layer coding processing on the target low-quality face image to obtain multi-layer image features of the target low-quality face image.

[0105] Among them, the target low-quality facial image can be obtained, for example, by photographing the target face with an image acquisition device, or a low-quality facial image in a public data set can be used as the target low-quality facial image, which is not limited in this application.

[0106] The implementation method of using the encoding sub-network to perform multi-layer encoding processing on the target low-quality face image can be referred to S301 in the aforementioned embodiment, which will not be repeated here.

[0107] S702, using the global attention sub-network to extract the global semantic information of the last layer of image features to obtain global features.

[0108] In this step, the global attention sub-network can be used to extract the global semantic information of the last layer of image features of the target low-quality face image to achieve modeling of global semantic information. In this way, the problem of insufficient global semantic information capture caused by the local connection characteristics of the traditional convolutional structure can be compensated, thereby effectively suppressing the generation of pseudo details in the face super-resolution process, thereby improving the face super-resolution effect.

[0109] The implementation method of using the global attention sub-network to extract the global semantic information of the last layer of image features of the target low-quality face image can be referred to S302 in the aforementioned embodiment, which will not be repeated here.

[0110] S703, using a decoding sub-network, combined with multi-layer image features, multi-layer decoding processing is performed on the global features to obtain a prediction result of a high-quality face image corresponding to the target low-quality face image.

[0111] In this step, the decoding subnetwork can be used to perform multi-layer decoding processing on the global features output by the global attention subnetwork in S702 in combination with the multi-layer image features obtained by the encoding subnetwork.

[0112] The implementation method of using a decoding sub-network to perform multi-layer decoding processing on the global features in combination with multi-layer image features can refer to S303 in the aforementioned embodiment, which will not be repeated here.

[0113] Combination of the above Figures 1 to 7 A related method for face super-resolution provided in an embodiment of the present application is introduced. To facilitate understanding of the embodiment of the present application, a possible implementation process of the face super-resolution method provided in an embodiment of the present application is introduced below with examples.

[0114] Figure 8 A schematic diagram of an example of using a face super-resolution network to perform face super-resolution on a low-quality face image provided in an embodiment of the present application. Figure 8 The face super-resolution network shown may include an encoding subnetwork (encoder), a global attention subnetwork (or called a global attention module) and a decoding subnetwork (decoder). For simplicity, the encoding subnetwork is referred to as the encoder and the decoding subnetwork is referred to as the decoder.

[0115] The processing flows of the encoder, global attention module, and decoder are introduced below.

[0116] 1) Encoder:

[0117] The encoder can be composed of components such as convolution and ReLU. Its function is to reduce the spatial dimension of the input low-quality face image and map the input image from the image space to the feature space. From a cognitive perspective, this module abstracts the input image information.

[0118] Given an input low-quality face image I in (H, W, C) and encoder Encoder, the calculation process of the encoder can be expressed as: X = Encoder (I in )(H1,W1,C1). Where H, W, and C represent the height, width, and number of channels of the feature map, respectively.

[0119] by Figure 8 For example, for a low-quality face image (image size is 1*512*512), 32 3×3 convolution kernels can be used to convolve the low-quality face image to convert the number of channels to 32. After that, multiple residual connection processes and downsampling processes can be performed alternately. Figure 8 In the encoder, a total of 6 residual connection processes and 5 downsampling processes are performed. Among them, the residual connection process can be implemented by two residual blocks in series, and the downsampling process can be implemented by a convolution kernel with a step size of 2.

[0120] exist Figure 8 In the figure, after the first downsampling process, the number of channels of the feature map changes from 32 to 64, after the third downsampling process, the number of channels of the feature map changes from 64 to 128, and after the fifth downsampling process, the number of channels of the feature map changes from 128 to 256.

[0121] 2) Global Attention Module:

[0122] The encoder can map the input low-quality face image to the feature space, but due to the local connection characteristics of the traditional convolutional structure, the encoder cannot perform global relationship modeling on the input face. To this end, the global attention module can make up for this shortcoming.

[0123] For the input feature X (i.e. the output of the encoder), the processing flow of the global attention module is as follows Figure 6 The processing flow may include:

[0124] a. Through three different mapping functions f(), g() and h(), the input feature X is mapped to three different spaces to obtain query = g(X), key = f(X) and value = h(X). The three mapping functions can be composed of 1x1 convolution, for example.

[0125] b. Calculate the attention map. This step is the core of the entire global attention module. The calculation formula of the attention map can be expressed as: attention map = softmax(g(X) × f(X) T / sqrt(d)). Where d is the dimension of the input feature (e.g. Figure 8For example, d=256). Through this step, the global relationship between each pixel in the image and other pixels can be established, thereby capturing the global semantic information in the image.

[0126] c. Use the attention map to re-weight the input features. This step can be expressed as: out =attentionmap×h(x)=(softmax(g(X)×f(X) T / sqrt(d)))×h(x).

[0127] The global attention module can correspond to the information link in cognition, that is, to re-induct and aggregate the local information in the image.

[0128] 3) Decoder:

[0129] The decoder corresponds to the encoder and can also be composed of components such as convolution and ReLU. The decoder is used to convert X out Re-map it back to the image space to obtain the restored high-quality face image Y. The calculation process of the decoder can be expressed as: Y = Decoder (X out )(H,W,C).

[0130] by Figure 8 For example, for the feature X output by the global attention module out , multiple upsampling processes and residual connection processes can be performed alternately. For example, Figure 8 In the decoder of , a total of 5 upsampling processes and 5 residual connection processes are performed. Among them, the downsampling process can be implemented by the nearest neighbor interpolation algorithm, and the residual connection process can be implemented by two residual blocks connected in series.

[0131] exist Figure 8 In the figure, after the first upsampling process, the number of channels of the feature map changes from 256 to 128, after the third upsampling process, the number of channels of the feature map changes from 128 to 64, and after the fifth upsampling process, the number of channels of the feature map changes from 64 to 32.

[0132] Compared with the basic U-net, the method of the embodiment of the present application can model global semantic information, thereby improving the clarity of the face while suppressing the generation of pseudo textures and improving the problem of unnatural hair after super-resolution. For example, the global skin information can be modeled to effectively suppress the generation of pseudo hair on the face skin; for another example, the global hair can be modeled to make the repaired hair more natural.

[0133] Fig. 9 and Fig.10 A schematic diagram of the face super-resolution result provided in the embodiment of the present application. Fig. 9 and Fig.10In the figure, the left picture shows the face super-resolution result obtained by the method of the embodiment of the present application; the right picture shows the face super-resolution result obtained by the traditional U-net method. The method of the embodiment of the present application can improve the clarity of the input face and suppress the generation of pseudo-hair on the face. Fig.10 It can be seen that the method of the embodiment of the present application can also improve the mosaic problem of the hair, making the repaired hair more natural.

[0134] Based on the above embodiments, the embodiments of the present application provide a device related to a face super-resolution network. The device includes the modules included, and the sub-modules included in each module, which can be implemented by a processor in a computer device with information processing capabilities; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0135] Fig.11 The structure of a training device 1100 for a face super-resolution network provided in an embodiment of the present application is shown. The face super-resolution network to be trained includes at least: an encoding sub-network to be trained, a global attention sub-network to be trained, and a decoding sub-network to be trained. Fig.11 As shown, the apparatus 1100 may include:

[0136] The encoding unit 1101 is used to use the encoding sub-network to perform multi-layer encoding processing on the low-quality face images in the training set to obtain multi-layer image features of the low-quality face images; the extraction unit 1102 is used to use the global attention sub-network to extract the global semantic information of the last layer of image features to obtain global features; the decoding unit 1103 is used to use the decoding sub-network to perform multi-layer decoding processing on the global features in combination with the multi-layer image features to obtain the prediction result of the high-quality face image corresponding to the low-quality face image; the determination unit 1104 is used to determine the first loss based on the prediction result and the real high-quality face image corresponding to the low-quality face image; the adjustment unit 1105 is used to adjust the network parameters of the face super-resolution network to be trained based on the first loss, so that the loss output by the adjusted face super-resolution network meets the convergence condition to obtain the target face super-resolution network.

[0137] In some embodiments, the encoding subnetwork includes multiple encoding modules connected in series; the encoding unit 1101 includes: a first encoding subunit, used to use the first encoding module to encode the convolution result of the low-quality face image after convolution processing; a second encoding subunit, used to use each encoding module except the first encoding module to encode the image features output by the previous encoding module.

[0138] In some embodiments, each encoding module except the last encoding module includes: a first residual connection submodule and a downsampling submodule; the first encoding subunit is specifically used to: use the first first residual connection submodule to perform residual connection processing on the convolution result of the low-quality face image after convolution processing; use the first downsampling submodule to downsample the residual connection result output by the first first residual connection submodule.

[0139] In some embodiments, the last encoding module includes a first residual connection submodule and an output submodule; the second encoding subunit is specifically used to: use each first residual connection submodule except the first first residual connection submodule to perform residual connection processing on the downsampling result output by the downsampling submodule in the previous encoding module; use each downsampling submodule except the first downsampling submodule to perform downsampling processing on the residual connection result output by the first residual connection submodule in the current encoding module; use the output submodule to output the residual connection result output by the last first residual connection submodule as the last layer of image features.

[0140] In some embodiments, the global attention subnetwork includes three convolution units, a first operation unit and a second operation unit; the extraction unit 1102 is specifically used to: use the three convolution units respectively to perform convolution processing on the last layer of image features to obtain a first processing result, a second processing result and a third processing result; use the first operation unit to perform a matrix multiplication operation on the first processing result and the second processing result, and then use the activation function to perform a nonlinear transformation to obtain an attention map; use the second operation unit to perform a matrix multiplication operation on the attention map and the third processing result to obtain a global feature.

[0141] In some embodiments, the decoding subnetwork includes a plurality of decoding modules connected in series, each decoding module corresponding to an encoding module; the decoding unit 1103 includes: a first decoding subunit, used to use the first decoding module, combined with the image features output by the encoding module corresponding to the current decoding module, to decode the global features; a second decoding subunit, used to use each decoding module except the first decoding module, combined with the image features output by the encoding module corresponding to the current decoding module, to decode the decoding result output by the previous decoding module.

[0142] In some embodiments, each decoding module includes an upsampling submodule and a second residual connection submodule; the first decoding subunit is specifically used to: use the first upsampling submodule to upsample the global features; use the first and second residual connection submodules to perform residual connection processing on the result obtained by adding the upsampling result output by the first upsampling submodule and the image features output by the encoding module corresponding to the current decoding module.

[0143] In some embodiments, the second decoding sub-unit is specifically used to: use each upsampling sub-module except the first upsampling sub-module to upsample the residual connection result output by the second residual connection sub-module in the previous decoding module; use each second residual connection sub-module except the first second residual connection sub-module to perform residual connection processing on the result obtained by adding the upsampling result output by the upsampling sub-module in the current decoding module and the image features output by the encoding module corresponding to the current decoding module; wherein, the result obtained after convolution processing of the residual connection result output by the last second residual connection sub-module is the target decoding result, and the result obtained by adding the target decoding result to the low-quality face image is the prediction result of the high-quality face image corresponding to the low-quality face image.

[0144] In some embodiments, the number of channels of the multi-layer image features of the low-quality facial image is increasing; and the number of channels of the decoding results output by the multiple decoding modules is decreasing.

[0145] Fig.12 FIG. 1 shows the composition structure of a face super-resolution device 1200 provided in an embodiment of the present application. Fig.12 As shown, the apparatus 1200 may include:

[0146] The encoding unit 1201 is used to use the encoding sub-network to perform multi-layer encoding processing on the target low-quality face image to obtain multi-layer image features of the target low-quality face image; the extraction unit 1202 is used to use the global attention sub-network to extract the global semantic information of the last layer of image features to obtain global features; the decoding unit 1203 is used to use the decoding sub-network to perform multi-layer decoding processing on the global features in combination with the multi-layer image features to obtain the prediction result of the high-quality face image corresponding to the target low-quality face image.

[0147] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. In some embodiments, the functions or modules included in the device provided in the embodiment of the present application can be used to execute the method described in the above method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0148] It should be noted that in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific hardware, software or firmware, or any combination of hardware, software, and firmware.

[0149] An embodiment of the present application also provides a face super-resolution device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0150] The embodiment of the present application further provides a chip, which includes: a processor, which is used to call and run a computer program from a memory, so that a device equipped with the chip executes some or all of the steps in the above method.

[0151] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, some or all of the steps in the above method are implemented. The computer-readable storage medium can be transient or non-transient.

[0152] An embodiment of the present application also provides a computer program, including a computer-readable code. When the computer-readable code is run in a device (such as a face super-resolution device), the processor in the device executes some or all of the steps in the above method.

[0153] The present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented specifically by hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium, and in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0154] It should be noted here that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. The description of the above device, chip, storage medium, computer program and computer program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiments of the device, chip, storage medium, computer program and computer program product of this application, please refer to the description of the method embodiment of this application for understanding.

[0155] Fig.13 A schematic diagram of a hardware entity of a face super-resolution device in an embodiment of the present application. Fig.13 The face super-resolution device 1300 shown includes a processor 1310, which can call and run a computer program from a memory to implement the method in the embodiment of the present application.

[0156] In some embodiments, Fig.13 As shown, the face super-resolution device 1300 may further include a memory 1320. The processor 1310 may call and run a computer program from the memory 1320 to implement the method in the embodiment of the present application. The memory 1320 may be a separate device independent of the processor 1310, or may be integrated in the processor 1310.

[0157] In some embodiments, Fig.13 As shown, the face super-resolution device 1300 may also include a transceiver 1330, and the processor 1310 may control the transceiver 1330 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 1330 may include a transmitter and a receiver. The transceiver 1330 may further include an antenna, and the number of antennas may be one or more.

[0158] It should be understood that "one embodiment", "an embodiment" or "some embodiments" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment", "in an embodiment" or "in some embodiments" appearing throughout the specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the serial number of each step / process mentioned above does not mean the order of execution, and the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The serial numbers of the embodiments of the present application mentioned above are for description only and do not represent the advantages and disadvantages of the embodiments.

[0159] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0160] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units or modules is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0161] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0162] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0163] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0164] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0165] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A training method for a face super-resolution network, characterized in that: The face super-resolution network to be trained includes at least: an encoding sub-network to be trained, a global attention sub-network to be trained, and a decoding sub-network to be trained; the method includes: Using the encoding subnetwork, performing multi-layer encoding processing on low-quality face images in a training set to obtain multi-layer image features of the low-quality face images; Using the global attention sub-network, extracting the global semantic information of the image features in the last layer to obtain global features; Using the decoding subnetwork, in combination with the multiple layers of the image features, the global features are subjected to multi-layer decoding processing to obtain a prediction result of a high-quality face image corresponding to the low-quality face image; Determine a first loss based on the prediction result and a real high-quality face image corresponding to the low-quality face image; Based on the first loss, the network parameters of the face super-resolution network to be trained are adjusted so that the loss output by the adjusted face super-resolution network meets the convergence condition to obtain the target face super-resolution network.

2. The method according to claim 1, characterized in that The encoding subnetwork includes a plurality of encoding modules connected in series; the encoding subnetwork is used to perform multi-layer encoding processing on low-quality face images in the training set, including: Using the first encoding module, encoding the convolution result of the low-quality face image after convolution processing; Each encoding module except the first encoding module is used to encode the image features output by the previous encoding module.

3. The method according to claim 2, characterized in that Each encoding module except the last encoding module includes: a first residual connection submodule and a downsampling submodule; the first encoding module is used to encode the convolution result of the low-quality face image after convolution processing, including: Using the first residual connection submodule, performing residual connection processing on the convolution result of the low-quality face image after the convolution processing; The first downsampling submodule is used to perform downsampling processing on the residual connection result output by the first residual connection submodule.

4. The method according to claim 3, characterized in that The last encoding module includes a first residual connection submodule and an output submodule; the encoding process of each encoding module except the first encoding module to encode the image features output by the previous encoding module includes: Using each of the first residual connection submodules except the first one of the first residual connection submodules, to perform residual connection processing on the downsampling result output by the downsampling submodule in the previous encoding module; Using each of the downsampling submodules except the first downsampling submodule, downsampling the residual connection result output by the first residual connection submodule in the current encoding module; The output submodule is used to output the residual connection result output by the last residual connection submodule as the image feature of the last layer.

5. The method according to any one of claims 1 to 4, characterized in that The global attention sub-network includes three convolution units, a first operation unit and a second operation unit; The global attention sub-network is used to extract the global semantic information of the image features of the last layer to obtain global features, including: Using the three convolution units respectively, perform convolution processing on the image features of the last layer to obtain a first processing result, a second processing result and a third processing result; Using the first operation unit, performing a matrix multiplication operation on the first processing result and the second processing result, and then using an activation function to perform a nonlinear transformation to obtain an attention map; The second operation unit is used to perform a matrix multiplication operation on the attention map and the third processing result to obtain the global feature.

6. The method according to any one of claims 1 to 4, characterized in that The decoding subnetwork includes a plurality of decoding modules connected in series, each of which corresponds to an encoding module; the decoding subnetwork is used to perform multi-layer decoding processing on the global feature in combination with the multi-layer image feature, including: Using the first decoding module, in combination with the image features output by the encoding module corresponding to the current decoding module, to decode the global features; Each decoding module except the first decoding module is used to decode the decoding result output by the previous decoding module in combination with the image features output by the encoding module corresponding to the current decoding module.

7. The method according to claim 6, characterized in that Each of the decoding modules comprises an upsampling submodule and a second residual connection submodule; The first decoding module is used to decode the global feature in combination with the image feature output by the encoding module corresponding to the current decoding module, including: Using the first upsampling submodule to perform upsampling processing on the global features; The first of the second residual connection submodules is used to perform residual connection processing on a result obtained by adding an upsampling result output by the first upsampling submodule and an image feature output by an encoding module corresponding to the current decoding module.

8. The method according to claim 7, characterized in that The method of using each decoding module except the first decoding module to decode the decoding result output by the previous decoding module in combination with the image features output by the encoding module corresponding to the current decoding module comprises: Using each of the upsampling submodules except the first upsampling submodule, upsampling the residual connection result output by the second residual connection submodule in the previous decoding module; Using each of the second residual connection submodules except the first one of the second residual connection submodules, a residual connection process is performed on a result obtained by adding an upsampling result output by the upsampling submodule in the current decoding module and an image feature output by the encoding module corresponding to the current decoding module; Among them, the result obtained after convolution processing of the last residual connection result output by the second residual connection sub-module is the target decoding result, and the result obtained by adding the target decoding result to the low-quality face image is the prediction result of the high-quality face image corresponding to the low-quality face image.

9. The method according to claim 6, characterized in that The number of channels of the multi-layer image features of the low-quality facial image is increasing; The number of channels of the decoding results output by the multiple decoding modules shows a decreasing trend.

10. A face super-resolution method, characterized in that: The method comprises: Using a coding sub-network, performing multi-layer coding processing on the target low-quality face image to obtain multi-layer image features of the target low-quality face image; A global attention sub-network is used to extract the global semantic information of the image features in the last layer to obtain global features; A decoding subnetwork is used to perform multi-layer decoding processing on the global features in combination with the multi-layer image features to obtain a prediction result of a high-quality face image corresponding to the target low-quality face image.

11. A training device for a face super-resolution network, characterized in that: The face super-resolution network to be trained includes at least: an encoding sub-network to be trained, a global attention sub-network to be trained, and a decoding sub-network to be trained; the device includes: An encoding unit, configured to use the encoding subnetwork to perform multi-layer encoding processing on low-quality face images in a training set to obtain multi-layer image features of the low-quality face images; An extraction unit, configured to extract global semantic information of the image features of the last layer using the global attention sub-network to obtain global features; A decoding unit, configured to use the decoding subnetwork to perform multi-layer decoding processing on the global feature in combination with the multi-layer image features, so as to obtain a prediction result of a high-quality face image corresponding to the low-quality face image; A determining unit, configured to determine a first loss based on the prediction result and a real high-quality face image corresponding to the low-quality face image; An adjustment unit is used to adjust the network parameters of the face super-resolution network to be trained based on the first loss, so that the loss output by the adjusted face super-resolution network meets the convergence condition to obtain a target face super-resolution network.

12. A face super-resolution device, characterized in that: The device comprises: An encoding unit, configured to perform multi-layer encoding processing on a target low-quality face image using an encoding sub-network to obtain multi-layer image features of the target low-quality face image; An extraction unit, used to extract the global semantic information of the image features of the last layer by using a global attention sub-network to obtain global features; A decoding unit is used to use a decoding sub-network to perform multi-layer decoding processing on the global feature in combination with multiple layers of image features to obtain a prediction result of a high-quality face image corresponding to the target low-quality face image.

13. A face super-resolution device, characterized in that: The computer device comprises: A memory for storing computer executable instructions; A processor, connected to the memory, and configured to implement the method of any one of claims 1 to 9, or the method of claim 10, by executing the computer executable instructions.

14. A chip, characterized in that: The chip comprises: A processor, used to call and run a computer program from a memory, so that a device equipped with the chip executes the method described in any one of claims 1 to 9, or executes the method described in claim 10.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by at least one processor, the method according to any one of claims 1 to 9 is implemented, or the method according to claim 10 is implemented.

Images