A human body image super-resolution reconstruction method, system, device and storage medium

Abstract

The present invention discloses a human image super-resolution reconstruction method, system, device and storage medium, wherein the method comprises: performing dimensionality reduction processing on a three-dimensional human data set to obtain a two-dimensional human image data set as a training set; improving the standard convolution by a spatial specific convolution module and a multi-head attention mechanism module to obtain a new convolution; using the new convolution to replace the standard convolution and apply it to the basic image super-resolution network method to obtain an initial network; using the training set to train the initial network to obtain an image super-resolution model; obtaining a low-resolution human image, inputting the low-resolution human image into the image super-resolution model for reconstruction, and obtaining a high-resolution human image. Based on the spatial specific convolution module and the multi-head attention mechanism module, the present invention fully extracts useful features with spatial differences, improves the image reconstruction performance, and can be widely used in image super-resolution technology.

Description

Technical Field

[0001] The present invention relates to image super-resolution technology, and in particular to a human body image super-resolution reconstruction method, system, device and storage medium. Background Art

[0002] With the continuous development of the times, people have higher requirements for the quality of various information around them, especially in recent years when short videos have been booming, people have higher expectations for the clarity and coherence of the picture. Videos are composed of multiple frames of images through correlation in time, so the clear pixels and rich details of the image will bring people more information and a better experience. In recent years, the development of convolutional neural networks (CNN) has pointed out a new direction in the field of computer vision. In the image super-resolution (SISR) task, the introduction of convolutional neural networks not only breaks through the limitations of traditional methods, but also greatly improves the quality of images. By learning the data distribution method to reconstruct the image, a more flexible model can be obtained.

[0003] Dong et al. proposed a simple neural network consisting of only three layers of convolution, called SRCNN. The algorithm directly learns the nonlinear mapping between LR-HR. Due to the introduction of CNN, although it only consists of a shallow network, its performance is good. This is the first time that deep learning has been introduced into the image super-resolution task and has achieved good results, laying the foundation for the continued research of deep learning in this task. At the same time, the author proposed FSRCNN, which allows the network to use a deconvolution layer at the end position, avoiding the overhead of enlarging the image size when inputting to the network processing. The use of deconvolution operations can avoid the network learning in high-dimensional space. The concept of sub-pixel convolution layer proposed by ESPCN can also play the same role. It only enlarges the feature map in the last layer and then rearranges the pixels. Specifically, the number of features output at the end of the convolution layer is consistent with the number of pixels of the high-resolution image to be restored. Rearranging the pixels can restore the high-resolution image.

[0004] In order to further improve the performance, Kim et al. increased the depth of the convolutional layer to 20 layers, introduced global residual learning, and proposed a more accurate VDSR. The deeper network is used to extract more features. The introduction of global residual learning can solve the gradient vanishing or exploding problem that may exist in the network too deep to a certain extent, which further improves the effect of the network. Considering the continuous increase in network depth, even if global residual learning is added, there will still be the problem of gradient vanishing or exploding. Therefore, Tai et al., inspired by ResNet, proposed DRRN, added local residual learning to the network design, and introduced a method including local residual learning. Of course, factors affecting SISR performance also include the diversity and informativeness of features in each layer. Therefore, the attention mechanism module is embedded in the image processing to improve the performance of CNN in image super-resolution tasks in a feature enhancement manner. In RCAN, Zhang et al. applied channel attention to image tasks, using the relationship between channels to treat the features of different channels differently, and finally significantly improved the performance of image super-resolution.

[0005] In addition, with the emergence of generative adversarial networks (GAN) and its achievements in image generation tasks, GAN has attracted the attention of more researchers. Leding et al. applied it to image super-resolution and proposed SRGAN. They also used perceptual loss functions and added VGG pre-training models to make the generated images more natural and in line with human visual perception. However, it has the disadvantages of being difficult to train and converge.

[0006] The performance of image super-resolution has been significantly improved after the adoption of CNN compared to traditional methods. However, as CNN-based methods have been continuously improved, the networks have become more and more complex, and the number of network layers has become deeper and deeper, the computational cost has obviously increased significantly. The expensive computational cost limits their practical use.

[0007] Current image super-resolution methods still have shortcomings. First of all, all methods are implemented using standard convolution, ignoring a large number of spatially diverse features. In addition, in order to improve performance, the network has been deepened and widened, which limits the application scenarios of the methods. How to effectively improve the image super-resolution performance without increasing the number of parameters is a problem that needs to be solved at present. Summary of the invention

[0008] In order to solve at least one of the technical problems existing in the prior art to a certain extent, the present invention aims to provide a method, system, device and storage medium for super-resolution reconstruction of human body images.

[0009] The technical solution adopted by the present invention is:

[0010] A human body image super-resolution reconstruction method comprises the following steps:

[0011] Perform dimensionality reduction processing on the three-dimensional human body data set to obtain a two-dimensional human body image data set as a training set;

[0012] The spatial-specific convolution module and the multi-head attention mechanism module are used to improve the standard convolution to obtain a new type of convolution;

[0013] Use a new type of convolution to replace the standard convolution in the basic image super-resolution network method to obtain the initial network; use the training set to train the initial network to obtain the image super-resolution model;

[0014] Acquire a low-resolution human body image, input the low-resolution human body image into an image super-resolution model for reconstruction, and obtain a high-resolution human body image;

[0015] Among them, the basic image super-resolution network method refers to a deep learning-based method used in image super-resolution tasks to improve performance.

[0016] Furthermore, the dimensionality reduction process is performed on the three-dimensional human body data set to obtain a two-dimensional human body image data set as a training set, including:

[0017] After the dimensionality reduction process of the three-dimensional human body image, a two-dimensional human body image is obtained as a high-resolution image;

[0018] The two-dimensional human body image is downsampled to generate a corresponding low-resolution image;

[0019] The high-resolution images and the corresponding low-resolution images are combined into training pairs to form a training set.

[0020] Furthermore, the spatial specific convolution module uses a different convolution kernel for each pixel by calculating the weight parameters of the convolution kernel, and the process is as follows:

[0021] (1) The convolution kernel function is determined according to the vector of the center pixel and the surrounding pixels, so that each position feature has its own convolution kernel parameters. For a k×k convolution kernel, during the convolution kernel function generation process, the entire kernel generation function is expressed as:

[0022]

[0023] in, represents the kernel generating function, represents the generated convolution kernel, ψ i，j represents a series of pixel locations; is the input feature, c refers to the number of channels of the feature map, h is the height of the feature map, and w is the width of the feature map;

[0024] (2) Perform multiplication and addition operations on the generated convolution kernel and the input features to obtain the final output feature map Defined as:

[0025]

[0026] Among them, k refers to the channel, X i，j，k Represents the input features on a certain channel, It represents the set of neighbor pixels offset relative to the center pixel when convolution is performed on the center pixel. The expression formula is as follows:

[0027]

[0028] Where K represents the convolution kernel size and × represents the Cartesian product operation.

[0029] Furthermore, the generation process of the convolution kernel function is as follows:

[0030] (1) Select a pixel and the same number of surrounding pixels as the convolution kernel size, and use the linear function f(x) to perform linear processing on the k×k×c tensor formed by all channels of these pixels. Project the features of the original pixel into a space suitable for calculating similarity, transform the channel from c to d, and obtain a k×k×d vector; the central pixel corresponds to a 1×1×d vector;

[0031] (2) After the linear function f(x) is applied, l2 normalization is used to eliminate the influence of the vector modulus. The d-dimensional vector of each pixel is used as the feature of the pixel, and the inner product between the vector of the central pixel and all k×k vectors is calculated. The inner product of two vectors after l2 normalization is equivalent to finding the cosine similarity of the two vectors.

[0032] (3) Divide all the obtained convolution kernel weight coefficients by k to keep their values ​​(all convolution kernel weight coefficients) within To ensure the stability of the experiment and the convergence speed of the model during training, a k×k matrix is ​​obtained, which represents the similarity between the central pixel and all the pixels around it. The matrix is ​​the convolution kernel unique to the central pixel.

[0033] The whole process is expressed by the following formula:

[0034]

[0035] Among them, X i，j represents the center pixel of the input, X i+u，j+v represents the neighboring pixels around the central pixel, f represents the linear function, ||f(x)||2 refers to the normalization operation of the result after the linear function processing, and ⊙ represents the inner product.

[0036] Furthermore, the processing process of the multi-head attention mechanism module is as follows:

[0037] In the step of calculating the convolution kernel, multiple vectors are copied for operation. Each vector has head, and after convolution, head 1×1×c vectors are obtained.

[0038] By averaging and concatenating them into a 1×1×head vector, the formula for the multi-head attention mechanism is expressed as:

[0039] MultiHead(Q,K,V)=[mean(head1)||mean(head2)||…||mean(head i )]

[0040]

[0041] Among them, Q represents the center pixel, K and V are the surrounding pixels, W is a 1×1 convolution, and head i It is the calculation method for each single head;

[0042] The output is obtained by linear function Feature map:

[0043] Y=MultiHead(Q,K,V)·g(x)

[0044] Where g(x) is a linear function.

[0045] Furthermore, the novel convolution is used to replace the standard convolution in the basic image super-resolution network method to obtain the initial network, including:

[0046] Use new convolutions to replace standard convolutions in small SAN networks;

[0047] The small SAN network includes 5 residual groups, each of which directly acts on any residual group, integrating the feature expressions within each group. Each residual group includes 5 residual blocks, and each residual block consists of 2 convolutions and 1 activation function.

[0048] Furthermore, the Charbonnier loss function is used as the objective function in the training process of the initial network:

[0049]

[0050] Among them, I SR ,I HRThey represent the reconstructed image and the real high-resolution image respectively, and ε is a constant. The Charbonnier loss adds a regularization term on the basis of the L1 loss, which makes the training loss more stable and can effectively reduce some reconstruction artifacts in the image.

[0051] Another technical solution adopted by the present invention is:

[0052] A human body image super-resolution reconstruction system, comprising:

[0053] A data acquisition module is used to perform dimensionality reduction processing on a three-dimensional human body data set to obtain a two-dimensional human body image data set as a training set;

[0054] The convolution improvement module is used to improve the standard convolution by using the spatial-specific convolution module and the multi-head attention mechanism module to obtain a new type of convolution;

[0055] The model training module is used to use a new type of convolution to replace the standard convolution in the basic image super-resolution network method to obtain an initial network; the initial network is trained using a training set to obtain an image super-resolution model;

[0056] An image reconstruction module is used to obtain a low-resolution human body image, input the low-resolution human body image into an image super-resolution model for reconstruction, and obtain a high-resolution human body image;

[0057] Among them, the basic image super-resolution network method refers to a deep learning-based method used in image super-resolution tasks to improve performance.

[0058] Another technical solution adopted by the present invention is:

[0059] A human body image super-resolution reconstruction device, comprising:

[0060] at least one processor;

[0061] at least one memory for storing at least one program;

[0062] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0063] Another technical solution adopted by the present invention is:

[0064] A computer-readable storage medium stores a program executable by a processor, wherein the program executable by the processor is used to execute the method described above when executed by the processor.

[0065] The beneficial effects of the present invention are: based on the spatial-specific convolution module and the multi-head attention mechanism module, the present invention fully extracts useful features with spatial differences without increasing the complexity of the network, reducing the number of parameters of the model, and using the extracted spatial and channel features to effectively restore the resolution of the image and improve the image reconstruction performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0067] Figure 1 is a flowchart of the steps of a method for super-resolution reconstruction of a human body image in an embodiment of the present invention;

[0068] Figure 2 is a processed two-dimensional human body image in an embodiment of the present invention;

[0069] Figure 3 is a schematic diagram of the structure of a SAN network in an embodiment of the present invention;

[0070] Figure 4 is a schematic diagram of a spatially specific convolution module in an embodiment of the present invention;

[0071] Figure 5 Schematic diagram of a multi-head attention mechanism module in an embodiment of the present invention. DETAILED DESCRIPTION

[0072] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0073] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0074] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0075] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0076] Terminology explanation:

[0077] SAN network: Second-order Attention Network.

[0078] like Figure 1 As shown, this embodiment provides a method for super-resolution reconstruction of human body images, which specifically includes the following steps:

[0079] S1. Perform dimensionality reduction processing on the three-dimensional human body data set to obtain a two-dimensional human body image data set as a training set.

[0080] See also Figure 2 In step S1, the existing three-dimensional human image is subjected to dimensionality reduction processing to obtain a two-dimensional human image as a high-resolution original image, and then the two-dimensional human image is subjected to downsampling operation to generate a corresponding low-resolution image. The two constitute a training pair and constitute an initial training set.

[0081] Specifically, the data set is processed as follows: the existing three-dimensional human body image is subjected to dimensionality reduction processing to obtain a two-dimensional human body image, and the two-dimensional data is processed through matlab to obtain the corresponding low-resolution image, which includes images with scale factors of 2, 3, and 4. Taking the multiple of 2 as an example, the two-dimensional human body image is downsampled to obtain the corresponding low-resolution image, and the size of the low-resolution image is half of the original image. Finally, the low-resolution image is enlarged by 2 times through network prediction to reconstruct a high-resolution image with the same size as the original image.

[0082] S2. Improve the standard convolution with the spatial-specific convolution module and the multi-head attention mechanism module to obtain a new type of convolution.

[0083] Build a network: In this embodiment, a small SAN is used as the basic structure of the network. The network structure of the SAN is as follows: Figure 3 shown.

[0084] See also Figure 3 This embodiment adopts a small SAN network, which includes 5 residual groups. Each residual group directly acts on any residual group, integrating the feature expressions within each group. Each residual group has 5 residual blocks, and each residual block consists of 2 convolutions and 1 activation function. At the same time, the new convolution replaces 0.6 times the standard convolution, and all are replaced at the tail position.

[0085] The new convolution is composed of a spatial-specific convolution module and a multi-head attention mechanism module. Figure 4 and Figure 5 The following are schematic diagrams of these two modules. Standard convolution only expresses channel-specific features, but because the interactive information on the channel is relatively redundant, it not only completely ignores the spatial feature information, but also extracts redundant features. The proposed spatial-specific convolution module mainly makes full use of the spatially differentiated features, so that the network can have its own convolution kernel for each pixel position, so that the entire model can extract effective spatial features, which is conducive to the task of image super-resolution reconstruction. The introduction of the multi-head attention mechanism allows the network to pay attention to spatial features without ignoring too many channel features. Channel features are also an indispensable part of image reconstruction tasks. Therefore, the multi-head attention mechanism allows the network to pay attention to the interactive information on the channel to a certain extent, which promotes the task.

[0086] In this embodiment, the spatial specific convolution module uses different convolution kernels to process each pixel by calculating the weight parameters of the convolution kernel. The use of spatial specific convolution allows more attention to the differences in spatial features, more effective learning of real features, and can reduce the number of parameters and calculations of most networks that increase depth in order to improve performance. The process is as follows:

[0087] (1) Determine the convolution kernel function based on the vector of the center pixel and the surrounding pixels, so that each position feature has its own convolution kernel parameters. In the features, c refers to the number of channels of the feature map, h is the height of the feature map, and w is the width of the feature map. Taking the k×k convolution kernel as an example, in the process of generating the convolution kernel function, we can express the entire kernel generation function as:

[0088]

[0089] in represents the kernel generating function, represents the generated convolution kernel, ψ i，j represents a series of pixel locations;

[0090] (2) Perform multiplication and addition operations on the generated convolution kernel and the input features to obtain the final output feature map It can be defined as:

[0091]

[0092] Where k refers to the channel, X i，j，k Represents the input features on a certain channel, It represents the set of neighbor pixels offset relative to the center pixel when convolving the center pixel, which can be expressed by the following formula:

[0093]

[0094] The generation of the convolution function is the key step of spatial specific convolution. The specific generation process of the convolution function is as follows:

[0095] (1) First, select a pixel and the same number of surrounding pixels as the convolution kernel size, and use the linear function f(x) to perform linear processing on all channels of these pixels to project the features of the original pixel into a space suitable for calculating similarity. At this time, the channel can be transformed from c to d to obtain a k×k×d vector. The central pixel is also processed separately to obtain a 1×1×d vector.

[0096] (2) After the linear function f(x) is applied, l2 normalization is used to eliminate the influence of the vector modulus. By taking the d-dimensional vector of each pixel as its feature, we calculate the inner product between the vector of the central pixel and all k×k vectors. The inner product of two vectors after l2 normalization is equivalent to finding the cosine similarity of the two vectors. Here, the influence of surrounding pixels on the central pixel is taken into account.

[0097] (3) Divide all elements by k to keep their values ​​in The stability of the experiment and the convergence speed of the model during training can be guaranteed. Now we get a k×k matrix, which represents the similarity between the central pixel and all the pixels around it. This matrix is ​​the convolution kernel unique to this central pixel.

[0098] The whole process can be expressed by the following formula:

[0099]

[0100] Where X i，j represents the center pixel of the input, X i+u，j+v represents the neighboring pixels around the pixel, f represents the linear function, ||f(x)||2 refers to the normalization operation of the result after the linear function processing, and ⊙ represents the inner product.

[0101] In this embodiment, the multi-head attention mechanism module is implemented by using multiple heads in the convolution kernel calculation process, and the multi-head attention mechanism is used to explore and express attention from different angles. For the image super-resolution reconstruction task, the multi-head mechanism is used in the form of generating and realizing channel specificity on the channel, and at the same time, the input feature is multiplied by a multi-dimensional vector to introduce channel specificity, so that the network can adaptively adjust the convolution kernel parameter weight coefficient. The specific processing process is as follows:

[0102] First, copy multiple vectors in the step of calculating the convolution kernel for operation. Each vector has head, and after convolution, head 1×1×c vectors are obtained;

[0103] Then, by averaging and concatenating them into a 1×1×head vector, the formula for the multi-head attention mechanism can be expressed as:

[0104] MultiHead(Q,K,V)=[mean(head1)||mean(head2)||…||mean(head i )]

[0105]

[0106] Where Q represents the center pixel, K and V are the surrounding pixels, W is a 1×1 convolution, and head i This is the calculation method for each single head.

[0107] Finally, the output is obtained through the linear function Feature map:

[0108] Y=MultiHead(Q,K,V)·g(x)

[0109] Where g(x) is a linear function.

[0110] The multi-head attention mechanism module and the space-specific convolution module are applied to the convolution calculation process to obtain a new type of convolution, which can not only extract channel features but also focus on spatial features. The new type of convolution replaces the standard convolution to form the final training model, which can reconstruct more realistic high-resolution images.

[0111] S3. Use the new convolution to replace the standard convolution in the basic image super-resolution network method to obtain the initial network; use the training set to train the initial network to obtain the image super-resolution model. Among them, the basic image super-resolution network method refers to some deep learning-based methods used to improve the performance of image super-resolution tasks, in which the convolution calculation is implemented by the standard convolution. The new convolution is independent and can be easily applied in these methods.

[0112] Training model: Use the finalized network model to train the processed data set. The purpose of training is to make the generated image as close to the original image as possible. The trained network can be used as a model for reconstructing the image.

[0113] In this embodiment, the image is randomly cropped to 64×64 before the data set is put into the network training as input. In order to make the network more robust, basic data enhancement operations are performed on the image before the training set enters the network, including random vertical flipping, horizontal flipping and transposition, which not only increases the diversity of the data so that the model can learn a better distribution, but also enables the model to have generalization capabilities. In addition, the small batch gradient descent method is used during training, and the experiment adopts an end-to-end training method, which does not require any pre-training links.

[0114] In this embodiment, the entire network uses the Charbonnier loss function as the objective function during training:

[0115]

[0116] Among them I SR ,I HR Represent the reconstructed image and the real high-resolution image respectively, and ε is a very small constant. Charbonnier loss adds a regularization term on the basis of L1 loss, which makes the training loss more stable and can effectively reduce some reconstruction artifacts in the image.

[0117] S4. Obtain a low-resolution human body image, input the low-resolution human body image into an image super-resolution model for reconstruction, and obtain a high-resolution human body image.

[0118] Prediction results: For the low-resolution human image to be enlarged, the trained image super-resolution model is used to reconstruct the image, and finally a high-resolution human image that is enlarged several times and the same size as the original image is obtained, completing the image reconstruction task. When reconstructing the image, in order not to be limited by the video memory, we cut a single low-resolution image into four equal parts (vertically and horizontally divided into two equal parts), and pass them through the network to obtain four high-resolution images, which are then spliced ​​into the final output.

[0119] In summary, the method of this embodiment has the following advantages and beneficial effects compared with the prior art:

[0120] (1) In order to solve the existing problems of ignoring spatial difference features and large number of parameters caused by too deep network, the present invention proposes an image super-resolution method based on multi-head attention mechanism and spatial specificity. A new convolution is used to replace the standard convolution to extract features at multiple levels. It combines spatial specificity to obtain different spatial information and associates the convolution kernel parameters with the influence between pixels. It generates appropriate kernels for all pixels, so that each feature has its own convolution parameters, which can further learn more realistic features.

[0121] (2) The multi-head mechanism is used to introduce more learning opportunities for information in the frequency domain, and effective features can be extracted in both space and channels. A variable vector is added to autonomously adjust the weight coefficient, which can adaptively adjust the convolution kernel parameters. These two improvements allow more specificity to be reflected in the channel before and after convolution.

[0122] (3) The entire new convolution method can be regarded as an independent model and can be used in any other network without changing its structure. It reflects the convenience of overall replacement of standard convolution and has a plug-and-play effect.

[0123] (4) Due to the convolutional structural design and operation, the number of learnable parameters and the amount of calculation can be significantly reduced, which can be applied in more scenarios and is more convenient to store.

[0124] This embodiment also provides a human body image super-resolution reconstruction system, including:

[0125] A data acquisition module is used to perform dimensionality reduction processing on a three-dimensional human body data set to obtain a two-dimensional human body image data set as a training set;

[0126] The convolution improvement module is used to improve the standard convolution by using the spatial-specific convolution module and the multi-head attention mechanism module to obtain a new type of convolution;

[0127] The model training module is used to use a new type of convolution to replace the standard convolution in the basic image super-resolution network method to obtain an initial network; the initial network is trained using a training set to obtain an image super-resolution model;

[0128] An image reconstruction module is used to obtain a low-resolution human body image, input the low-resolution human body image into an image super-resolution model for reconstruction, and obtain a high-resolution human body image;

[0129] Among them, the basic image super-resolution network method refers to a deep learning-based method used in image super-resolution tasks to improve performance.

[0130] A human image super-resolution reconstruction system of the present embodiment can execute a human image super-resolution reconstruction method provided by the method embodiment of the present invention, can execute any combination of implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.

[0131] This embodiment also provides a human body image super-resolution reconstruction device, including:

[0132] at least one processor;

[0133] at least one memory for storing at least one program;

[0134] When the at least one program is executed by the at least one processor, the at least one processor implements Figure 1 The method shown.

[0135] A human body image super-resolution reconstruction device of this embodiment can execute a human body image super-resolution reconstruction method provided by the method embodiment of the present invention, can execute any combination of implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.

[0136] The present application also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes Figure 1 The method shown.

[0137] This embodiment also provides a storage medium storing instructions or programs that can execute a method for super-resolution reconstruction of human images provided by an embodiment of the method of the present invention. When the instructions or programs are run, any combination of implementation steps of the method embodiment can be executed, and the corresponding functions and beneficial effects of the method can be obtained.

[0138] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.

[0139] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0140] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0141] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0142] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0143] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0144] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0145] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0146] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A human body image super-resolution reconstruction method, characterized in that: The following steps are involved: Perform dimensionality reduction processing on the three-dimensional human body data set to obtain a two-dimensional human body image data set as a training set; The spatial-specific convolution module and the multi-head attention mechanism module are used to improve the standard convolution to obtain a new type of convolution; Use a new type of convolution to replace the standard convolution in the basic image super-resolution network method to obtain the initial network; The initial network is trained using the training set to obtain an image super-resolution model; Acquire a low-resolution human body image, input the low-resolution human body image into an image super-resolution model for reconstruction, and obtain a high-resolution human body image; Among them, the basic image super-resolution network method refers to a deep learning-based method used in image super-resolution tasks to improve performance; The spatial specific convolution module calculates the weight parameters of the convolution kernel and uses a different convolution kernel for each pixel. The process is as follows: (1) The convolution kernel function is determined according to the vector of the center pixel and the surrounding pixels, so that each position feature has its own convolution kernel parameters. For a k×k convolution kernel, during the convolution kernel function generation process, the entire kernel generation function is expressed as: in, represents the kernel generating function, represents the generated convolution kernel, ψ i,j represents a series of pixel locations; is the input feature, c refers to the number of channels of the feature map, h is the height of the feature map, and w is the width of the feature map; (2) Perform multiplication and addition operations on the generated convolution kernel and the input features to obtain the final output feature map Defined as: Among them, k refers to the channel, X i+u,j+v,k Represents the input features on a certain channel, It represents the set of neighbor pixels offset relative to the center pixel when convolution is performed on the center pixel. The expression formula is as follows: Where K represents the convolution kernel size and × represents the Cartesian product operation.

2. The method for super-resolution reconstruction of human body images according to claim 1, characterized in that: The dimensionality reduction process is performed on the three-dimensional human body data set to obtain a two-dimensional human body image data set as a training set, including: After the dimensionality reduction process of the three-dimensional human body image, a two-dimensional human body image is obtained as a high-resolution image; The two-dimensional human body image is downsampled to generate a corresponding low-resolution image; The high-resolution images and the corresponding low-resolution images are combined into training pairs to form a training set.

3. A human body image super-resolution reconstruction method according to claim 2, characterized in that: The generation process of the convolution kernel function is as follows: (1) Select a pixel and the same number of surrounding pixels as the convolution kernel size, and use the linear function f(x) to perform linear processing on the k×k×c tensor formed by all channels of these pixels. Project the features of the original pixel into a space suitable for calculating similarity, transform the channel from c to d, and obtain a k×k×d vector; the central pixel corresponds to a 1×1×d vector; (2) After the linear function f(x) is applied, l2 normalization is used to eliminate the influence of the vector modulus. The d-dimensional vector of each pixel is used as the feature of the pixel, and the inner product between the vector of the central pixel and all k×k vectors is calculated. The inner product of two vectors after l2 normalization is equivalent to finding the cosine similarity of the two vectors. (3) Divide all the obtained convolution kernel weight coefficients by k to keep their values ​​within To ensure the stability of the experiment and the convergence speed of the model during training, a k×k matrix is ​​obtained, which represents the similarity between the central pixel and all the pixels around it. The matrix is ​​the convolution kernel unique to the central pixel. The whole process is expressed by the following formula: Among them, X i,j represents the center pixel of the input, X i+u,j+v represents the neighboring pixels around the central pixel, f represents the linear function, ‖f(x)‖2 refers to the normalization operation of the result after the linear function processing, and ⊙ represents the inner product.

4. The method for super-resolution reconstruction of human body images according to claim 3, characterized in that: The processing process of the multi-head attention mechanism module is as follows: In the step of calculating the convolution kernel, multiple vectors are copied for operation. Each vector has head, and after convolution, head 1×1×c vectors are obtained. By averaging and concatenating them into a 1×1×head vector, the formula for the multi-head attention mechanism is expressed as: MultiHead(Q,K,V)=[mean(head1)||mean(head2)||…||mean(head i )] Among them, Q represents the center pixel, K and V are the surrounding pixels, W is a 1×1 convolution, and head i It is the calculation method for each single head; The output is obtained by linear function Feature map: Y=MultiHead(Q,K,V)·g(x) Where g(x) is a linear function.

5. The method for super-resolution reconstruction of human body images according to claim 4, characterized in that: The novel convolution is used to replace the standard convolution in the basic image super-resolution network method to obtain the initial network, including: Use new convolutions to replace standard convolutions in small SAN networks; The small SAN network includes 5 residual groups, each of which directly acts on any residual group, integrating the feature expressions within each group. Each residual group includes 5 residual blocks, and each residual block consists of 2 convolutions and 1 activation function.

6. The method for super-resolution reconstruction of human body images according to claim 5, characterized in that: The Charbonnier loss function is used as the objective function in the process of training the initial network: Among them, I SR ,I HR represent the reconstructed image and the true high-resolution image respectively, and ε is a constant.

7. A human body image super-resolution reconstruction system, applied to a human body image super-resolution reconstruction method according to any one of claims 1 to 6, characterized in that: include: A data acquisition module is used to perform dimensionality reduction processing on a three-dimensional human body data set to obtain a two-dimensional human body image data set as a training set; The convolution improvement module is used to improve the standard convolution by using the spatial-specific convolution module and the multi-head attention mechanism module. Get a new type of convolution; The model training module is used to use a new type of convolution to replace the standard convolution in the basic image super-resolution network method to obtain an initial network; the initial network is trained using a training set to obtain an image super-resolution model; An image reconstruction module is used to obtain a low-resolution human body image, input the low-resolution human body image into an image super-resolution model for reconstruction, and obtain a high-resolution human body image; Among them, the basic image super-resolution network method refers to a deep learning-based method used in image super-resolution tasks to improve performance.

8. A human body image super-resolution reconstruction device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to perform the method according to any one of claims 1 to 6 when executed by the processor.

Images