Human pose transformation method based on attention mechanism

Abstract

The application discloses a human body posture conversion method based on an attention mechanism, which is characterized in that a source human body image and a target posture image are sent into a generator to be converted into a target human body image of the source human body image under the target posture image; a self-attention module of all semantic styles in the source human body image is constructed; a cross-attention module between all semantic styles in the source human body image and the target posture image is constructed, and a target semantic segmentation image is generated by using a cross-attention matrix in the cross-attention module; the self-attention module and the cross-attention module are embedded into the generator, a generative adversarial network is trained, and human body posture conversion is realized. Compared with the prior art, the application has the characteristics that the target human body image and the target semantic segmentation image can be simultaneously generated in one training stage, the image quality is significantly improved, the method is simple, the efficiency is high, and the problem that the generated image quality is poor due to the lack of efficient fusion between the source human body image and the target posture is better solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical fields of computer vision and digital image processing, and particularly relates to a human pose transformation method based on an attention mechanism. BACKGROUND

[0002] At present, human pose transformation has many potential applications, such as pedestrian re-identification, virtual clothes fitting, etc. The current methods usually use Squeeze-and-Extraction (SE) or Adaptive Instance Normalization (AdaIN) to fuse the source human image and the target pose image, however, these methods lack the ability to align the source human image and the target pose image; or use two-dimensional or three-dimensional deformation to estimate the correspondence between the source pose image and the target pose image, thereby guiding the distribution of the source human image under the target pose image, these methods can generate real textures, but when facing huge deformation, they may produce obvious artifacts.

[0003] In addition, there are also methods that use cross-attention mechanisms to calculate the dense correlation matrix between the source human image and the target pose image, but each target position is only associated with a small area in the source human image, which means that the dense correlation matrix should be a sparse matrix, and such a dense correlation matrix will lead to quadratic memory consumption.

[0004] Therefore, an efficient method for fusing the source human image and the target pose image using the attention mechanism is necessary. SUMMARY

[0005] The present application aims at the deficiencies of the prior art and provides a human pose transformation method based on an attention mechanism, which embeds an attention module into a human pose transformation network to improve the quality of generated images: a source human image and a target pose image are input into a generator to transform the source human image into a target human image under the target pose image; a self-attention module for all semantic styles in the source human image is constructed; a cross-attention module between all semantic styles in the source human image and the target pose image is constructed, and a cross-attention matrix in the cross-attention module is used to generate a target semantic segmentation map; the self-attention module and the cross-attention module are embedded into the generator, a generative adversarial network is trained, human pose transformation is realized, and the image quality is significantly improved, the method is simple and efficient, and it better solves the problem of poor image quality caused by the lack of efficient fusion between the source human image and the target pose image.

[0006] The present application is achieved in this way:

[0007] The application discloses a human body posture conversion method based on an attention mechanism, which is characterized in that an attention module is embedded into a human body posture conversion network, a source human body image and a target posture image are sent into a generator to be converted into a target human body image of the source human body image under the target posture image; a self-attention module of all semantic styles in the source human body image is constructed; a cross-attention module between all semantic styles in the source human body image and the target posture image is constructed, and a target semantic segmentation image is generated by using a cross-attention matrix in the cross-attention module; the self-attention module and the cross-attention module are embedded into the generator, a generative adversarial network is trained, and human body posture conversion is realized, and the method comprises the following steps:

[0008] Step 1: sending the source human body image and the target posture image into the generator to be converted into the target human body image of the source human body image under the target posture image;

[0009] Step 2: constructing the self-attention module of all semantic styles in the source human body image;

[0010] Step 3: constructing the cross-attention module between all semantic styles in the source human body image and the target posture image, and generating the target semantic segmentation image by using the cross-attention matrix in the cross-attention module;

[0011] Step 4: embedding the self-attention module and the cross-attention module into the generator, training a generative adversarial network, and realizing human body posture conversion.

[0012] The generator is composed of a posture encoder, a style encoder, an AdaIN module and a decoder, wherein the target posture image is input into the posture encoder to obtain a target posture feature, the source human body image and a source semantic segmentation image containing K semantics are input into the style encoder to obtain K semantic style encodings, then the K semantic style encodings are injected into the target posture feature in the form of side branches through the AdaIN module to obtain a preliminary fused feature F crs , and finally the preliminary fused feature F crs is input into the decoder to generate the target human body image of the source human body image under the target posture image.

[0013] The step 2 specifically comprises:

[0014] Step 2-1: taking the K semantic style encodings as query values query, key values key and value items value, wherein the dimension sizes of the query values query, the key values key and the value items value are all K*C, the similarity between the K semantic style encodings is calculated, that is, the query values query and the key values key are multiplied, and a self-attention matrix AM S is obtained after being processed by a softmax function, and the dimension size of the self-attention matrix AM S is K*K;

[0015] Step 2-2: multiply the self-attention matrix AM S with the value item value matrix to obtain the output self_out of the self-attention mechanism, which has a dimension size of K*C;

[0016] Step 2-3: add the output self_out of the self-attention mechanism and the K semantic style encodings to obtain the residual output residual_self of the self-attention mechanism, which has a dimension size of K*C;

[0017] The step 3 specifically comprises:

[0018] Step 3-1: add the preliminarily fused feature F crs as the query value query, which has a dimension size of HW*C, and the residual output residual_self of the self-attention mechanism as the key value key and the value item value, which has a dimension size of K*C, to obtain the similarity between the preliminarily fused feature F crs and the residual output residual_self of the self-attention mechanism, that is, the query value query and the key value key matrix are multiplied, and the cross-attention matrix AM is obtained after the softmax function processing, which has a dimension size of HW*K;

[0019] Step 3-2: multiply the cross-attention matrix AM and the value item value matrix to obtain the output cross_out of the cross-attention mechanism, which has a dimension size of HW*C;

[0020] Step 3-3: add the output cross_out of the cross-attention mechanism and the preliminarily fused feature F crs to obtain the residual output residual_cross of the cross-attention mechanism, which has a dimension size of HW*C;

[0021] Step 3-4: input the semantic segmentation graph of the source human body image under the target posture image as the real label GroudTruth, and use the cross-entropy loss to constrain the cross-attention matrix AM, so as to generate the target semantic segmentation graph by using the cross-attention matrix AM.

[0022] The attention module is embedded into the generator in the following manner: the self-attention module and the cross-attention module are connected after the AdaIN module and before the decoder.

[0023] Compared with the prior art, the present application has the characteristics that the target human body image and the target semantic segmentation graph can be generated simultaneously only by using one training stage, the quality of the human body image and the accuracy of the semantic segmentation graph are significantly improved, the method is simple, efficient, and saves the display memory, and the problem of poor image quality caused by the lack of efficient fusion between the source human body image and the target posture image is solved. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 Flowchart of the present application;

[0025] Figure 2 Schematic diagram of all modules in the generator of the present application;

[0026] Figure 3 Schematic diagram of the self-attention module of the present application;

[0027] Figure 4 Schematic diagram of the cross-attention module of the present application;

[0028] Figure 5 Schematic diagram of the generator embedding the self-attention module and the cross-attention module of the present application. DETAILED DESCRIPTION

[0029] In order to more clearly illustrate the technical means, technical improvements and beneficial effects of the present application, the present application will be described in detail below with reference to the accompanying drawings.

[0030] The present application discloses a human pose conversion method based on an attention mechanism, comprising the following specific steps:

[0031] Step 1: input the source human image and the target pose image into the generator to convert the source human image into a target human image under the target pose image;

[0032] Step 2: construct a self-attention module for all semantic styles in the source human image;

[0033] Step 3: construct a cross-attention module between all semantic styles in the source human image and the target pose image, and generate a target semantic segmentation map using the cross-attention matrix in the cross-attention module;

[0034] Step 4: embed the self-attention module and the cross-attention module into the generator, train a generative adversarial network, and realize human pose conversion.

[0035] The specific steps of step 1 are as follows: input the target pose image into a pose encoder to obtain target pose features, input the source human image and a source semantic segmentation map containing K semantics into a style encoder to obtain K semantic style encodings, then inject the K semantic style encodings into the target pose features in the form of side branches through an AdaIN module to obtain preliminary fused features F crs , and finally input the preliminary fused features F crs into a decoder to generate a target human image of the source human image under the target pose image.

[0036] The step 2 specifically comprises the following steps:

[0037] Step 2-1: K semantic style encodings are all taken as query value query, key value key and value item value, wherein the dimension sizes of query value query, key value key and value item value are all K*C, similarity between K semantic style encodings is calculated, that is, query value query is multiplied with key value key matrix, and after being processed by a softmax function, a self-attention matrix AM S with a dimension size of K*K is obtained.

[0038] Step 2-2: The self-attention matrix AM S is multiplied with value item value matrix, and output self_out of the self-attention mechanism with a dimension size of K*C is obtained.

[0039] Step 2-3: The output self_out of the self-attention mechanism is added with K semantic style encodings, and residual output residual_self of the self-attention mechanism with a dimension size of K*C is obtained.

[0040] The step 2-1 is specifically: K semantic style encodings are taken as query value query, key value key and value item value, and the dimension sizes are K*C, and the self-attention matrix AM S is obtained by the following formula a:

[0041] AM S =softmax(query×key T ) (a);

[0042] Wherein: the dimension size of AM S is K*K; × represents matrix multiplication; and softmax(·) represents a softmax function.

[0043] The step 2-2 is specifically: the AM S obtained in step 2-1 is multiplied with value item value matrix, and output self_out of the self-attention mechanism with a dimension size of K*C is obtained, and the calculation of the output self_out of the self-attention mechanism is represented by the following formula b:

[0044] self_out=AM S ×value (b);

[0045] Wherein: the dimension size of self_out is K*C; and × represents matrix multiplication.

[0046] The step 2-3 is specifically: K semantic style encodings F sAdd self_out obtained in step 2-2 to obtain residual output residual_self of the self-attention mechanism, and the calculation of the residual output residual_self of the self-attention mechanism is represented by the following formula c:

[0047] residual_self = F s + self_out (c).

[0048] Wherein: the dimension size of residual_self is K*C; + represents element-wise addition.

[0049] The step 3 specifically comprises the following steps:

[0050] Step 3-1: input the preliminarily fused feature F crs as a query value query, the dimension size of which is HW*C, and the residual output residual_self of the self-attention mechanism as a key value key and a value item value, the dimension size of which is K*C, to calculate the preliminarily fused feature F crs and the residual output residual_self of the self-attention mechanism, that is, the query value query and the key value key matrix are multiplied, and the cross-attention matrix AM is obtained after the softmax function processing, the dimension size of which is HW*K.

[0051] Step 3-2: multiply the cross-attention matrix AM and the value item value matrix to obtain the output cross_out of the cross-attention mechanism, the dimension size of which is HW*C.

[0052] Step 3-3: add the output cross_out of the cross-attention mechanism and the preliminarily fused feature F crs to obtain the residual output residual_cross of the cross-attention mechanism, the dimension size of which is HW*C.

[0053] Step 3-4: input the semantic segmentation graph of the source human body image under the target posture image as the real label GroudTruth, and use the cross-entropy loss to constrain the cross-attention matrix AM, so as to generate the target semantic segmentation graph by using the cross-attention matrix AM.

[0054] The step 3-1 specifically comprises: inputting the preliminarily fused feature F crs as a query value query, the dimension size of which is HW*C, and the residual output residual_self of the self-attention mechanism as a key value key and a value item value, the dimension size of which is K*C, to obtain the cross-attention matrix AM by the following formula d:

[0055] AM = softmax(query x key T ) (d);

[0056] wherein: the dimension size of AM is HW*K; x represents matrix multiplication; softmax(·) represents a softmax function.

[0057] The step 3-2 is specifically: multiplying the AM obtained in the step 3-1 with a value matrix to obtain cross attention mechanism output cross_out, the dimension size of which is HW*C, and the calculation of the cross attention mechanism output cross_out is represented by the following e formula:

[0058] cross_out = AM x value (e);

[0059] wherein: the dimension size of cross_out is HW*K; x represents matrix multiplication.

[0060] The step 3-3 is specifically: adding the cross_out obtained in the step 3-2 with the preliminary fused feature F crs to obtain cross attention mechanism residual output residual_cross, the dimension size of which is HW*C, and the calculation of the cross attention mechanism residual output residual_cross is represented by the following f formula:

[0061] residual_cross = F crs + cross_out (f);

[0062] wherein: the dimension size of residual_cross is HW*C; + represents element-wise addition.

[0063] The step 3-4 is specifically: inputting the semantic segmentation graph of the source human body image under the target posture image as the real label GroudTruth, using a cross entropy loss to constrain the cross attention matrix AM obtained in the step 3-3, so as to generate the target semantic segmentation graph by using the cross attention matrix AM, and the calculation of the cross entropy loss is represented by the following g formula respectively:

[0064]

[0065] wherein: i represents the position in the spatial dimension of the cross attention matrix AM, and j represents the position in the semantic dimension of the cross attention matrix AM.

[0066] Embodiment 1

[0067] Reference Figure 1The application embeds the attention module into the human body posture conversion network, inputs the source human body image and the target posture image into the generator to convert into the target human body image of the source human body image under the target posture image; a self-attention module of all semantic styles in the source human body image is constructed; a cross-attention module between all semantic styles in the source human body image and the target posture image is constructed, and a target semantic segmentation image is generated by using the cross-attention matrix in the cross-attention module; the self-attention module and the cross-attention module are embedded into the generator, a generative adversarial network is trained, and human body posture conversion is realized, specifically including the following steps:

[0068] S1: inputting the source human body image and the target posture image into the generator to convert into the target human body image of the source human body image under the target posture image;

[0069] Referring to Figure 2 , the step specifically includes:

[0070] Step 0: inputting the target posture image into a posture encoder to obtain target posture features;

[0071] Step 1: inputting the source human body image and the source semantic segmentation image containing K semantics into a style encoder to obtain K semantic style encodings;

[0072] Step 2: injecting the K semantic style encodings into the target posture features in the form of side branches through an AdaIN module to obtain preliminary fused features F crs ;

[0073] Step 3: inputting the preliminary fused features F crs into a decoder to generate the target human body image of the source human body image under the target posture image.

[0074] S2: constructing a self-attention module of all semantic styles in the source human body image;

[0075] Referring to Figure 3 , the step specifically includes:

[0076] Step 0: taking the K semantic style encodings as query values query, key values key and value items value, calculating the similarity between the K semantic style encodings, that is, multiplying the query values query and the key values key matrix, and obtaining a self-attention matrix AM S after processing by a softmax function;

[0077] Step 1: multiplying the self-attention matrix AM S and the value item value matrix to obtain the output self_out of the self-attention mechanism;

[0078] Step 2. Add the output self_out of the self-attention mechanism and the K semantic style encodings to obtain the residual output residual_self of the self-attention mechanism;

[0079] S3: Construct a cross-attention module between all semantic styles in the source human body image and the target pose image, and generate a target semantic segmentation map using the cross-attention matrix in the cross-attention module;

[0080] Referring to Figure 4 , the step is specifically:

[0081] Step 0. Fuse the preliminary features F crs as the query value query, the residual output residual_self of the self-attention mechanism as the key value key and the value item value, calculate the similarity between the preliminary fused features F crs and the residual output residual_self of the self-attention mechanism, that is, the query value query and the key value key matrix are multiplied, and the cross-attention matrix AM is obtained after the softmax function is processed.

[0082] Step 1. Multiply the cross-attention matrix AM and the value item value matrix to obtain the output cross_out of the cross-attention mechanism;

[0083] Step 2. Add the output cross_out of the cross-attention mechanism and the preliminary fused features F crs to obtain the residual output residual_cross of the cross-attention mechanism;

[0084] Step 3. Input the semantic segmentation map of the source human body image under the target pose image as the real label GroudTruth, use the cross-entropy loss to constrain the cross-attention matrix AM, and generate a target semantic segmentation map using the cross-attention matrix AM.

[0085] S4: Embed the self-attention module and the cross-attention module into the generator, train a generative adversarial network, and realize human pose conversion.

[0086] Referring to Figure 5 , the step is specifically:

[0087] Step 0. Input the target pose image into the pose encoder to obtain the target pose feature;

[0088] Step 1. Input the source human body image and the source semantic segmentation map containing K semantics into the style encoder to obtain K semantic style encodings;

[0089] Step2. K semantic style encodings are injected into the target pose feature in the form of lateral branches through the AdaIN module to obtain the preliminary fused feature F crs ;

[0090] Step3. The K semantic style encodings obtained by the style encoder are input into the self-attention module to obtain the residual output of the self-attention module residual_self;

[0091] Step4. The preliminary fused feature F crs obtained by the AdaIN module and the residual output of the self-attention module residual_self are input into the cross-attention module to obtain the residual output of the cross-attention module residual_cross and the target semantic segmentation map.

[0092] Step5. The residual output of the cross-attention module residual_cross is input into the decoder to generate the target human body image of the source human body image under the target pose image.

[0093] The present application embeds the attention module into the human body pose conversion network, inputs the source human body image and the target pose image into the generator to convert the target human body image of the source human body image under the target pose image; constructs the self-attention module of all semantic styles in the source human body image; constructs the cross-attention module between all semantic styles in the source human body image and the target pose image, and generates the target semantic segmentation map by using the cross-attention matrix in the cross-attention module; embeds the self-attention module and the cross-attention module into the generator, trains a generative adversarial network, and realizes the human body pose conversion.

[0094] The above is only the preferred embodiment of the present application, and is not used to limit the present application, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A human pose transformation method based on an attention mechanism, characterized in that, The method comprises the following specific steps: Step 1: input the source human body image and the target posture image into the generator to convert the source human body image into a target human body image under the target posture image; Step 2: construct a self-attention module for all semantic styles in the source human body image; Step 3: construct a cross-attention module between all semantic styles in the source human body image and the target posture image, and generate a target semantic segmentation map by using a cross-attention matrix in the cross-attention module; Step 4: embed the self-attention module and the cross-attention module into the generator, train a generative adversarial network, and realize human body posture conversion; wherein: The generator is composed of a pose encoder, a style encoder, an AdaIN module and a decoder, wherein a target pose image is input into the pose encoder to obtain target pose features, a source human body image and a source semantic segmentation image containing K semantics are input into the style encoder to obtain K semantic style encodings, then the K semantic style encodings are injected into the target pose features in the form of side branches through the AdaIN module to obtain preliminary fused features , and finally the preliminary fused features are input into the decoder to generate a target human body image of the source human body image under the target pose image. The step 2 specifically comprises: Step 2-1: all K semantic style encodings are taken as query values query, key values key and value items value, wherein the dimension sizes of the query values query, the key values key and the value items value are all K*C, the similarity between the K semantic style encodings is calculated, that is, the query values query are multiplied with the key values key matrix, and a self-attention matrix is obtained after being processed by a softmax function , the dimension size of which is K*K; Step 2-2: multiply the self-attention matrix with the value item value matrix to obtain the output self_out of the self-attention mechanism, which has a dimension size of K*C; Step 2-3: add the output self_out of the self-attention mechanism and the K semantic style encodings to obtain a residual output residual_self of the self-attention mechanism, and the dimension size of the residual output residual_self is K*C; The step 3 specifically comprises: Step 3-1: preliminary fused features As the query value query, the dimension size is HW*C, and the residual output residual_self of the self-attention mechanism is the key value key and the value item value, the dimension size is K*C, and the preliminary fused features are calculated The similarity between the residual output residual_self of the self-attention mechanism, that is, the query value query and the key value key matrix are multiplied, and the cross attention matrix is obtained after the softmax function is processed , the dimension size is HW*K; Step 3-2: multiply the cross-attention matrix with the value item value matrix to obtain the output of the cross-attention mechanism cross_out, which has a dimension size of HW*C; Step 3-3: add the output of the cross-attention mechanism cross_out and the preliminarily fused features to obtain the residual output of the cross-attention mechanism residual_cross, which has a dimension size of HW*C; Step 3-4: input the semantic segmentation map of the source human body image under the target posture image, use the semantic segmentation map as a real label GroudTruth, use a cross-entropy loss to constrain the cross-attention matrix AM, and thus generate the target semantic segmentation map by using the cross-attention matrix AM.

2. The human pose transformation method based on an attention mechanism according to claim 1, characterized in that, The embedding manner of the attention module into the generator is that the self-attention module and the cross-attention module are connected after an AdaIN module and before a decoder.

Images