Motion migration model design method and device, storage medium and program product

Abstract

The invention provides an action migration model design method and device, a storage medium and a program product, and relates to the technical field of computer vision. The motion migration model design method comprises the following steps: training a mask extraction model; two human body images are obtained, one image is an original image I, and the other image is a driving image T; extracting an original limb mask and an original limb image from the original image; extracting a driving limb mask from the driving image; according to the driving limb mask image, transforming the original limb mask image and the original limb image to obtain a target limb mask image and a target limb image; generating a target human body image by using an action generation module; generating a target background image by using a background generation module; a final image is generated using a synthesis module. According to the method, the overall accuracy and the image quality of motion migration image generation are improved, and related generation flaws and errors are avoided.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method, device, storage medium and program product for designing an action migration model. Background Art

[0002] Motion transfer technology generates a new image through the original image and the driving image, so that the characters in the original image perform the same actions as the characters in the driving image. This technology is often used in the fields of virtual reality and augmented reality to enhance the expressiveness of the image.

[0003] Traditional motion transfer technology operates based on image key points. For example, Chinese patent CN113870314A published on December 31, 2021 and Chinese patent CN114049652A published on February 15, 2022 both detect the key points of the original image and the driving image, and then transform the image based on the comparison relationship between the key points of the two, and finally obtain the motion transfer image.

[0004] In the above-mentioned prior art, the key points of the human body usually need to be obtained through a large amount of data training, and their stability and accuracy are not ideal. Therefore, the action transfer technology based on key points usually has errors and cannot cover the entire human body, resulting in low accuracy and quality of the generated image, and prone to generation defects and errors. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention proposes a motion migration method, device, storage medium and program product, which extract the mask images of each limb of the human body, blur them, and then obtain the corresponding limb images through calculation. On this basis, the motion migration task is completed, which improves the overall accuracy and image quality of the generated image and avoids the occurrence of related generation defects and errors.

[0006] In a first aspect, the present invention provides a method for designing an action migration model, comprising the following steps:

[0007] Train a mask extraction model;

[0008] Obtain two human body images, one is the original image I, and the other is the driven image T;

[0009] Extracting an original limb mask and an original limb image from the original image;

[0010] extracting a driving limb mask from the driving image;

[0011] According to the driving limb mask, the original limb mask and the original limb image are transformed to obtain a target limb mask and a target limb image;

[0012] Generate a target human image using an action generation module;

[0013] Use the background generation module to generate the target background image;

[0014] Use the compositing module to generate the final image.

[0015] As a further improvement of the present invention, the step of acquiring two human body images, one of which is an original image I and the other is a driving image T, includes:

[0016] The obtained original image I is a person image, and contains only one person;

[0017] The obtained driving image T is a person image, and contains only one person.

[0018] As a further improvement of the present invention, the step of extracting the original limb mask and the original limb image from the original image includes:

[0019] The mask extraction model is used to extract the body mask of the person from the original image I, and then fuzzy process is performed to obtain the original body mask M I1 ,M I2 …M In ;

[0020] The original limb mask image is used to perform calculations with the original image to obtain the original limb image.

[0021] As a further improvement of the present invention, the step of extracting a driving limb mask from a driving image comprises:

[0022] The mask extraction model is used to extract the body mask of the character from the driving image, and then fuzzy process is performed to obtain the driving body mask M. T1 ,M T2 …M Tn , where, for i = 1, 2, ..., n, M Ii With M Ti Corresponding to the same limb parts;

[0023] As a further improvement of the present invention, the fuzzy processing includes:

[0024] Blur the edge area of ​​the mask image using Gaussian blur;

[0025] The size of the Gaussian kernel is obtained by customization, and the value of the Gaussian kernel is obtained by Gaussian distribution.

[0026] As a further improvement of the present invention, the original limb mask and the original limb image are transformed according to the driving limb mask to obtain the target limb mask and the target limb image, including:

[0027] Select image benchmark;

[0028] According to the image benchmark and the driving limb mask M t1 The displacement and rotation relationship of M t1 The corresponding action driving parameter is Q(0,1);

[0029] According to the driving limb mask M Ti With M Tj The displacement and rotation relationship of the action driving parameters Q(i,j) are determined, where i = 1, 2, ..., n; j = 2, 3, ..., n. For each i and j in Q(i,j), M Ti With M Tj For connection relationship;

[0030] According to the image benchmark and the action driving parameter Q(0,1), the original limb mask M I1 Transform and obtain the target limb mask M F1 ;

[0031] According to the target limb mask M Fi And the corresponding action driving parameters Q(i,j), the original limb mask M Ij Transform and obtain the target limb mask M Fj , where i = 1, 2, ..., n; j = 2, 3, ..., n;

[0032] The original limb image and the corresponding original limb mask image are transformed in the same way to obtain the target limb image.

[0033] As a further improvement of the present invention, the step of using the action generation module to generate the target human body image includes:

[0034] Preliminarily splicing the target limb mask and the target limb image;

[0035] The preliminary stitching results are input into the deep neural network model for deep stitching to obtain the target human body image.

[0036] As a further improvement of the present invention, the step of using the background generation module to generate the target background image includes:

[0037] The original image and the mask images of each original limb are used for calculation to obtain the original background image with the original limb image removed;

[0038] The original background image is input into the deep neural network model for restoration to obtain the target background image.

[0039] As a further improvement of the present invention, the step of using the synthesis module to generate the final image includes:

[0040] Preliminarily splicing the target human image and the target background image;

[0041] The preliminary stitching results are input into the deep neural network model for synthesis to obtain the final image.

[0042] In a second aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the first aspect.

[0043] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0044] In a fourth aspect, the present invention provides a computer program product, which implements the steps of the method described in the first aspect when the computer program is executed by a processor.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] The present invention proposes a motion migration method, device, storage medium and program product using Gaussian blur and deep learning, which extracts the mask images of each limb of the human body, blurs them, and then obtains the corresponding limb images through calculation. On this basis, the motion migration task is completed, the overall accuracy and image quality of the generated image are improved, and the occurrence of related generation defects and errors is avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A flowchart of a method for designing an action migration model disclosed in the present invention

[0048] Figure 2 Example of a limb mask DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the present invention will combine the accompanying drawings to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments, wherein steps S1, S2... in the embodiments described in the present invention do not limit the only execution steps of the present invention; the various models, simulation environments, and software described in the present invention are not the only limiting methods of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0050] In the present invention, computer devices / equipment / systems refer to related entities applied to computers, such as hardware, a combination of hardware and software, software or software in execution, etc. In detail, for example, software includes but is not limited to a process running on a processor, a processor, an object, executable software, an execution thread, a program and / or a computer. In addition, an application or script program running on a server, a server can also be software. One or more software can be in an execution process and / or thread, and the software can be localized on one computer and / or distributed between two or more computers, and can be run by various computer-readable media.

[0051] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application may be combined with each other.

[0052] In a first aspect, the present invention provides an embodiment of a method for designing an action migration model. Figure 1 As shown, the specific process can be as follows:

[0053] S1: Training the mask extraction model.

[0054] Preferably, the training step comprises:

[0055] S11: Use a person image as a pre-training image, and annotate the body parts of the pre-training image as a training target.

[0056] S12: Set the loss function.

[0057] Preferably, the cross entropy loss function is used as the loss function.

[0058] S13: Use a deep neural network model to extract human limbs from training images, compare the extracted human limb images with the annotated limbs, feed the comparison results back to the model for deep learning, and use the loss function as the constraint condition of the model.

[0059] S14: When the model converges, a mask extraction model is obtained.

[0060] S2: Obtain two human body images, one is the original image I, and the other is the driving image T.

[0061] Preferably, both images contain complete human images, that is, there are people in both images, and the number of people in both images is one person.

[0062] S3: Extract the original limb mask and the original limb image from the original image.

[0063] Preferably, the extracting step comprises:

[0064] S31: Extracting a body mask from the original image through a mask extraction model.

[0065] The extracted limb parts are determined by the mask extraction model;

[0066] Preferably, limb annotation can also be performed according to the needs of action migration.

[0067] In one embodiment of the present invention, the limbs include: head, torso, upper arms, lower arms, hands, thighs, calves, feet, etc.

[0068] In another embodiment of the present invention, the limbs include: head, left upper arm, right upper arm, left lower arm, right lower arm, left hand, right hand, torso, left thigh, right thigh, left calf, right calf, left foot, right foot, etc.

[0069] Figure 2 An example of masking a person's body parts.

[0070] S32: performing blur processing on the person's limb mask image to obtain an original limb mask image.

[0071] Preferably, Gaussian blur is used to blur the person's limb mask image, that is, convolution processing is performed on the image through a Gaussian kernel.

[0072] In one embodiment of the present invention, the size of the Gaussian kernel is obtained by customization, and the value of the Gaussian kernel is obtained by Gaussian distribution.

[0073] S33: Calculate using the original limb mask image and the original image to obtain each original limb image.

[0074] In one embodiment of the present invention, the original limb mask is M I1 ,M I2 …M In , by calculating M Ii ⊙I (where i = 1, 2, ..., n, ⊙ represents the element-by-element result between two matrices, i.e., the Hadamard product), the corresponding original limb image can be obtained.

[0075] The resulting original limb image is left for use in the image transformation stage.

[0076] S4: Using the mask extraction model, extract the driving limb mask from the driving image.

[0077] Preferably, the same method as S31 is used to extract the driving limb mask;

[0078] Each limb part of the driving limb mask image corresponds to each limb part of the original limb mask image.

[0079] S5: Image transformation: transforming the original limb mask and the original limb image according to the driving limb mask to obtain a target limb mask and a target limb image.

[0080] Preferably, the image transformation step comprises:

[0081] S51: Select an image benchmark; and select a driving limb mask M according to the image benchmark. t1 The displacement and rotation relationship of the action is determined to determine the action driving parameter Q(0,1); according to the driving limb mask M Ti With M Tj The displacement and rotation relationship of the action drive parameters Q(i,j) (i=1,2,…,n; j=2,3,…,n) is determined. For each parameter i and j, M Ti With M Tj For mutual connection.

[0082] In one embodiment of the present invention, the image reference is a vertical line of the image, M t1 To drive the torso mask in the image, the driving parameters Q(0,1) of the torso are obtained by calculating the displacement and angle relationship of the torso mask relative to the vertical line of the image; M t2 is the upper arm mask in the driving image (M t2 It can be other body parts, but it must be consistent with M t1 The driving parameters Q(1,2) are obtained by calculating the displacement and angle relationship between the upper arm mask and the torso mask. The above process is repeated until all the limb masks of the character in the driving image are traversed.

[0083] S52: According to the image reference and the action driving parameter Q(0,1), the original limb mask M I1 Transform and obtain the target limb mask M F1 ; According to the target limb mask M Fi And the corresponding action driving parameters Q(i,j), the original limb mask M Ij Transform and obtain the target limb mask M Fj , where i = 1, 2,…, n; j = 2, 3,…, n.

[0084] Preferably, the image reference is the image vertical line, M I1 is the original torso mask. According to the driving parameter Q(0,1), the displacement and angle relationship of the original torso mask relative to the vertical line of the image are adjusted to obtain the target torso mask M. F1 ;M I2 is the original upper arm mask, and M is adjusted by the driving parameter Q(1,2). I2 With M F1 The position and angle relationship between them is used to obtain the target upper arm mask M F2; According to the remaining parameters in the driving parameter set, execute the above process to traverse all the body parts of the person in the driving image.

[0085] S53: Perform the same transformation on the original limb image and the corresponding original limb mask image to obtain a target limb image.

[0086] S6: Generate target human images using the action generation module.

[0087] Preferably, the processing steps of the action production module include:

[0088] S61: Preliminarily stitching the target limb mask image and the target limb image.

[0089] S62: Input the preliminary stitching results into a deep neural network model to perform deep stitching to obtain a target human body image.

[0090] In one embodiment of the present invention, a channel stitching method is used for preliminary stitching: the target limb mask and the target limb image are both RGB images with a width and height of 128. Both are converted into a 128x128x3 matrix, and after channel stitching, a 128x128x6 matrix is ​​obtained.

[0091] The above 128x128x6 matrix is ​​input into the deep neural network model for deep stitching to obtain the target human body image.

[0092] S7: Generate a target background image using a background generation module.

[0093] Preferably, the processing steps of the background generation module include:

[0094] S71: Calculate using the original image and each original limb mask image to obtain an original background image with the original limb image removed.

[0095] In one embodiment of the present invention, the method for removing the original limb image is as follows: suppose the original mask image is converted into a matrix M, and the original image is converted into a matrix I, then by calculating (1-M)⊙I (where (1-M) represents the subtraction of the M matrix from the full-1 matrix), a background mask image for removing the original limb image is obtained; then calculations are performed based on this background mask image and the original image to obtain the original background image.

[0096] S72: Input the original background image into the deep neural network model for restoration to obtain a target background image.

[0097] S8: Generate the final image using the synthesis module.

[0098] Preferably, the specific steps include:

[0099] S81: Preliminarily splicing the target human image and the target background image.

[0100] S82: Input the preliminary stitching results into the deep neural network model for synthesis to obtain the final image.

[0101] In one embodiment of the present invention, a channel stitching method is used for preliminary stitching: the target limb mask and the target limb image are both RGB images with a width and height of 128. Both are converted into a 128x128x3 matrix, and after channel stitching, a 128x128x6 matrix is ​​obtained.

[0102] The above 128x128x6 matrix is ​​input into the deep neural network model for synthesis to obtain the final image.

[0103] In a second aspect, the present invention provides an embodiment of a computer device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the first aspect.

[0104] In a third aspect, the present invention provides an embodiment of a computer-readable storage medium having a computer program stored thereon, which implements the steps of the method described in the first aspect when executed by a processor.

[0105] In a fourth aspect, the present invention provides an embodiment of a computer program product, which implements the steps of the method described in the first aspect when the computer program is executed by a processor.

[0106] The present invention proposes a motion migration method, device, storage medium and program product using Gaussian blur and deep learning, which extracts the mask images of each limb of the human body, blurs them, and then obtains the corresponding limb images through calculation. On this basis, the motion migration task is completed, the overall accuracy and image quality of the generated image are improved, and the occurrence of related generation defects and errors is avoided.

Claims

1. A method for designing an action migration model, characterized in that: The following steps are involved: Train a mask extraction model; Obtain two human body images, one is the original image I, and the other is the driven image T; Extracting an original limb mask and an original limb image from the original image; extracting a driving limb mask from the driving image; According to the driving limb mask, the original limb mask and the original limb image are transformed to obtain a target limb mask and a target limb image; Generate a target human image using an action generation module; Use the background generation module to generate the target background image; Use the compositing module to generate the final image.

2. The method according to claim 1, characterized in that The method of obtaining two human body images, one of which is an original image I and the other is a driving image T, includes: The obtained original image I is a person image, and contains only one person; The obtained driving image T is a person image, and contains only one person.

3. The method according to claim 1, characterized in that The step of extracting the original limb mask image and the original limb image from the original image comprises: The mask extraction model is used to extract the body mask of the person from the original image I, and then fuzzy process is performed to obtain the original body mask M I1 ,M I2 …M In ; The original limb mask image is used to perform calculations with the original image to obtain the original limb image.

4. The method according to claim 1, characterized in that The step of extracting a driving limb mask from a driving image comprises: The mask extraction model is used to extract the body mask of the character from the driving image, and then fuzzy process is performed to obtain the driving body mask M. T1 ,M T2 …M Tn , where, for i = 1, 2, ..., n, M Ii With M Ti Corresponding to the same limb parts.

5. The method according to claim 3 or 4, characterized in that: The fuzzy processing includes: Blur the edge area of ​​the mask image using Gaussian blur; The size of the Gaussian kernel is obtained by customization, and the value of the Gaussian kernel is obtained by Gaussian distribution.

6. The method according to claim 1, characterized in that The method of transforming the original limb mask and the original limb image according to the driving limb mask to obtain the target limb mask and the target limb image includes: Select image benchmark; According to the image benchmark and the driving limb mask M t1 The displacement and rotation relationship of M t1 The corresponding action driving parameter is Q(0,1); According to the driving limb mask M Ti With M Tj The displacement and rotation relationship of the action driving parameters Q(i,j) are determined, where i = 1, 2, ..., n; j = 2, 3, ..., n. For each i and j in Q(i,j), M Ti With M Tj For connection relationship; According to the image benchmark and the action driving parameter Q(0,1), the original limb mask M I1 Transform and obtain the target limb mask M F1 ; According to the target limb mask M Fi And the corresponding action driving parameters Q(i,j), the original limb mask M Ij Transform and obtain the target limb mask M Fj , where i = 1, 2, ..., n; j = 2, 3, ..., n; The original limb image and the corresponding original limb mask image are transformed in the same way to obtain the target limb image.

7. The method according to claim 1, characterized in that The step of using the action generation module to generate a target human body image includes: Preliminarily splicing the target limb mask and the target limb image; The preliminary stitching results are input into the deep neural network model for deep stitching to obtain the target human body image.

8. The method according to claim 1, characterized in that The method of using the background generation module to generate the target background image includes: The original image and the mask images of each original limb are used for calculation to obtain the original background image with the original limb image removed; The original background image is input into the deep neural network model for restoration to obtain the target background image.

9. The method according to claim 1, characterized in that: The method of using the synthesis module to generate the final image includes: Preliminarily splicing the target human image and the target background image; The preliminary stitching results are input into the deep neural network model for synthesis to obtain the final image.

10. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 9.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

12. A computer program product, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images