A method and system for generating voice-driven hand motion images based on deep learning and hand transformation, and a computer-readable storage medium

Abstract

The present application provides a method and system for generating hand motion images driven by voice based on deep learning and hand transformation, and a computer-readable storage medium. The technical solution includes the following steps: obtaining audio and corresponding character images containing hands; inputting the audio into a transformation network to obtain corresponding hand transformation parameters; inputting the character images containing hands into a segmentation network to obtain corresponding hand layered images and background images; bringing the hand transformation parameters into the hand layered images to obtain corresponding hand transformation images; inputting the hand transformation images and the background images into a generation network to obtain the final image. In view of the problem that generating hand movements that match voice often results in poor effects such as hand truncation and dislocation, the present invention uses hand transformation to depict hand movements, and effectively and accurately generates images with hand movements through a deep network.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method for generating voice-driven hand motion images based on deep learning and hand transformation. Background Art

[0002] Image generation is a very important field in computer vision, and its related technologies are used in various scenarios, including virtual avatars, video production, etc. With the development of deep learning, image generation technology has entered a new period of development. A large number of deep models including adversarial neural networks, autoencoders, neural rendering fields, and diffusion models have been applied to the generation field.

[0003] With the rise of the concept of the metaverse, technologies and issues related to virtual humans have received widespread attention. Virtual humans can be simply divided into two categories based on the image dimension: 2D virtual humans and 3D virtual humans. Regardless of the type of virtual human, the following two issues are involved: voice drive of the virtual face and virtual human motion generation. Among them, due to the complexity of the hands, virtual human motion generation has always been a very difficult problem. For 3D virtual humans, it is common to use pre-configured data such as action libraries; for 2D virtual humans, deep learning algorithms are usually used to generate hand movements that match the voice, but there are often problems with poor results such as hand truncation and dislocation. Summary of the invention

[0004] In response to the above problems, the present invention provides a method and system for generating voice-driven hand motion images based on deep learning and hand transformation, and a computer-readable storage medium, which uses hand transformation to characterize hand movements and uses a deep network to effectively and accurately generate images with hand movements.

[0005] In a first aspect, the present invention provides a method for generating a speech-driven hand motion image based on deep learning and hand transformation, comprising:

[0006] S1: Get audio and corresponding character image including hands;

[0007] Preferably, the audio to drive the hand movement and an RGB image are input, and the image is a person image including arms and palms, which can be a half-body image or a full-body image;

[0008] S2: Input the audio into a transformation network to obtain corresponding hand transformation parameters;

[0009] Preferably, the driving audio is input into a trained neural network, and the network outputs the transformation parameters of the hand according to the input driving audio. The hand is divided into the upper arm, the lower arm and the palm, so the hand transformation parameters output by the neural network will include the transformation parameters of these three parts. These transformation parameters involve translation, scaling, rotation, mirroring and different combinations of the above, and conversion is performed according to the above transformation parameters to obtain the corresponding transformation matrix of the hand.

[0010] S3: inputting the person image including the hand into a segmentation network to obtain corresponding hand layered images and background images;

[0011] Preferably, after the RGB image is input into the segmentation network, segmentation mask images of different parts of the hand are obtained, wherein the different parts of the hand include the upper arm, the lower arm and the palm. According to the segmentation mask images of different parts of the hand and the input image, the corresponding hand image and background image can be obtained through calculation.

[0012] S4: bringing the hand transformation parameters into the hand layered image to obtain a corresponding hand transformation image;

[0013] Preferably, the transformation parameters of different hand positions obtained by the transformation network can be used to calculate the corresponding transformation matrix. At the same time, the segmentation network can obtain images of different hand positions, and the above transformation matrix is ​​applied to the hand image to obtain the transformed hand image.

[0014] S5: Input the hand transformation image and the background image into a generation network to obtain a final image.

[0015] Preferably, the transformed images of multiple hand positions and the background image are spliced ​​together in a certain order according to channels; and then input into a trained generation network to finally generate an image with corresponding hand movements.

[0016] In a second aspect, the present invention provides a speech-driven hand action image preprocessing system based on deep learning and hand transformation, comprising:

[0017] An information collection module, used to collect audio and corresponding character images including hands;

[0018] A parameter generation module, used for inputting the audio into a transformation network to obtain corresponding hand transformation parameters;

[0019] An image generation module, used for inputting the character image into a segmentation network to obtain corresponding hand layer images and background images;

[0020] An information reorganization module, used for bringing the hand transformation parameters into the hand layered image to obtain a corresponding hand transformation image;

[0021] The image generation module is used to input the hand transformation image and the background image into the generation network to obtain the final image.

[0022] In a third aspect, the present invention provides a computer-readable storage medium, which executes the readable storage function to implement the steps of the above-mentioned voice-driven hand motion image generation method based on deep learning and hand transformation.

[0023] Beneficial effects:

[0024] The present invention adopts the method of bringing hand transformation parameters into the hand layered image to obtain the corresponding hand transformation image, and inputting the hand transformation image and the background image into the generation network to obtain the final image. This method can accurately generate images with hand movements, and solve the common problems in existing methods such as hand truncation, dislocation and other poor effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A flowchart of a method for generating speech-driven hand motion images based on deep learning and hand transformation;

[0026] Figure 2 A schematic diagram of hand transformation for a method for generating a speech-driven hand action image based on deep learning and hand transformation;

[0027] Figure 3 Schematic diagram of a model of a voice-driven hand motion image generation method based on deep learning and hand transformation. DETAILED DESCRIPTION

[0028] The exemplary embodiments will be described in more detail below in conjunction with the accompanying drawings. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0029] In the embodiment of the present invention, combined with Figure 1 , Figure 2 and Figure 3 , a method for generating a speech-driven hand action image based on deep learning and hand transformation is proposed, and the method specifically comprises the following steps:

[0030] S1: Get audio and corresponding character image including hands;

[0031] In one embodiment, audio to drive hand movements and an RGB image are input, where the image is an image of a person including hands, typically a half-body image or a full-body image.

[0032] S2: Input the audio into the transformation network to obtain the corresponding hand transformation parameters;

[0033] The input audio will be input into the transformation network. According to the different input audio, the transformation network will output a series of hand transformation parameters corresponding to the audio. Since the hand contains multiple parts, the hand transformation parameters output by the network will also contain transformation parameters of multiple parts.

[0034] The hand is divided into three parts: upper arm, lower arm and palm, so the hand transformation parameters output by the network will cover these three parts respectively. The hand transformation parameters also include translation, scaling, rotation, and mirroring parameters. If the values ​​of these parameters are all non-zero, it means that it is a combination of the above transformations. If only the translation parameter value is non-zero, it is equivalent to only a translation transformation.

[0035] According to these parameters, the transformation matrix M to be used can be obtained by conversion. Usually, the transformation matrix M to be used is a 2x3 matrix.

[0036] S3: Input the image into the segmentation network to obtain the corresponding hand layer image and background image;

[0037] By inputting the input RGB image into the segmentation network, we can get the segmentation mask of the hand. The hand is divided into three parts: upper arm, lower arm and palm. Depending on the different positions of the hand, the segmentation network will output different segmentation masks.

[0038] The segmentation network will output segmentation masks for the upper arm, lower arm, and palm respectively. Common segmentation masks include images with values ​​between 0 and 1, or grayscale images with values ​​between 0 and 255. By calculating the value of each pixel in the segmentation mask and the value of each pixel in the input image, we can obtain the hand layer images at different positions and the background image.

[0039] S4: bringing the hand transformation parameters into the hand layered image to obtain a corresponding hand transformation image;

[0040] refer to Figure 2 , the hand transformation matrix is ​​brought into different hand layer images according to the position, and the corresponding hand transformation image can be obtained through matrix operation.

[0041] S5: Input the hand transformation image and the background image into a generation network to obtain a final image.

[0042] The obtained hand transformation image and background image are input into the generative network to obtain the final image, wherein the hand transformation image and the background image will be spliced ​​in the channel dimension in a certain order, and then input into the generative network to obtain the final image.

[0043] In an embodiment of the present invention, a voice-driven hand motion image preprocessing system based on deep learning and hand transformation is provided corresponding to the steps of the above methods, including:

[0044] An information collection module, used to collect audio and corresponding character images including hands;

[0045] A parameter generation module, used for inputting the audio into a transformation network to obtain corresponding hand transformation parameters;

[0046] An image generation module, used for inputting the character image into a segmentation network to obtain corresponding hand layer images and background images;

[0047] An information reorganization module, used for bringing the hand transformation parameters into the hand layered image to obtain a corresponding hand transformation image;

[0048] The image generation module is used to input the hand transformation image and the background image into the generation network to obtain the final image.

[0049] In an embodiment of the present invention, a computer-readable storage medium is provided corresponding to the steps of each of the above methods, and the computer-readable storage medium executes the readable storage function to implement the above-mentioned method of voice-driven hand movement image generation based on deep learning and hand transformation, the method comprising: obtaining audio and corresponding character images containing hands; inputting the audio into a transformation network to obtain corresponding hand transformation parameters; inputting the character images containing hands into a segmentation network to obtain corresponding hand layered images and background images; bringing the hand transformation parameters into the hand layered images to obtain corresponding hand transformation images; inputting the hand transformation images and the background images into a generation network to obtain the final image.

[0050] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person having ordinary knowledge in the art can still modify the technical solutions recorded in the aforementioned embodiments, or replace part or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

Claims

1. A method for generating speech-driven hand motion images based on deep learning and hand transformation, characterized in that: include: S1: Get audio and corresponding character image including hands; The hand includes different parts of the hand, and the different parts of the hand include the upper arm, the lower arm and the palm; S2: Input the audio into a transformation network to obtain corresponding hand transformation parameters; The hand transformation parameters include transformation parameters of different parts of the hand, and the transformation network includes rotation, translation, scaling, mirroring, and different corresponding combinations thereof; S3: Input the human image containing the hand into the segmentation network to obtain the corresponding hand layered image and background image: The segmentation network segments the input person image to obtain segmentation masks of different parts of the hand, and obtains hand layer images and background images of different parts of the hand based on the segmentation masks and the person image; S4: Bring the hand transformation parameters into the hand layered image to obtain a corresponding hand transformation image: The transformation parameters of different parts of the hand are converted to obtain corresponding transformation matrices of different parts of the hand; Applying the transformation matrices of different parts of the hand to the corresponding hand layered images to obtain corresponding hand transformation images; S5: Input the hand transformation image and the background image into a generation network to obtain a final image.

2. The method for generating a speech-driven hand motion image based on deep learning and hand transformation according to claim 1, characterized in that: The hand transformation image and the background image are spliced ​​by channel in a certain order, and then input into the generation network to obtain the final image.

3. A speech-driven hand motion image preprocessing system based on deep learning and hand transformation as claimed in any one of claims 1-2, characterized in that: include: An information collection module, used to collect audio and corresponding character images including hands; A parameter generation module, used for inputting the audio into a transformation network to obtain corresponding hand transformation parameters; An image generation module, used for inputting the character image into a segmentation network to obtain corresponding hand layer images and background images; An information reorganization module, used for bringing the hand transformation parameters into the hand layered image to obtain a corresponding hand transformation image; The image generation module is used to input the hand transformation image and the background image into the generation network to obtain the final image.

4. A computer-readable storage medium, characterized in that: The computer-readable storage medium executes the readable storage function to implement the steps of the method for generating voice-driven hand motion images based on deep learning and hand transformation as described in any one of claims 1-2.

Images