Digital human generation method, digital human generation system and computer readable storage medium

Abstract

The invention provides a digital human generation method, a digital human generation system and a computer readable storage medium, and the method comprises the steps: obtaining user text information, and converting the user text information into a user feature vector; comparing the user feature vector with a digital human part feature vector in each database, and querying each target digital human part feature vector with the highest similarity with the user feature vector from each database; calling a to-be-replaced video, and replacing the target digital person partial feature corresponding to the target digital person partial feature vector with the video feature of the to-be-replaced video to obtain a replaced video; inputting the replacement video into a preset digital human model for training to obtain a current digital human model; and obtaining audio information, and inputting the audio information into the current digital human model to obtain a digital human video. The method can save processing resources of the processor.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a digital human generation method, a digital human generation method and a computer-readable storage medium. Background Art

[0002] Digital humans are lifelike, interactive virtual characters created through computer graphics, artificial intelligence, motion capture, and speech synthesis technologies, and they interact with users. Today, digital humans can be applied to entertainment, education, medical care, and other fields to improve the convenience and entertainment of users' lives. At present, the production process of digital humans is relatively complicated and difficult, requiring the assistance and guidance of professional technicians, and a large amount of resources. For example, traditional digital human models require real-life videos to be shot, and after pre-processing the real-life videos, digital human model training is performed, and then the digital human model is deployed on the GPU server. From real people to generating digital humans, each step requires a lot of time to produce and debug.

[0003] There is an existing method for text-driven digital human to generate video, which is to clean and standardize the input text content, extract key information from the text content, identify entities; perform emotional polarity annotation to determine emotional tendency; generate virtual scenes related to the text content based on key information and entities, and select different background colors, lighting and environment settings according to emotional tendency; create a digital human anchor and place the generated digital human in the virtual scene; generate facial expressions and movements of the digital human according to emotional tendency; generate speech according to the text content; synchronize the movements, facial expressions, speech and lip shape of the digital human; generate video according to the information of virtual scene, digital human, synchronization, facial expressions and movements, and add transition effects when the scene switches and the lens changes to generate the final video. However, the digital human creates the corresponding 3D model and animation through text description, which requires a processor with higher processing performance to generate, consuming a lot of processing resources. Summary of the invention

[0004] A first object of the present invention is to provide a method for generating a digital human being that reduces the processing load of a processor.

[0005] The second objective of the present invention is to provide a digital human generation system that implements the above-mentioned digital human generation method.

[0006] The third object of the present invention is to provide a computer-readable storage medium for implementing the above-mentioned digital human generation method.

[0007] To achieve the first objective of the present invention, the present invention provides a method for generating a digital human, the method comprising: obtaining user text information, converting the user text information into a user feature vector, the user feature vector comprising at least one of the following: a face feature vector, a clothing feature vector, a hairstyle feature vector, a voice feature vector and a style feature vector; comparing the user feature vector with the partial feature vectors of digital humans in each database, querying each target digital human partial feature vector with the highest similarity to the user feature vector from each database; calling a video to be replaced, replacing the video features of the video to be replaced with the partial features of the target digital human corresponding to the partial feature vector of the target digital human, to obtain a replacement video; inputting the replacement video into a preset digital human model for training, to obtain a current digital human model; obtaining audio information, inputting the audio information into the current digital human model, to obtain a digital human video.

[0008] As can be seen from the above scheme, by presetting multiple databases and comparing the user feature vector with the partial feature vector of the digital human in the database, the features required by the user can be queried in the existing database without generating an image separately. While building a personalized digital human for the user, the processing resources of the processor can be saved, and the difficulty of making the digital human can be reduced. The digital human features required by the user are selected, and then the digital human features required by the user are replaced with the video features of the video to be replaced, so as to obtain a training video. The training video is trained on a preset digital human model to obtain the current digital human model required by the user. In this way, the training video is obtained instead of generating the video after building the character, thereby reducing the operation of generating the video and reducing the processing amount of the processor.

[0009] In a further solution, after querying the target digital human partial feature vectors with the highest similarity corresponding to the user feature vectors from each database, it is also performed: judging whether the similarity values ​​of multiple target digital human partial feature vectors and the user feature vector are less than a preset value, and if so, calling the feature generation module according to the user feature vector to generate a replacement digital human partial feature.

[0010] In a further solution, the step of replacing the target digital human partial features corresponding to the target digital human partial feature vector with the video features of the video to be replaced also includes: replacing the video features with replacement digital human partial features corresponding to the target digital human partial features that are smaller than a preset value.

[0011] It can be seen from this that if the similarity value of the partial feature vector of the target digital human queried in the database is too small, it is necessary to generate and replace the partial features of the digital human so that the digital human features meet the needs of the user.

[0012] In a further solution, after obtaining the replacement video, the following steps are performed: sending the replacement video to the user, receiving user confirmation information, and then inputting the replacement video into the preset digital human model for training.

[0013] It can be seen from this that after obtaining the replacement video, the user needs to confirm whether the replacement video meets the user's requirements, and then conduct training to ensure that the digital human is the digital human the user needs.

[0014] In a further solution, the step of obtaining audio information includes: obtaining user audio text information, and generating audio information from the user audio text information.

[0015] It can be seen that the audio information is generated through the text information input by the user, thereby generating a digital human video.

[0016] In a further solution, after obtaining the digital human video, the following steps are further performed: generating background music according to the audio information, and merging the background music with the digital human video.

[0017] It can be seen that by generating background music through audio information, the background music can be integrated with the video content to enhance the user experience.

[0018] In a further solution, the database includes at least one of the following: a face database, a clothing database, a hairstyle database, a sound database, a style database, and a real-life video database.

[0019] It can be seen that the features required by the user are obtained according to the user feature vector, thereby generating the digital human required by the user.

[0020] In order to achieve the second purpose, the digital human generation system provided by the present invention is characterized in that it includes a control module, a feature generation module, a speech synthesis module, a feature replacement module, a video synthesis module, and an audio synchronization module; the control module is connected to the feature generation module, the speech synthesis module, the feature replacement module, the video synthesis module, and the audio synchronization module respectively through a system interface; the control module can execute the above-mentioned digital human generation method when executing a computer program.

[0021] In order to achieve the third purpose, the present invention provides a computer-readable storage medium, on which a computer program is stored, characterized in that: when the computer program is executed by the control module, the above-mentioned digital human generation method is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a system block diagram of an embodiment of a digital human generation system of the present invention.

[0023] Figure 2 It is a flow chart of an embodiment of a method for generating a digital human according to the present invention.

[0024] The present invention is further described below in conjunction with the accompanying drawings and embodiments. DETAILED DESCRIPTION

[0025] The present invention obtains user text information, queries the partial feature vector of the target digital human closest to the user from a database, replaces the video features of the video in the video to be replaced, obtains the replacement video, and uses the replacement video to train the current digital human model, thereby saving the processing resources of the processor while building a personalized digital human for the user.

[0026] Digital human generation system implementation example: See also Figure 1 The digital human generation system of this embodiment includes a control module 1, a feature generation module 21, a speech synthesis module 22, a feature replacement module 23, a video synthesis module 24, an audio synchronization module 25 and a database 26. The control module 1 includes an artificial intelligence entity 11, a system interface 12 and an RPA system 13. The artificial intelligence entity 11 is an AI Agent, and the artificial intelligence entity 11 can call each module through the system interface 12 to complete the digital human generation operation. The speech synthesis module 22 is a TTS speech synthesis module, and the speech synthesis module 22 includes a variety of sound templates. The database 26 includes at least one of a face database, a clothing database, a hairstyle database, a sound database, a style database, and a real-life video database. Among them, the face database contains copyrighted faces of different groups of people of different ages, genders, races, etc., the real-life video database contains copyrighted real-life videos of different body shapes and different actions, the clothing database contains pre-generated clothes of different colors, styles and types, and the hairstyle generation module contains pre-generated hairstyles of different colors, styles and types. The style database contains style features such as cartoon style, anime style, and clay figure style. The feature generation module 21 includes a face generation module, a clothing generation module, a hairstyle generation module, etc. Among them, the feature generation module 21 can generate corresponding features by means of text-to-image and image-to-image. The feature replacement module 23 includes a face replacement module, a clothing replacement module, a hairstyle replacement module, a voice replacement module, a style replacement module, etc.

[0027] The artificial intelligence agent 11 can obtain user text information, call the database 26 to obtain the target digital human partial feature module or call the feature generation module 21 to generate the replacement digital human partial features, and then replace the video features in the video to be replaced with the target digital human partial feature module or the replacement digital human partial features through the feature replacement module 23, thereby forming a replacement video. The artificial intelligence agent 11 inputs the replacement video into the preset digital human model for training to obtain the current digital human model. And deploy the current digital human model into the RPA system 13. The artificial intelligence agent 11 inputs the user audio text information into the speech synthesis module 22 to obtain audio information. The artificial intelligence agent 11 inputs the audio information into the current digital human model to obtain the digital human video.

[0028] Digital human generation method embodiment: See also Figure 2 , the digital human generation method of this embodiment is executed by the digital human generation system of the above embodiment. When the user needs to create his own personalized digital human, he needs to input the requirements for the digital human through the text box, such as face requirements, clothing requirements, hairstyle requirements, voice requirements and style requirements. Among them, the artificial intelligence entity obtains the user's requirements, first executes step S11, the artificial intelligence entity obtains the user's text information, and converts the user's text information into a user feature vector. The artificial intelligence entity can interact with the user, conduct a dialogue, and guide the user to input the user's text information. The artificial intelligence entity inputs the user's text information into the language model to obtain the reply information. In this embodiment, the language model is the LLM language model. The artificial intelligence entity confirms the final user text information by dialogue with the user, and then converts the user text information into a user feature vector. The user feature vector includes at least one of the following: a face feature vector, a clothing feature vector, a hairstyle feature vector, a voice feature vector, and a style feature vector.

[0029] After the artificial intelligence entity converts the user text information into a user feature vector, step S12 is executed to compare the user feature vector with the partial features of the digital human in each database, and query the target digital human feature vector with the highest similarity to the user feature vector. Among them, the database includes at least one of the following: a face database, a clothing database, a hairstyle database, a sound database, a style database, and a real-life video database. Since the user feature vector includes multiple feature vectors, the artificial intelligence entity compares multiple different feature vectors in the user feature vector with the corresponding database, for example, the face feature vector is compared with the partial features of the digital human in the face database. The artificial intelligence entity queries the target digital human feature vector with the highest similarity to the user feature vector through the RAG model.

[0030] After searching for the target digital human feature vector with the highest similarity to the user feature vector, it is also determined whether the similarity values ​​of the partial feature vectors of the target digital humans and the user feature vector are less than a preset value. The similarity value is the cosine distance between the partial feature vector of the target digital human and the user feature vector. The larger the similarity value, the higher the similarity between the partial feature vector of the target digital human and the user feature vector. The preset value is the value for determining whether the partial features of the digital human in the database meet the user's requirements in this embodiment.

[0031] If the similarity value between the partial feature vector of the target digital person and the user feature vector is less than the preset value, it proves that there is no data similar to the user feature vector in the database. At this time, the data of the user feature vector needs to be generated, that is, the feature generation module is called according to the user feature vector to generate the replacement digital person partial feature. The feature generation module includes a face generation module, a clothing generation module, a hairstyle generation module, a sound generation module, etc. The user feature vector corresponding to the partial feature vector of the target digital person that is less than the preset value needs to generate a replacement digital person partial feature, and the replacement digital person partial feature replaces the replacement digital person partial feature that is less than the preset value. For example, the clothes in the clothing database that are closest to the user's needs do not meet the user's needs, that is, the replacement clothes are generated by the clothing generation module.

[0032] After generating the partial features of the replacement digital human, step S13 is also executed to call the video to be replaced, replace the partial features of the target digital human with the video features, and obtain the replacement video. The video to be replaced is obtained from the real-life video database. It can be obtained by comparing the user feature vector information with the data human partial feature vector of the real-life video database. The data human partial feature vector of the real-life video database is a video feature vector. The video features are replaced according to the partial features of the digital human corresponding to the multiple target digital human partial features obtained in step S12 and the target digital human partial features less than the preset value, and the replacement video is obtained. That is, the real person in the video to be replaced is replaced with a face, clothes and hairstyle, etc., and the style of the video to be replaced is also replaced by the style replacement module. The replacement video contains the user's digital human image information. After obtaining the replacement video, the replacement video is sent to the user through a text box, and the user confirmation information is received.

[0033] After receiving the user's confirmation information, step S14 is further performed to input the replacement video into the preset digital human model for training to obtain the current digital human model. The preset digital human model is a generative adversarial network or a variational autoencoder. By training the preset digital human model with the replacement video, the current digital human model including the user's digital human is obtained.

[0034] After the current digital human model is generated, it is deployed to the RPA system, and the deployment results are fed back to the artificial intelligence body. The artificial intelligence body analyzes and understands the feedback information through the language model.

[0035] After obtaining the current digital human model, step S15 is further executed to input the audio information into the current digital human model to obtain a digital human video. The audio information is synthesized by a speech synthesis module. When the user inputs the user audio text information, the audio information is generated for the user audio text information. The artificial intelligence entity can call the audio synchronization module to input the audio information into the current digital human model, extract the voice features, calculate each frame of the image through the audio information, and synthesize each frame of the image and the voice features into a digital human video through the video synthesis module. Background music is also generated according to the audio information, that is, background music with a high similarity to the audio information is obtained from the audio database according to the audio information, and the background music is merged with the digital human video to obtain a digital human video that meets the user's needs.

[0036] In this embodiment, by presetting multiple databases and comparing the user feature vector with some feature vectors of the digital human in the database, the features required by the user can be queried in the existing database, reducing the generation of features, and saving the processing resources of the processor while building a personalized digital human for the user.

[0037] Computer readable storage medium embodiment: The method for generating a digital human in a computer device described in the above embodiment can be stored in a computer-readable storage medium in the form of a computer program. When the computer program is executed by the control module, the steps of the embodiment of the method for generating a digital human in the above digital human generating device can be completed. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0038] The above are only preferred embodiments of the present invention, but the design concept of the invention is not limited thereto. Without departing from the concept of the present invention, more other equivalent embodiments may be included. It is possible for those skilled in the art to make various obvious changes, readjustments and substitutions without departing from the protection scope of the present invention.

Claims

1. A method for generating a digital human, characterized in that: The method includes: Acquire user text information, and convert the user text information into a user feature vector, wherein the user feature vector includes at least one of the following: a face feature vector, a clothing feature vector, a hairstyle feature vector, a voice feature vector, and a style feature vector; Compare the user feature vector with the partial feature vectors of digital humans in each database, and query the partial feature vectors of target digital humans with the highest similarity to the user feature vector from each database; Calling the video to be replaced, replacing the video features of the video to be replaced with the partial features of the target digital human corresponding to the partial feature vector of the target digital human, to obtain a replacement video; Inputting the replacement video into a preset digital human model for training to obtain a current digital human model; Acquire audio information, input the audio information into the current digital human model, and obtain a digital human video.

2. The method for generating a digital human according to claim 1, characterized in that: After searching each database for the target digital human partial feature vectors with the highest similarity corresponding to the user feature vector, the following is further performed: It is determined whether the similarity values ​​between the partial feature vectors of the target digital humans and the user feature vector are less than a preset value. If so, a feature generation module is called according to the user feature vector to generate partial features of the replacement digital human.

3. The method for generating a digital human according to claim 2, characterized in that: The step of replacing the video features of the video to be replaced with the partial features of the target digital human corresponding to the partial feature vector of the target digital human also includes: The video feature is replaced by the replacement digital human partial feature corresponding to the target digital human partial feature which is smaller than a preset value.

4. The method for generating a digital human according to any one of claims 1 to 3, characterized in that: After getting the replacement video, also execute: The replacement video is sent to the user, and after receiving the user's confirmation information, the replacement video is input into the preset digital human model for training.

5. The method for generating a digital human according to any one of claims 1 to 3, characterized in that: The step of obtaining the audio information includes: Acquire user audio text information, and generate the audio information from the user audio text information.

6. The method for generating a digital human according to claim 5, characterized in that: After obtaining the digital human video, the following steps are also performed: Background music is generated according to the audio information, and the background music is merged with the digital human video.

7. The method for generating a digital human according to claim 6, characterized in that: The database includes at least one of the following: a face database, a clothing database, a hairstyle database, a sound database, a style database, and a real-life video database.

8. A digital human generation system, characterized in that: It includes a control module, a feature generation module, a speech synthesis module, a feature replacement module, a video synthesis module, and an audio synchronization module; the control module is connected to the feature generation module, the speech synthesis module, the feature replacement module, the video synthesis module, and the audio synchronization module respectively through a system interface; The control module can execute the digital human generation method as claimed in any one of claims 1 to 7 when executing the computer program.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the control module, the digital human generation method as described in any one of claims 1 to 7 is implemented.

Images