Digital human lip and limb real-time collaboration method, device and equipment and storage medium

Abstract

The invention discloses a digital human lip and limb real-time collaboration method, device and equipment and a storage medium, and the method comprises the steps: firstly obtaining a digital human picture, an answer text and a plurality of different limb action videos; then segmenting the answer text into a plurality of sub-texts, and generating a voice segment corresponding to each sub-text; cutting the neck of the digital human picture to obtain a first picture containing the head; obtaining a lip synchronization video according to the voice segment and the first picture; then extracting a limb action video corresponding to the keyword to obtain a target limb action video; splicing the target limb action video and the lip synchronization video at the cutting position to obtain a lip and limb collaborative video corresponding to each sub-text; and finally, combining the collaborative videos in sequence to obtain a continuous lip and limb real-time collaborative video. According to the invention, the limb movement of the digital human can be synchronized with the question and answer content.

Description

Technical Field

[0001] The present invention relates to the field of real-time collaboration, and in particular to a method, device, equipment and storage medium for real-time collaboration between digital human lips and body parts. Background Art

[0002] With the development of science and technology, 2D digital human technology covers knowledge and technology in multiple fields such as computer graphics, artificial intelligence, computer vision, virtual reality and augmented reality, as well as user interface design and interaction design. The integration and application of these technologies make 2D digital humans more realistic and interactive in the field of cultural and museums, providing users with a richer and more immersive experience.

[0003] At present, in the field of culture and museums, similar 2D digital humans only have synchronized lip movements. The body movements of the digital humans, such as hands, shoulders, etc., are not synchronized with the content of the questions and answers. They are unable to make corresponding body movements based on the specific content of the questions and answers, lack flexibility and interactivity, and the experience is monotonous.

[0004] Therefore, how to synchronize the body movements of 2D digital humans with the content of questions and answers is an urgent problem that needs to be solved. Summary of the invention

[0005] The present invention provides a method, device, equipment and storage medium for real-time coordination of digital human's lips and body, which can synchronize the body movements of digital human with the content of questions and answers.

[0006] An embodiment of the present invention provides a method for real-time collaboration between a digital mouth, lips and body, comprising:

[0007] Obtaining a digital human image, answer text, and several videos of different body movements; different body movement videos correspond to different keywords;

[0008] The answer text is divided into several sub-texts, and a speech segment corresponding to each sub-text is generated;

[0009] Cut the digital human image at the neck to obtain a first image including the head;

[0010] Obtaining a lip-sync video according to the voice clip and the first image;

[0011] Extract the body movement videos corresponding to the above keywords to obtain the target body movement videos;

[0012] The above-mentioned target body movement video and lip synchronization video are spliced ​​at the cutting position to obtain the lip and body coordination video corresponding to each sub-text;

[0013] The above collaborative videos are merged in sequence to obtain a continuous real-time collaborative video of lips and body parts.

[0014] Furthermore, the above answer text is divided into several sub-texts, including:

[0015] Identify all punctuation marks in the above answer text;

[0016] According to the above punctuation marks, the above answer text is divided into several sub-texts.

[0017] Furthermore, the above answer text is divided into several sub-texts according to the above punctuation marks, including:

[0018] Convert all the above punctuation marks into regular expressions;

[0019] Use the above regular expression to segment the above answer text and remove empty strings to obtain the above sub-texts.

[0020] Furthermore, the digital human image is cut at the neck to obtain a first image including the head, including:

[0021] Inputting the digital human image into a preset neck recognition model so that the neck recognition model recognizes the neck area of ​​the human object in the digital human image;

[0022] According to the identified neck area, the digital human image is cut to obtain a first image including the head;

[0023] The construction of the neck recognition model includes:

[0024] Acquire a number of digital human image samples, and mark the neck area of ​​each of the number of digital human image samples to obtain a number of training samples;

[0025] The neural network model is iteratively trained with each training sample as input and the neck prediction area corresponding to each training sample as output until the neural network model converges, and the converged neural network model is used as the above-mentioned neck recognition model; wherein, in each iterative training process, the neck prediction area is compared with the actual neck area corresponding to the training sample, the loss function value is calculated according to the comparison result, and the parameters of the neural network model are adjusted according to the loss function value.

[0026] Based on the above method embodiment, the present invention provides a corresponding device embodiment;

[0027] The present invention provides a device for real-time coordination of digital mouth, lips and body parts, comprising:

[0028] Media content acquisition module, text segmentation module, image cutting module, video synthesis module, target video extraction module, video splicing module and video merging module;

[0029] The above-mentioned media content acquisition module is used to obtain digital human images, answer texts and several different body movement videos; wherein different body movement videos correspond to different keywords;

[0030] The text segmentation module is used to segment the answer text into a plurality of sub-texts and generate a speech segment corresponding to each sub-text;

[0031] The image cutting module is used to cut the digital human image at the neck to obtain a first image including the head;

[0032] The video synthesis module is used to obtain a lip-sync video based on the voice clip and the first image;

[0033] The target video extraction module is used to extract the body movement video corresponding to the keyword to obtain the target body movement video;

[0034] The video splicing module is used to splice the target body movement video and lip synchronization video at the cut position to obtain the lip and body coordination video corresponding to each sub-text;

[0035] The video merging module is used to merge the above-mentioned collaborative videos in sequence to obtain a continuous real-time collaborative video of lips and body parts.

[0036] Furthermore, the above-mentioned text segmentation module includes: a punctuation mark recognition submodule and a subtext generation submodule;

[0037] The punctuation mark recognition submodule is used to recognize all punctuation marks in the answer text;

[0038] The sub-text generation sub-module is used to divide the answer text into a number of sub-texts according to the punctuation marks.

[0039] Furthermore, the sub-text generation sub-module includes: a regular expression generation unit and a text segmentation unit;

[0040] The regular expression generation unit is used to convert all the punctuation marks into regular expressions;

[0041] The text segmentation unit is used to segment the answer text using the regular expression and remove empty strings to obtain the multiple sub-texts.

[0042] Furthermore, the above-mentioned image cutting module includes: a model recognition unit and an image cutting unit;

[0043] The model recognition unit is used to input the digital human image into a preset neck recognition model, so that the neck recognition model recognizes the neck area of ​​the human object in the digital human image;

[0044] The image cutting unit is used to cut the digital human image according to the identified neck area to obtain a first image including the head.

[0045] Based on the above method embodiment, the present invention provides a terminal device embodiment;

[0046] The present invention provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, a method for real-time coordination of digital mouth, lips and body parts as described in any embodiment of the present invention is implemented.

[0047] Based on the above method embodiment, the present invention provides a storage medium embodiment;

[0048] The present invention provides a storage medium, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, a method for real-time coordination of digital mouth, lips and body parts as described in any embodiment of the present invention is implemented.

[0049] The embodiments of the present invention have the following beneficial effects:

[0050] The present invention provides a method, device, terminal device and storage medium for real-time coordination of digital human lips and limbs. The above method first obtains a digital human image, an answer text and several different limb action videos; wherein different limb action videos correspond to different keywords; then the above answer text is divided into several sub-texts, and a voice segment corresponding to each sub-text is generated; then the above digital human image is cut at the neck to obtain a first image containing the head; then a lip synchronization video is obtained based on the above voice segment and the above first image; then the limb action video corresponding to the above keyword is extracted to obtain a target limb action video; then the above target limb action video and the lip synchronization video are spliced ​​at the cutting point to obtain a coordinated video of lips and limbs corresponding to each sub-text; finally, the above coordinated videos are merged in sequence to obtain a continuous real-time coordinated video of lips and limbs. Therefore, the present invention cuts the answer text into several small texts, and then processes each small text in turn to obtain the voice segment corresponding to each small text and the lip synchronization video corresponding to each voice segment, and then identifies the keywords in the small text, splices the body movement video corresponding to each keyword with the lip synchronization video, so as to achieve the effect of synchronizing the body movement of the 2D digital human with the content of the question and answer. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a flowchart of a method for real-time coordination of digital mouth lips and body parts provided by an embodiment of the present invention.

[0052] Figure 2 It is a schematic diagram of the structure of a device for real-time coordination of digital lips and limbs provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] like Figure 1 As shown, a method for real-time coordination of digital mouth lips and body parts provided by an embodiment of the present invention includes:

[0055] Step S101: obtaining a digital human image, answer text, and several different body movement videos; wherein different body movement videos correspond to different keywords;

[0056] Specifically, the above answer text is the answer text of the user's question given by the big model.

[0057] In this step, the digital human image to be processed, the answer text and several videos of different body movements are obtained.

[0058] Step S102: dividing the answer text into a plurality of sub-texts, and generating a speech segment corresponding to each sub-text;

[0059] Specifically, a plurality of subtexts are synthesized into speech segments corresponding to each subtext through a speech synthesis interface.

[0060] In a preferred embodiment, the above-mentioned answer text is divided into several sub-texts, including:

[0061] Identify all punctuation marks in the above answer text;

[0062] According to the above punctuation marks, the above answer text is divided into several sub-texts.

[0063] Preferably, the complete answer text is divided into several sub-texts to facilitate the subsequent separate processing of each sub-text, thereby greatly shortening the processing time of each subsequent step.

[0064] In this preferred embodiment, the answer text is segmented according to punctuation marks in the answer text to obtain a number of sub-texts.

[0065] In another preferred embodiment, the answer text is divided into several sub-texts according to the punctuation marks, including:

[0066] Convert all the above punctuation marks into regular expressions;

[0067] Use the above regular expression to segment the above answer text and remove empty strings to obtain the above sub-texts.

[0068] Specifically, the segmentation algorithm is as follows:

[0069] def split_by_multiple_punctuation(text,punctuation_list):

[0070] #Convert punctuation to regular expressions

[0071] pattern='|'.join(re.escape(p)for p in punctuation_list)

[0072] #Use regular expressions to cut

[0073] words=re.split(pattern,text)

[0074] #Remove empty strings

[0075] return[word for word in words if word]

[0076] punctuation_list=[',',',','. ','.','! ','? ',';']

[0077] sentences=split_by_multiple_punctuation(answer,punctuation_list).

[0078] In this preferred embodiment, punctuation marks are converted into regular expressions through a segmentation algorithm, and the answer text is segmented using the regular expressions, and then empty strings are removed to obtain a number of sub-texts.

[0079] Step S103: cutting the digital human image at the neck to obtain a first image including the head;

[0080] In a preferred embodiment, the digital human image is cut at the neck to obtain a first image including the head, including:

[0081] Inputting the digital human image into a preset neck recognition model so that the neck recognition model recognizes the neck area of ​​the human object in the digital human image;

[0082] According to the identified neck area, the digital human image is cut to obtain a first image including the head;

[0083] The construction of the neck recognition model includes:

[0084] Acquire a number of digital human image samples, and mark the neck area of ​​each of the number of digital human image samples to obtain a number of training samples;

[0085] The neural network model is iteratively trained with each training sample as input and the neck prediction area corresponding to each training sample as output until the neural network model converges, and the converged neural network model is used as the above-mentioned neck recognition model; wherein, in each iterative training process, the neck prediction area is compared with the actual neck area corresponding to the training sample, the loss function value is calculated according to the comparison result, and the parameters of the neural network model are adjusted according to the loss function value.

[0086] Preferably, the digital human image can also be cut using software such as PS.

[0087] In this preferred embodiment, the neck recognition model is trained to obtain a trained neck recognition model, and then the digital human image is input into the neck recognition model so that the neck recognition model recognizes the neck area in the digital human image, and the digital human image is cut according to the recognized neck area to obtain a first image containing the head.

[0088] Step S104: obtaining a lip-sync video according to the voice clip and the first image;

[0089] Preferably, the voice segment and the first image are synthesized into a lip-synced video using a voice-driven algorithm wav2lip, and the lip-synced video is transmitted to the front end in the form of a stream.

[0090] In this step, a lip-sync video corresponding to each voice segment is obtained through the voice segment and the first image.

[0091] Step S105: extracting the body movement video corresponding to the above keyword to obtain the target body movement video;

[0092] Specifically, keywords in several sub-texts are first identified, and the body movement videos corresponding to the keywords are identified from a body movement video and keyword correspondence table.

[0093] In this preferred embodiment, the target body movement video is obtained by extracting the body movement video corresponding to the keyword.

[0094] Step S106: splicing the above-mentioned target body movement video and lip synchronization video at the cut position to obtain the lip and body coordination video corresponding to each sub-text;

[0095] Specifically, the cutting position is the cutting position where the digital human is cut at the neck of the digital human image.

[0096] In this preferred embodiment, by splicing the target body movement video and the lip synchronization video, a coordinated video of the lips and body corresponding to each sub-text is obtained.

[0097] Step S107: Merge the above collaborative videos in sequence to obtain a continuous real-time collaborative video of lips and body parts.

[0098] Preferably, the collaborative videos are merged in sequence and played continuously to produce a real-time effect.

[0099] In this preferred embodiment, by merging the collaborative videos corresponding to each sub-text, a complete and continuous real-time collaborative video of the lips and body is obtained.

[0100] Based on the above method embodiment, the present invention provides a corresponding device embodiment.

[0101] like Figure 2 As shown, an embodiment of the present invention provides a device for real-time coordination of digital mouth, lips and body parts, including:

[0102] Media content acquisition module, text segmentation module, image cutting module, video synthesis module, target video extraction module, video splicing module and video merging module;

[0103] The above-mentioned media content acquisition module is used to obtain digital human images, answer texts and several different body movement videos; wherein different body movement videos correspond to different keywords;

[0104] The text segmentation module is used to segment the answer text into a plurality of sub-texts and generate a speech segment corresponding to each sub-text;

[0105] The image cutting module is used to cut the digital human image at the neck to obtain a first image including the head;

[0106] The video synthesis module is used to obtain a lip-sync video based on the voice clip and the first image;

[0107] The target video extraction module is used to extract the body movement video corresponding to the keyword to obtain the target body movement video;

[0108] The video splicing module is used to splice the target body movement video and lip synchronization video at the cut position to obtain the lip and body coordination video corresponding to each sub-text;

[0109] The video merging module is used to merge the above-mentioned collaborative videos in sequence to obtain a continuous real-time collaborative video of lips and body parts.

[0110] In a preferred embodiment, the text segmentation module includes: a punctuation mark recognition submodule and a subtext generation submodule;

[0111] The punctuation mark recognition submodule is used to recognize all punctuation marks in the answer text;

[0112] The sub-text generation sub-module is used to divide the answer text into a number of sub-texts according to the punctuation marks.

[0113] In another preferred embodiment, the sub-text generation submodule comprises: a regular expression generation unit and a text segmentation unit;

[0114] The regular expression generation unit is used to convert all the punctuation marks into regular expressions;

[0115] The text segmentation unit is used to segment the answer text using the regular expression and remove empty strings to obtain the multiple sub-texts.

[0116] In another preferred embodiment, the above-mentioned image cutting module includes: a model recognition unit and an image cutting unit;

[0117] The model recognition unit is used to input the digital human image into a preset neck recognition model, so that the neck recognition model recognizes the neck area of ​​the human object in the digital human image;

[0118] The image cutting unit is used to cut the digital human image according to the identified neck area to obtain a first image including the head.

[0119] It should be noted that the device embodiments described above are merely schematic, wherein the modules described above as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the accompanying drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines. Ordinary technicians in this field can understand and implement it without creative work. The above schematic diagram is only an example of a device for real-time collaboration of digital mouth lips and limbs, and does not constitute a limitation on a device for real-time collaboration of digital mouth lips and limbs, and may include more or fewer components than shown in the figure, or combine certain components, or different components.

[0120] Based on the above method item embodiments, the present invention provides corresponding terminal device item embodiments.

[0121] Another embodiment of the present invention provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the method for real-time coordination of digital mouth, lips and body parts described in any embodiment of the present invention is implemented.

[0122] Exemplarily, in this embodiment, the computer program may be divided into one or more modules, which are stored in the memory and executed by the processor to implement the present invention. The one or more module elements may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the device;

[0123] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The device may include, but is not limited to, a processor, a memory;

[0124] The processor referred to herein may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The above-mentioned processor is the control center of the above-mentioned device, and uses various interfaces and lines to connect various parts of the entire device;

[0125] The above-mentioned memory can be used to store the above-mentioned computer programs and / or modules. The above-mentioned processor realizes various functions of the above-mentioned device by running or executing the computer programs and / or modules stored in the above-mentioned memory, and calling the data stored in the memory. The above-mentioned memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; in addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0126] Based on the above method item embodiments, the present invention provides a corresponding storage medium item embodiment.

[0127] Another embodiment of the present invention provides a storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the storage medium is located is controlled to execute a method for real-time coordination of digital mouth, lips and limbs as described in any embodiment of the present invention.

[0128] In this embodiment, the storage medium is a computer-readable storage medium, the computer program includes computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0129] Compared with the prior art, by implementing the above-mentioned embodiments of the present invention, the body movements of the digital human can be synchronized with the content of the question and answer.

[0130] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for real-time coordination of digital mouth lips and body parts, characterized in that: include: Obtaining a digital human image, answer text, and several videos of different body movements; different body movement videos correspond to different keywords; The answer text is divided into a plurality of sub-texts, and a speech segment corresponding to each sub-text is generated; Cutting the digital human image at the neck to obtain a first image including the head; Obtaining a lip-sync video according to the voice clip and the first picture; Extracting the body movement video corresponding to the keyword to obtain the target body movement video; The target body movement video and the lip synchronization video are spliced ​​at the cut position to obtain the lip and body coordination video corresponding to each sub-text; The collaborative videos are merged in sequence to obtain a continuous real-time collaborative video of lips and body parts.

2. The method for real-time coordination of digital mouth lips and body parts according to claim 1, characterized in that: The step of dividing the answer text into a plurality of sub-texts comprises: identifying all punctuation marks in the answer text; According to the punctuation marks, the answer text is divided into a number of sub-texts.

3. The method for real-time coordination of digital mouth lips and body parts according to claim 2, characterized in that: The answer text is divided into a plurality of sub-texts according to the punctuation marks, including: Convert all the punctuation marks into regular expressions; The answer text is segmented using the regular expression and empty character strings are removed to obtain the plurality of sub-texts.

4. The method for real-time coordination of digital mouth lips and body parts according to claim 1, characterized in that: The step of cutting the digital human image at the neck to obtain a first image including the head includes: Inputting the digital human image into a preset neck recognition model so that the neck recognition model recognizes the neck area of ​​the human object in the digital human image; According to the identified neck area, the digital human image is cut to obtain a first image including the head; Wherein, the construction of the neck recognition model includes: Acquire a number of digital human image samples, and mark the neck area of ​​each of the number of digital human image samples to obtain a number of training samples; The neural network model is iteratively trained with each training sample as input and the neck prediction area corresponding to each training sample as output until the neural network model converges, and the converged neural network model is used as the neck recognition model; wherein, in each iterative training process, the neck prediction area is compared with the actual neck area corresponding to the training sample, the loss function value is calculated according to the comparison result, and the parameters of the neural network model are adjusted according to the loss function value.

5. A device for real-time coordination of digital mouth lips and body parts, characterized in that: include: Media content acquisition module, text segmentation module, image cutting module, video synthesis module, target video extraction module, video splicing module and video merging module; The media content acquisition module is used to acquire digital human images, answer texts, and several different body movement videos; wherein different body movement videos correspond to different keywords; The text segmentation module is used to segment the answer text into a plurality of sub-texts and generate a speech segment corresponding to each sub-text; The image cutting module is used to cut the digital human image at the neck to obtain a first image including the head; The video synthesis module is used to obtain a lip-sync video according to the voice segment and the first picture; The target video extraction module is used to extract the body movement video corresponding to the keyword to obtain the target body movement video; The video splicing module is used to splice the target body movement video and the lip synchronization video at the cut position to obtain the lip and body coordination video corresponding to each sub-text; The video merging module is used to merge the collaborative videos in sequence to obtain a continuous real-time collaborative video of lips and body parts.

6. The device for real-time coordination of digital mouth lips and body parts according to claim 5, characterized in that: The text segmentation module includes: a punctuation mark recognition submodule and a subtext generation submodule; The punctuation mark recognition submodule is used to recognize all punctuation marks in the answer text; The sub-text generation sub-module is used to divide the answer text into a plurality of sub-texts according to the punctuation marks.

7. The device for real-time coordination of digital mouth lips and body parts according to claim 6, characterized in that: The sub-text generation sub-module includes: a regular expression generation unit and a text segmentation unit; The regular expression generating unit is used to convert all the punctuation marks into regular expressions; The text segmentation unit is used to segment the answer text using the regular expression and remove empty character strings to obtain the plurality of sub-texts.

8. The device for real-time coordination of digital mouth lips and body parts according to claim 7, characterized in that: The image cutting module includes: a model recognition unit and an image cutting unit; The model recognition unit is used to input the digital human image into a preset neck recognition model so that the neck recognition model recognizes the neck area of ​​the human object in the digital human image; The image cutting unit is used to cut the digital human image according to the identified neck area to obtain a first image including the head.

9. A terminal device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the method for real-time collaboration of digital mouth lips and body parts as described in any one of claims 1 to 5 is implemented.

10. A storage medium, characterized in that: The storage medium includes a stored computer program, wherein when the computer program is running, the device where the storage medium is located is controlled to execute a method for real-time coordination of digital mouth, lips and body parts as described in any one of claims 1 to 5.

Images