Method, device, equipment and medium for generating virtual object video

Abstract

The present application relates to a method, device, equipment and medium for generating a virtual object video, and relates to the field of computer technology. The method for generating a virtual object video includes: obtaining input information; generating facial key point information of a virtual object based on the input information, and determining a target action label and a target playback duration corresponding to the input information; generating an action video frame sequence based on the target playback duration and the action video corresponding to the target action label; generating a virtual object video corresponding to the input information based on the action video frame sequence and the facial key point information. The present application improves the generation effect of virtual objects and meets the generation requirements of multimodal interactive virtual objects.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, equipment and medium for generating a virtual object video. Background Art

[0002] Multimodal interaction is one of the current trends in the development of human-computer interaction. Currently, the generation of multimodal interactive virtual objects is a hot research direction.

[0003] In the process of multimodal interaction, interactive virtual objects need to make gestures that match semantics. Specifically, in the process of communication between people, the use of body gestures plays a very important role in the communication process, such as facilitating intention understanding and emotional expression. Therefore, it is very necessary for interactive virtual objects to make gestures that match semantics according to different semantics. Summary of the invention

[0004] In view of this, the present application provides a method, device, equipment and medium for generating a virtual object video to improve the generation effect of virtual objects and meet the needs of multi-modal interactive virtual object generation.

[0005] In a first aspect, an embodiment of the present application provides a method for generating a virtual object video, comprising:

[0006] Get input information;

[0007] Generate facial key point information of the virtual object according to the input information, and determine the target action label and target playback duration corresponding to the input information;

[0008] Generate an action video frame sequence according to the target playback duration and the action video corresponding to the target action tag;

[0009] A virtual object video corresponding to the input information is generated according to the action video frame sequence and the facial key point information.

[0010] Optionally, generating facial key point information of the virtual object according to the input information includes:

[0011] Based on the input information, generating facial key point information through a pre-trained key point model;

[0012] The facial key point information is determined as the facial key point information of the virtual object.

[0013] Optionally, generating facial key point information based on the input information by using a pre-trained key point model includes:

[0014] If the input information is input text information, converting the input text information into voice information, and inputting the voice information into the key point model to obtain the face key point information generated by the key point model;

[0015] If the input information is input voice information, voice feature information corresponding to the input voice information is extracted, and the voice feature information is input into the key point model to obtain the face key point information generated by the key point model.

[0016] Optionally, determining a target action label and a target playback duration corresponding to the input information includes:

[0017] If the input information is input voice information, convert the input voice information into text information, classify the text information to obtain the target action label, and determine the audio duration corresponding to the input voice information as the target playback duration;

[0018] If the input information is input text information, the input text information is converted into target audio information, and the playback duration corresponding to the target audio information is determined as the target playback duration, and the action tag corresponding to the input text information is determined as the target action tag.

[0019] Optionally, before determining the action tag corresponding to the input text information as the target action tag, the method further includes:

[0020] Determine semantic information corresponding to the input text information, and determine a preset action tag corresponding to the semantic information as the action tag corresponding to the input text information; or,

[0021] The input text information is input into a pre-trained text classification model to obtain an action label corresponding to the text information output by the text classification model.

[0022] Optionally, generating an action video frame sequence according to the target playback duration and the action video corresponding to the target action tag includes:

[0023] Get the action video corresponding to the target action label;

[0024] Based on the target playback duration, the action video frame sequence is formed using video frames in the action video, and the video frame playback duration of the action video frame sequence matches the target playback duration.

[0025] Optionally, the forming the action video frame sequence by using video frames in the action video based on the target playback duration includes:

[0026] Using all video frames contained in the action video to form an action video frame sequence;

[0027] The target playback duration is used as the video frame playback duration of the action video frame sequence.

[0028] Optionally, the forming the action video frame sequence by using video frames in the action video based on the target playback duration includes:

[0029] Based on the target playback duration, sampling is performed on the video frames in the action video to obtain sampled video frames, where the sampled video frames include the first video frame in the action video;

[0030] The sequence formed by the sampled video frames is determined as an action video frame sequence, and the playback duration corresponding to the sequence formed by the sampled video frames is the same as the target playback duration.

[0031] Optionally, before obtaining the action video corresponding to the target action tag, the method further includes:

[0032] Obtaining skeleton graph information corresponding to the target action label;

[0033] Inputting the skeleton graph information into a pre-trained action generation model to obtain a posture action image frame sequence output by the action generation model;

[0034] Based on the posture action image frame sequence, an action video corresponding to the target action label is generated.

[0035] Optionally, generating an action video corresponding to the target action label based on the posture action image frame sequence includes:

[0036] For the gesture action image frame sequence, obtaining preset default image frame information;

[0037] According to the default image frame information, weighted smoothing processing is performed on the target image frame information in the posture action image frame sequence to obtain spliced ​​frame information corresponding to the target image frame information;

[0038] The spliced ​​frame information corresponding to the target image frame information is added to the posture action image frame sequence to form an action video corresponding to the target action label.

[0039] Optionally, obtaining an action video corresponding to the target action tag includes:

[0040] Matching the target action tag with action tags in a preset action library;

[0041] The action video corresponding to the matching action tag is extracted from the action library as the action video corresponding to the target action tag.

[0042] Optionally, before matching the target action tag with an action tag in a preset action library, the method further includes:

[0043] Get the skeleton image information corresponding to the preset action label;

[0044] Inputting the skeleton image information into a pre-trained action generation model to obtain an action image frame sequence output by the action generation model;

[0045] According to the preset default image frame information, the target image frame in the output action image frame sequence is smoothed to obtain the action video corresponding to the action label, and store it in the action library.

[0046] Optionally, before obtaining the input information, the step further includes:

[0047] Get recorded action video data;

[0048] Extracting action skeleton image information from the recorded action video data;

[0049] Model training is performed based on the action skeleton image information to obtain the action generation model.

[0050] Optionally, generating a virtual object video corresponding to the input information according to the action video frame sequence and the facial key point information includes:

[0051] Generate head image information of a virtual object based on the facial key point information;

[0052] According to the head image information of the virtual object, the head image information of each action video frame in the action video frame sequence is updated to obtain an updated action video frame sequence;

[0053] Based on the updated action video frame sequence, a virtual object video corresponding to the input information is generated.

[0054] In a second aspect, an embodiment of the present application provides a device for generating a virtual object video, including:

[0055] An input acquisition module is used to acquire input information;

[0056] A facial key point module, used to generate facial key point information of a virtual object based on the input information;

[0057] An information determination module, used to determine a target action label and a target playback duration corresponding to the input information;

[0058] An action video frame sequence module, used to generate an action video frame sequence according to the target playback duration and the action video corresponding to the target action tag;

[0059] The virtual object video module is used to generate a virtual object video corresponding to the input information according to the action video frame sequence and the facial key point information.

[0060] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; the memory is used to store computer programs; and the processor is used to implement the steps of the method for generating a virtual object video as described in any one of the first aspects when executing the program stored in the memory.

[0061] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method for generating a virtual object video as described in any one of the first aspects are implemented.

[0062] The embodiment of the present application obtains input information, generates facial key point information of the virtual object, and determines the target action label and target playback duration corresponding to the input information, so as to generate an action video frame sequence according to the action video corresponding to the target playback duration and the target action label, and then generates a virtual object video according to the action video frame sequence and the facial key point information, so that the virtual object in the virtual object video can have body movements corresponding to the semantics of the input information, thereby realizing the generation of interactive virtual image postures driven by the semantics of the input information, so that the virtual image behaves more naturally and humanely in the process of interaction with people, better expresses its own intentions and emotions, improves the generation effect of virtual objects, and meets the needs of multi-modal interactive virtual object generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0064] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0065] Figure 1 is a flowchart of a method for generating a virtual object video provided in an embodiment of the present application;

[0066] Figure 2 A flowchart of a method for generating a virtual object video provided in an optional embodiment of the present application;

[0067] Figure 3 A structural block diagram of a device for generating a virtual object video provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0069] The existing relevant action generation implementation solutions actually only involve the action generation process, but do not combine the action generation and interactive virtual digital humans to apply them to real-time physical interactive virtual digital human products.

[0070] One of the core concepts of the embodiments of the present application is to propose a method for generating a virtual object video, by inferring facial key point information and target action labels of the virtual object based on input information, and then generating an action video frame sequence of the virtual object based on the action video corresponding to the target action label, so as to generate a virtual object video based on the action video frame sequence and facial key point information, so that the virtual object in the virtual object video can have body movements corresponding to the semantics of the input information, so that the virtual image can behave more naturally and humanely in the process of interaction with people, better express its own intentions and emotions, improve the generation effect of the virtual object, and thus meet the needs of multi-modal interactive virtual object generation.

[0071] Reference Figure 1 , shows a flowchart of the steps of a method for generating a virtual object video provided by an embodiment of the present application. Specifically, the method for generating a virtual object video provided by the present application may include the following steps:

[0072] Step 110, obtaining input information.

[0073] The input information may include various information input by the user, such as voice information, text information, etc. input by the user, and the embodiment of the present application does not impose any specific restrictions on this.

[0074] Step 120 , generating facial key point information of the virtual object according to the input information, and determining a target action label and a target playback duration corresponding to the input information.

[0075] Specifically, after obtaining input information, the embodiment of the present application can generate facial key point information of the virtual object based on the obtained input information by inference. For example, in the case where the input information is voice information input by the user, the voice information input by the user can be input into a pre-trained key point model, and 68 key point information of the face can be generated by inference through the key point model as the facial key point information of the virtual object; for another example, in the case where the input information is input text information, the input text information can be converted into voice information through voice conversion, such as based on a text to speech (Text To Speech, TTS) model, the input text information is converted into voice information, and the generated voice information can be output through a pre-trained key point model. 68 key point information of the face can be used as the facial key point information of the virtual object.

[0076] In addition, the embodiment of the present application can infer the action label corresponding to the semantics of the input information based on the acquired input information as the target action label; and the voice duration corresponding to the input information can be determined as the target playback duration. Further, the embodiment of the present application determines the target action label corresponding to the input information, which can specifically include: determining the semantic information corresponding to the input information; extracting the preset action label corresponding to the semantic information as the target action label corresponding to the input information. For example, action keywords can be extracted from the input information as the semantic information corresponding to the input information, and then it can be seen that the preset action label corresponding to the action keyword is determined as the target action label.

[0077] Step 130 , generating an action video frame sequence according to the target playback duration and the action video corresponding to the target action tag.

[0078] Specifically, after determining the target action tag and the target playback duration, the embodiment of the present application can generate or obtain a posture action image frame sequence according to the target action tag, so as to use the generated or obtained posture action image frame sequence as the action video corresponding to the target action tag, and then dynamically adjust the action video corresponding to the target action tag based on the target playback duration, so that the video frame playback duration of the adjusted action video matches the target playback duration, so as to use the video frames in the adjusted action video to form an action video frame sequence corresponding to the input information. Among them, the video frame playback duration of the action video can refer to the duration required to play all the video frames contained in the action video.

[0079] In an optional embodiment, the embodiment of the present application generates an action video frame sequence based on the target playback duration and the action video corresponding to the target action tag, which may specifically include: obtaining the action video corresponding to the target action tag; based on the target playback duration, using the video frames in the action video to form the action video frame sequence, and the video frame playback duration of the action video frame sequence matches the target playback duration. Specifically, the embodiment of the present application can store the action videos corresponding to various preset action tags in the action library in advance. After the target action tag is determined by reasoning based on the input information, the target action tag can be matched with the action tag in the action library, and the action video corresponding to the action tag matching the target action tag can be extracted from the action library as the action video corresponding to the target action tag. Then, the action video can be dynamically deleted according to the target playback time, so as to form an action video frame sequence using the video frames in the action video, so that the video frame duration of the finally formed action video frame sequence matches the target playback time, thereby ensuring the time consistency of the finally generated action video frame sequence and the voice corresponding to the input information, and realizing the real-time generation of the action video frame sequence matching the input information, so that the action posture video of the virtual interactive object can be generated according to the action video frame sequence corresponding to the input information, that is, executing step 140. Among them, the video frame duration of the action video frame sequence can refer to the duration required to play all the action video frames contained in the action video frame sequence. The action library can refer to a database for storing action tags and action videos corresponding to the action tags.

[0080] Step 140: Generate a virtual object video corresponding to the input information based on the action video frame sequence and the facial key point information.

[0081] Specifically, after obtaining the action video frame sequence, the embodiment of the present application can generate the action posture video of the virtual object capable of physical interaction based on the action video frame sequence and the facial key point information of the virtual object through a preset network model, such as the generator CAN in the conditional generative adversarial network (CGAN) model, as the virtual object video corresponding to the input information. For example, the head image information of the virtual object in each action video frame in the action video frame sequence can be cut off, and the other part of the information can be retained, that is, the non-head image information of the virtual object in each action video frame in the action video frame sequence can be retained, and the facial key point information of the virtual object can be used as a condition, based on the optimized conditional generative adversarial network CGAN, to generate a virtual object video capable of physical interaction, such as generating a virtual digital human video capable of physical interaction, so as to achieve the purpose of generating a virtual image capable of physical interaction.

[0082] In summary, after obtaining input information, the embodiment of the present application generates facial key point information of the virtual object based on the input information, and determines the target action label and target playback time corresponding to the input information, and then generates an action video frame sequence based on the action video corresponding to the target playback time and the target action label, so as to generate a virtual object video according to the action video frame sequence and the facial key point information, so that the virtual object in the virtual object video can have body movements corresponding to the semantics of the input information, and realizes the generation of interactive virtual image postures driven by the semantics of the input information, so that the virtual image behaves more naturally and humanely in the process of interaction with people, better expresses its own intentions and emotions, improves the generation effect of virtual objects, and meets the needs of multimodal interactive virtual object generation.

[0083] In actual processing, after obtaining the input information, the embodiment of the present application can use the pre-trained key point model to infer and generate the facial key point information of the virtual object. For example, the input information can be input into the pre-trained key point model to generate the facial key point information of the digital human through the key point model, so that a semantic-based interactive posture digital human video can be generated based on the inferred generated facial key point information and the action video frame sequence corresponding to the input information, as the virtual object video corresponding to the input information. Optionally, based on the above embodiment, the embodiment of the present application generates the facial key point information of the virtual object based on the input information, which can specifically include: based on the input information, generating the facial key point information through the pre-trained key point model; determining the facial key point information as the facial key point information of the virtual object.

[0084] Reference Figure 2 , shows a flowchart of a method for generating a virtual object video provided by an optional embodiment of the present application. Figure 2 As shown, the method for generating a virtual object video provided in the embodiment of the present application may specifically include the following steps:

[0085] Step 201, obtaining input information.

[0086] Step 202: Based on the input information, generate facial key point information through a pre-trained key point model, and determine the facial key point information as facial key point information of the virtual object.

[0087] In actual processing, the embodiment of the present application can pre-train a key point model using video data, so that the trained key point model can be used to generate corresponding facial key point information based on the input voice information in the subsequent model application stage. The video data may include one or more video frame data, and each video frame data includes audio frame data and image frame data. For example, after obtaining the video data for model training, one or more audio frame data and image frame data corresponding to each audio frame data can be extracted from the video data; then, based on the image frame data corresponding to each audio frame data, the target 3D information corresponding to each audio frame data can be generated through a preset 3D model, and the corresponding facial key point information can be extracted from the image frame data corresponding to each audio frame data, so as to perform model training based on the target 3D information, audio frame data and facial key point information, and obtain a key point model, so that the corresponding facial key point information can be generated based on the sound inference through the key point information. It should be noted that the target 3D information may include main component information representing the shape and texture of the object. For example, when the target 3D information is face 3D information, the target 3D information may include main component information representing the face shape and texture, which may specifically include: expression parameters (expression), shape parameters (face_id / shape), affine parameters (projection), face texture parameters (texture), posture (pose) parameter information, etc. This embodiment does not limit this.

[0088] Furthermore, after obtaining the input information, the embodiment of the present application can determine whether the input information is input voice information or input text information, so as to generate facial key point information by inferring through a pre-trained key point model based on the voice information corresponding to the input voice information or the input text information, and the facial key point information can include mouth shape key point information. Optionally, the embodiment of the present application generates facial key point information through a pre-trained key point model based on the input information, which can specifically include: if the input information is input text information, converting the input text information into voice information, and inputting the voice information into the key point model to obtain the facial key point information generated by the key point model; if the input information is input voice information, extracting the voice feature information corresponding to the input voice information, and inputting the voice feature information into the key point model to obtain the facial key point information generated by the key point model. Specifically, when the input information obtained is input text information, the input text information can be converted into voice through a voice conversion method. For example, based on a TTS model, the input text information can be converted into voice information, and then the converted voice information can be input into a pre-trained key point model, and the key point model is used to infer the key point information of the face, which can be used as the facial key point information of the virtual object. When the input information is input voice information, the input voice information can be feature extracted to obtain voice feature information corresponding to the input voice information, and then the voice feature information corresponding to the input voice information can be input into the pre-trained key point model, so as to use the voice feature information to perform inference through the key point model to generate facial key point information, which can be used as the facial key point information of the virtual object.

[0089] Step 203: Determine the target action label and target playback duration corresponding to the input information.

[0090] In actual processing, the embodiment of the present application can perform text classification based on the input information to determine the target action label corresponding to the input information. For example, when the input information is input voice information, the input voice information can be converted into text information, and the text information obtained after the conversion can be used for classification to obtain the action label corresponding to the text information, and the action label corresponding to the text information can be determined as the target action label; for another example, when the input information is input text information, it can be directly classified based on the input text information to obtain the action label corresponding to the input text information as the target action label.

[0091] Furthermore, the embodiment of the present application determines the target action label and target playback time corresponding to the input information, which may specifically include: if the input information is input voice information, converting the input voice information into text information, and classifying it according to the text information to obtain the target action label, and determining the audio time corresponding to the input voice information as the target playback time; if the input information is input text information, converting the input text information into target audio information, and determining the playback time corresponding to the target audio information as the target playback time, and determining the action label corresponding to the input text information as the target action label.

[0092] In an optional embodiment, before determining the action label corresponding to the input text information as the target action label, the embodiment of the present application may also include: determining the semantic information corresponding to the input text information, and determining the preset action label corresponding to the semantic information as the action label corresponding to the input text information; or, inputting the input text information into a pre-trained text classification model to obtain the action label corresponding to the text information output by the text classification model. Among them, the text classification model can be used to output the corresponding action label based on the input text information. For example, in the case where the Transformer model is used as a pre-trained text classification model, the text information converted from the input text information or the input voice information can be input into the Transformer model, so as to pass through the Transformer model and output the corresponding action label as the target action label corresponding to the input information.

[0093] In actual processing, corresponding action video data can be recorded for different actions and the texts and scenes corresponding to different actions, so as to use the recorded action video data for model training, so as to obtain the correspondence between the semantics of different texts and body movements through the trained text classification model.

[0094] As an example of the present application, 20 conventional actions and their corresponding different texts and scenarios can be first extracted, and then action video data can be recorded. For example, a waving action can correspond to "hello" or "goodbye". After recording action video data corresponding to different semantics, the recorded action video data can be used to train a unique action label corresponding to each short text using a Transformer model. As a result, when the interactive virtual image interacts with a person, the trained Transformer model can be used to perform a text classification task on the acquired input information to obtain the action label corresponding to the text as the target action label corresponding to the input information.

[0095] It should be noted that the text classification model can not only be a Transformer model, but also other types of models, such as an Lstm model or a Bert model, etc. The embodiment of the present application specifically restricts the type steps of the text classification model used to perform classification tasks.

[0096] Step 204: Obtain the action video corresponding to the target action tag.

[0097] Specifically, after determining the target action tag corresponding to the input information, the embodiment of the present application can match the target action tag with the action tags in the action library to extract a pre-generated action video corresponding to the action tag matching the target action tag from the action library as the action video corresponding to the target action tag.

[0098] In an optional embodiment, the step of obtaining the action video corresponding to the target action tag in the embodiment of the present application may specifically include: matching the target action tag with the action tags in a preset action library; extracting the action video corresponding to the matching action tag from the action library as the action video corresponding to the target action tag. Specifically, the embodiment of the present application may compare the target action tag with the action tags stored in the preset action library to find the action tag identical to the target action tag in the action library, thereby determining the action tag identical to the target action tag as the action tag matching the target action tag, and then extracting the action video corresponding to the matching action tag from the action library, so as to determine the action video corresponding to the matching action tag as the action video corresponding to the target action tag.

[0099] Furthermore, in order to ensure the stability and smooth transition of the action video, when generating the action video corresponding to the action label, the embodiment of the present application can use the first frame of the preset default action to smooth the target image frame in the action image frame sequence corresponding to the action label to obtain the action video corresponding to the action label, and the obtained action video corresponding to the action label can be saved in the action library, so that the action video corresponding to the target action label can be obtained from the action library later. Optionally, the method for generating a virtual object video provided by the embodiment of the present application, before matching the target action label with the action label in the preset action library, can also include: obtaining the skeletal image information corresponding to the preset action label; inputting the skeletal image information into a pre-trained action generation model to obtain the action image frame sequence output by the action generation model; according to the preset default image frame information, smoothing the target image frame in the output action image frame sequence to obtain the action video corresponding to the action label, and storing it in the action library. Among them, the skeleton map information may include the skeleton point information and / or depth map information of the action posture, and the implementation example of the present application does not limit this; the preset default image frame information may refer to a video frame image of the preset default action, such as the first video image frame of the default action; the target image frame in the action image frame sequence may refer to the image frame corresponding to the action image frame sequence to which the splicing frame needs to be added, such as the first image frame in the action image frame sequence, or the last image frame in the action image frame sequence, etc., and the present embodiment does not limit this. It should be noted that the skeleton point information of the action posture can be used to determine the skeleton point of the action posture, such as the coordinate point information of the skeleton point of a virtual object such as a digital human when performing a certain action posture; the depth map information of the action posture may refer to the depth map of a virtual object such as a digital human when performing a certain action posture.

[0100] As an example of the present application, an action tag can be selected from all action tags as a default action tag to set the action corresponding to the selected action tag as the default action. For example, a still action tag can be selected from all action tags as the default action tag, that is, the still action corresponding to the selected still action tag is preset as the default action, and the first frame image of the default action can be extracted as the preset default image frame information, so that a spliced ​​frame can be subsequently generated based on the first frame image of the default action, so that a new action image frame sequence can be formed by adding the spliced ​​frame to the action image frame sequence corresponding to the action tag, so as to generate an action video corresponding to the action tag based on the new action image frame sequence, thereby ensuring a smooth and smooth transition of the action video and solving the problem of frame jumping when splicing different actions in the original action video. Specifically, after the skeletal image information corresponding to the action label is input into a pre-trained action generation model, the action generation model can be used to generate and output an action image frame sequence corresponding to the action label. Subsequently, the action image frame sequence corresponding to the action label obtains preset default image frame information, so as to perform weighted smoothing processing on the target image frame information in the action image frame sequence according to the default image frame information, and add the spliced ​​frame information obtained after the weighted smoothing processing to the action image frame sequence, thereby implementing weighted smoothing processing during the adding process, thereby ensuring a smooth and smooth transition of the action video, and solving the problem of frame jumping when splicing different actions in the original action video.

[0101] For example, five first-frame skeleton points or depth maps of default actions can be added to the skeleton points or depth map data corresponding to all action labels, and weighted smoothing can be performed during the adding process, so as to ensure the smoothness of the action video corresponding to the action label and the smooth transition between different action videos. Taking the OK action as an example, when the action image frame sequence corresponding to the OK action label contains 100 image frames, the first image frame in the action image frame sequence can be used as the target image frame information, and the first image frame in the action image frame sequence can be weighted smoothed using the preset default image frame information in a weighted smoothing manner to obtain a spliced ​​frame corresponding to the first image frame, and the spliced ​​frame corresponding to the first image frame can be added to the action image frame sequence to add five first-frame skeleton points or depth maps of the default action at the beginning of the Ok action to form an action video corresponding to the OK action label. For example, the stitched frames added to the first frame may be the skeleton points or depth map of the default action; the stitched frames added to the second frame may be the image frames obtained by summing the static action and the OK action, wherein the static action accounts for 0.8 times of the action in the image frame, and the OK action accounts for 0.2 times of the action in the image frame; the stitched frames added to the third frame may be the image frames wherein the static action accounts for 0.6 times and the OK action accounts for 0.4 times; the stitched frames added to the fourth frame may be the image frames wherein the static action accounts for 0.4 times and the OK action accounts for 0.6 times; the stitched frames added to the fifth frame may be the image frames wherein the static action accounts for 0.2 times and the OK action accounts for 0.8 times. Similarly, 5 corresponding splicing frames can be added at the end of the OK action. For example, the splicing frames corresponding to the last image frame in the action image frame sequence can be generated in the manner of 1, 0.8, 0.6, 0.4, 0.2, and 0 according to the weight of the OK action, so that 5 corresponding splicing frames can be inserted into the last image frame in the action image frame sequence, so that the OK action video can be freely spliced ​​with the still action video, thereby improving the generation effect of the action video.

[0102] Of course, in addition to obtaining the action video corresponding to the target action label from the action library, the embodiment of the present application can also obtain the action video corresponding to the target action video in other ways, such as generating the corresponding action video in real time according to the preset skeletal graph information corresponding to the target action label, as the action video corresponding to the target action label, and the embodiment of the present application does not impose specific restrictions on this. In another optional embodiment, the preset skeletal graph information corresponding to the target action label can be input into a pre-trained action generation model, so that the action generation model can be used to infer and generate a posture action image frame sequence using the preset skeletal graph information corresponding to the target action label, and then the action video corresponding to the target action label can be generated based on the inferred posture action image frame sequence.

[0103] Optionally, the method for generating a virtual object video provided in an embodiment of the present application may further include, before obtaining input information: obtaining recorded action video data; extracting action skeleton image information from the recorded action video data; and performing model training based on the action skeleton image information to obtain the action generation model. The recorded action video data may refer to action video data recorded for an action tag. For example, in the case where action videos corresponding to 20 action tags are required, video data of at least 20 actions may be recorded. Of course, in addition to video data of 20 actions, the recorded video data may also include video data of other actions, and the recording time may be greater than or equal to 1 hour, which is not limited in this embodiment.

[0104] It should be noted that the input of the action generation model can be the skeleton point information and / or depth map information of the action posture, and the embodiment of the present application does not limit this. Specifically, the embodiment of the present application can extract the body skeleton points and the corresponding depth map from the recorded action video data to perform model training as the action skeleton map information to obtain the action generation model. For example, based on the open source OpenPose and DensePose models, the recorded action video data can be extracted to extract the body skeleton points and the corresponding depth map as the action skeleton map information for model training, and then based on the generative adversarial model GAN, the action video frame sequence corresponding to the action skeleton map information is generated, and then the action video frame sequence can be compared with the action video frame sequence in the recorded video data to determine whether the loss of the current training model meets the preset model convergence condition, so that the model convergence can be determined when the loss of the current training model meets the preset model convergence condition, and the model can be used as the action generation model, so that the subsequent model application stage can use the action generation model to infer and generate action image frame sequences. After the model is trained, the corresponding skeleton graph information can be obtained according to the target action label corresponding to the input information, so as to input the skeleton graph information corresponding to the target action label into the trained action generation model, and infer and generate a posture action image frame sequence corresponding to the target action label. Therefore, the action video corresponding to the target action label can be generated based on the posture action image frame sequence, and then the action video corresponding to the target action label can be used to generate the action posture video of the virtual interactive object, so as to meet the requirements of multi-modal interactive virtual object generation.

[0105] Furthermore, the method for generating a virtual object video provided by an embodiment of the present application may also include, before obtaining the action video corresponding to the target action label, the following: obtaining the skeleton diagram information corresponding to the target action label; inputting the skeleton diagram information into a pre-trained action generation model to obtain a posture action image frame sequence output by the action generation model; and generating the action video corresponding to the target action label based on the posture action image frame sequence. Optionally, the embodiment of the present application generates the action video corresponding to the target action label based on the posture action image frame sequence, which may specifically include: obtaining preset default image frame information for the posture action image frame sequence; performing weighted smoothing processing on the target image frame information in the posture action image frame sequence based on the default image frame information to obtain the spliced ​​frame information corresponding to the target image frame information; and adding the spliced ​​frame information corresponding to the target image frame information to the posture action image frame sequence to form the action video corresponding to the target action label.

[0106] Step 205 : Based on the target playback duration, an action video frame sequence is formed using video frames in the action video, and the video frame playback duration of the action video frame sequence matches the target playback duration.

[0107] In an optional implementation, the embodiment of the present application forms the action video frame sequence using the video frames in the action video based on the target playback duration, which may specifically include: using all the video frames contained in the action video to form the action video frame sequence; using the target playback duration as the video frame playback duration of the action video frame sequence. For example, in the case where the input information is input voice information, the target playback duration is the audio duration of the input voice information, and the FFMPEG tool can be used to accelerate or slow down the playback speed of the video frames of the action video frame sequence according to the audio duration, so that the video frame playback duration of the action video frame sequence matches the target playback duration, that is, the playback duration of the video frames of the action video matches the duration of the voice, and the action video frame sequence corresponding to the action label is obtained, so that it can be combined with the facial contour line later and input into the digital human generation model to generate the final digital human that can interact physically, meeting the generation requirements of multimodal interactive virtual digital humans.

[0108] In another optional embodiment, the embodiment of the present application forms the action video frame sequence using the video frames in the action video based on the target playback duration, which may specifically include: based on the target playback duration, sampling the video frames in the action video to obtain sampled video frames, the sampled video frames including the first video frame in the action video; determining the sequence formed by the sampled video frames as an action video frame sequence, and the playback duration corresponding to the sequence formed by the sampled video frames is the same as the target playback duration. For example, based on the audio duration of the input voice information, the video frames in the action video may be upsampled or downsampled, but the first frame in the action generated video may not be deleted to obtain a sampled video frame sequence, and the sampled video frame sequence may be used as the action video frame sequence corresponding to the input voice information to ensure the temporal consistency of the audio and action video, so that the virtual object video corresponding to the input information may be subsequently generated based on the action video frame sequence corresponding to the input voice information.

[0109] Step 206: Generate a virtual object video corresponding to the input information based on the action video frame sequence and the facial key point information.

[0110] Further, the embodiment of the present application generates a virtual object video corresponding to the input information based on the action video frame sequence and the facial key point information, which may specifically include: generating head image information of the virtual object based on the facial key point information; updating the head image information of each action video frame in the action video frame sequence based on the head image information of the virtual object to obtain an updated action video frame sequence; based on the updated action video frame sequence, generating a virtual object video corresponding to the input information. Specifically, the facial key point information generated by inference in the embodiment of the present application may include facial key point information corresponding to each action video frame in the action video frame sequence. After generating a sequence of action video frames, the facial key point information corresponding to each action video frame can be used to generate the head image information of the corresponding virtual object, and for each action video frame in the action video frame sequence, the head image information of the virtual object corresponding to the action video frame can be used to update the head image information in the action video frame, so as to merge the head image information of the virtual object corresponding to the action video frame with the non-head image information in the action video frame, crop the original head image in the action video frame, and generate the corresponding virtual object video frame, so that the generated virtual object video frames can be used to generate a virtual object video that can interact with the body, and then the virtual object video that can interact with the body can be used to interact in real time with the user who inputs the information, thereby improving the user experience and meeting the requirements for generating multi-modal interactive virtual objects.

[0111] It can be seen that after obtaining the input information, the embodiment of the present application can generate the facial key point information and target action label of the virtual object based on the input information by inference, and then generate the action video frame sequence of the virtual object based on the action video corresponding to the target action label, so as to generate the virtual object video based on the action video frame sequence and the facial key point information, so that the virtual object in the virtual object video can have body movements corresponding to the semantics of the input information, so that the virtual image can perform more naturally and humanely in the process of interaction with people, better express its own intentions and emotions, improve the generation effect of virtual objects, and meet the needs of multi-modal interactive virtual object generation.

[0112] In addition, in the process of generating action videos corresponding to action labels, the embodiments of the present application use preset default image frame information to perform weighted smoothing processing on the action image frame sequence generated by reasoning corresponding to the action label, that is, adding the spliced ​​frame information to the action image frame sequence, and performing weighted smoothing processing during the adding process, thereby ensuring the stability and smooth transition of the generated action video, solving the problem of frame jumping when splicing different actions in the original action video, ensuring that different actions can be switched without frame skipping or lag, and improving the generation effect of the action video.

[0113] It should be noted that, for the purpose of simple description, the method embodiments are expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited to the described order of actions, because according to the embodiments of the present application, certain steps may be performed in other orders or simultaneously.

[0114] The present application also provides a device for generating a virtual object video. Figure 3 As shown, the device for generating a virtual object video provided in an embodiment of the present application includes the following modules:

[0115] An input acquisition module 310 is used to acquire input information;

[0116] A facial key point module 320, used to generate facial key point information of a virtual object according to the input information;

[0117] An information determination module 330 is used to determine a target action label and a target playback duration corresponding to the input information;

[0118] An action video frame sequence module 340 is used to generate an action video frame sequence according to the target playback duration and the action video corresponding to the target action tag;

[0119] The virtual object video module 350 is used to generate a virtual object video corresponding to the input information according to the action video frame sequence and the facial key point information.

[0120] Optionally, the facial key point module 320 specifically includes:

[0121] A generating submodule, used to generate facial key point information based on the input information through a pre-trained key point model;

[0122] The determination submodule is used to determine the facial key point information as the facial key point information of the virtual object.

[0123] Optionally, the generation submodule is specifically used to: when the input information is input text information, convert the input text information into voice information, and input the voice information into the key point model to obtain facial key point information generated by the key point model; when the input information is input voice information, extract voice feature information corresponding to the input voice information, and input the voice feature information into the key point model to obtain facial key point information generated by the key point model.

[0124] Optionally, the information determination module 330 includes: a first determination submodule and a second determination submodule. The first determination submodule is used to convert the input voice information into text information when the input information is input voice information, and classify it according to the text information to obtain the target action label, and determine the audio duration corresponding to the input voice information as the target playback duration; the second determination submodule is used to convert the input text information into target audio information when the input information is input text information, and determine the playback duration corresponding to the target audio information as the target playback duration, and determine the action label corresponding to the input text information as the target action label.

[0125] Optionally, the device for generating the virtual object video further includes:

[0126] An action label determination module, used to determine semantic information corresponding to the input text information, and determine a preset action label corresponding to the semantic information as the action label corresponding to the input text information;

[0127] The action label generation module is used to input the input text information into a pre-trained text classification model to obtain the action label corresponding to the text information output by the text classification model.

[0128] Optionally, the action video frame sequence module 340 includes:

[0129] The action video acquisition submodule is used to acquire the action video corresponding to the target action label;

[0130] The action video frame sequence submodule is used to form the action video frame sequence based on the target playback duration using the video frames in the action video, and the video frame playback duration of the action video frame sequence matches the target playback duration.

[0131] Optionally, the action video frame sequence submodule is specifically used to: use all video frames contained in the action video to form an action video frame sequence; and use the target playback duration as the video frame playback duration of the action video frame sequence.

[0132] Optionally, the action video frame sequence submodule is specifically used to: based on the target playback duration, sample the video frames in the action video to obtain sampled video frames, wherein the sampled video frames include the first video frame in the action video; determine the sequence formed by the sampled video frames as an action video frame sequence, and the playback duration corresponding to the sequence formed by the sampled video frames is the same as the target playback duration.

[0133] Optionally, the device for generating the virtual object video further includes:

[0134] A skeleton diagram information acquisition module, used to acquire the skeleton diagram information corresponding to the target action label;

[0135] A posture action image frame sequence module is used to input the skeleton image information into a pre-trained action generation model to obtain a posture action image frame sequence output by the action generation model;

[0136] The action video generation module is used to generate an action video corresponding to the target action label based on the posture action image frame sequence.

[0137] Optionally, the action video generation module includes:

[0138] A default image frame information submodule, used for obtaining preset default image frame information for the gesture action image frame sequence;

[0139] A weighted smoothing processing submodule is used to perform weighted smoothing processing on the target image frame information in the posture action image frame sequence according to the default image frame information, so as to obtain spliced ​​frame information corresponding to the target image frame information;

[0140] The splicing frame information adding submodule is used to add the splicing frame information corresponding to the target image frame information to the posture action image frame sequence to form an action video corresponding to the target action label.

[0141] Optionally, the action video acquisition submodule includes:

[0142] A matching unit, used to match the target action label with an action label in a preset action library;

[0143] The extraction unit is used to extract the action video corresponding to the matching action label from the action library as the action video corresponding to the target action label.

[0144] Optionally, the device for generating the virtual object video further includes:

[0145] A skeleton image information module is used to obtain skeleton image information corresponding to a preset action label;

[0146] A skeleton image information input module, used for inputting the skeleton image information into a pre-trained motion generation model to obtain a motion image frame sequence output by the motion generation model;

[0147] The smoothing processing module is used to perform smoothing processing on the target image frame in the output action image frame sequence according to preset default image frame information, obtain the action video corresponding to the action label, and store it in the action library.

[0148] Optionally, the device for generating the virtual object video further includes:

[0149] A recording action video data module is used to obtain recording action video data;

[0150] An action skeleton image information extraction module, used to extract action skeleton image information from the recorded action video data;

[0151] The model training module is used to perform model training based on the action skeleton image information to obtain the action generation model.

[0152] Optionally, the virtual object video module 350 includes:

[0153] A head image generation submodule, used to generate head image information of a virtual object based on the facial key point information;

[0154] A head image updating submodule, used for updating the head image information of each action video frame in the action video frame sequence according to the head image information of the virtual object, to obtain an updated action video frame sequence;

[0155] The virtual object video generation submodule is used to generate a virtual object video corresponding to the input information based on the updated action video frame sequence.

[0156] It should be noted that the device for generating a virtual object video provided above can execute the method for generating a virtual object video provided in any embodiment of the present application, and has the corresponding functions and beneficial effects of the execution method.

[0157] Furthermore, an embodiment of the present application also provides an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and the processor is used to implement the steps of the method for generating a virtual object video described in any one of the above method embodiments when executing the program stored in the memory.

[0158] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method for generating a virtual object video described in any one of the above method embodiments are implemented.

[0159] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device, equipment, and storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0160] In this article, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0161] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A method for generating a virtual object video, It is characterized in that include: Acquiring input information, wherein the input information is text information or voice information; Generate facial key point information of the virtual object according to the input information, and determine the target action label and target playback duration corresponding to the input information; Generate an action video frame sequence according to the target playback duration and the action video corresponding to the target action tag; Generate a virtual object video corresponding to the input information according to the action video frame sequence and the facial key point information; The step of generating a virtual object video corresponding to the input information based on the action video frame sequence and the facial key point information includes: Generating head image information of a virtual object according to the facial key point information; According to the head image information of the virtual object, the head image information of each action video frame in the action video frame sequence is updated to obtain an updated action video frame sequence; Based on the updated action video frame sequence, a virtual object video corresponding to the input information is generated.

2. The method for generating a virtual object video according to claim 1, It is characterized in that The step of generating facial key point information of the virtual object according to the input information includes: Based on the input information, generating facial key point information through a pre-trained key point model; The facial key point information is determined as the facial key point information of the virtual object.

3. The method for generating a virtual object video according to claim 2, It is characterized in that The generating of facial key point information based on the input information by using a pre-trained key point model includes: If the input information is input text information, converting the input text information into voice information, and inputting the voice information into the key point model to obtain the face key point information generated by the key point model; If the input information is input voice information, voice feature information corresponding to the input voice information is extracted, and the voice feature information is input into the key point model to obtain the face key point information generated by the key point model.

4. The method for generating a virtual object video according to claim 1, It is characterized in that The determining the target action label and the target playback duration corresponding to the input information includes: If the input information is input voice information, convert the input voice information into text information, classify the text information to obtain the target action label, and determine the audio duration corresponding to the input voice information as the target playback duration; If the input information is input text information, the input text information is converted into target audio information, and the playback duration corresponding to the target audio information is determined as the target playback duration, and the action tag corresponding to the input text information is determined as the target action tag.

5. The method for generating a virtual object video according to claim 4, It is characterized in that Before determining the action tag corresponding to the input text information as the target action tag, the method further includes: Determine semantic information corresponding to the input text information, and determine a preset action tag corresponding to the semantic information as the action tag corresponding to the input text information; or, The input text information is input into a pre-trained text classification model to obtain an action label corresponding to the text information output by the text classification model.

6. The method for generating a virtual object video according to claim 1, It is characterized in that Generating an action video frame sequence according to the target playback duration and the action video corresponding to the target action tag, including: Get the action video corresponding to the target action label; Based on the target playback duration, the action video frame sequence is formed using video frames in the action video, and the video frame playback duration of the action video frame sequence matches the target playback duration.

7. The method for generating a virtual object video according to claim 6, It is characterized in that The forming the action video frame sequence by using the video frames in the action video based on the target playback duration includes: Using all video frames contained in the action video to form an action video frame sequence; The target playback duration is used as the video frame playback duration of the action video frame sequence.

8. The method for generating a virtual object video according to claim 6, It is characterized in that The forming the action video frame sequence by using the video frames in the action video based on the target playback duration includes: Based on the target playback duration, sampling is performed on the video frames in the action video to obtain sampled video frames, where the sampled video frames include the first video frame in the action video; The sequence formed by the sampled video frames is determined as an action video frame sequence, and the playback duration corresponding to the sequence formed by the sampled video frames is the same as the target playback duration.

9. The method for generating a virtual object video according to claim 6, It is characterized in that Before obtaining the action video corresponding to the target action tag, the method further includes: Obtaining skeleton graph information corresponding to the target action label; Inputting the skeleton graph information into a pre-trained action generation model to obtain a posture action image frame sequence output by the action generation model; Based on the posture action image frame sequence, an action video corresponding to the target action label is generated.

10. The method for generating a virtual object video according to claim 9, It is characterized in that The step of generating an action video corresponding to the target action label based on the posture action image frame sequence includes: For the gesture action image frame sequence, obtaining preset default image frame information; According to the default image frame information, weighted smoothing processing is performed on the target image frame information in the posture action image frame sequence to obtain spliced ​​frame information corresponding to the target image frame information; The spliced ​​frame information corresponding to the target image frame information is added to the posture action image frame sequence to form an action video corresponding to the target action label.

11. The method for generating a virtual object video according to claim 6, It is characterized in that The step of obtaining the action video corresponding to the target action tag includes: Matching the target action tag with action tags in a preset action library; The action video corresponding to the matching action tag is extracted from the action library as the action video corresponding to the target action tag.

12. The method for generating a virtual object video according to claim 11, It is characterized in that Before matching the target action tag with the action tag in the preset action library, the method further includes: Get the skeleton image information corresponding to the preset action label; Inputting the skeleton image information into a pre-trained action generation model to obtain an action image frame sequence output by the action generation model; According to the preset default image frame information, the target image frame in the output action image frame sequence is smoothed to obtain the action video corresponding to the action label, and store it in the action library.

13. The method for generating a virtual object video according to claim 9 or 12, It is characterized in that Before obtaining the input information, the method further includes: Get recorded action video data; Extracting action skeleton image information from the recorded action video data; Model training is performed based on the action skeleton image information to obtain the action generation model.

14. A device for generating a virtual object video, It is characterized in that include: An input acquisition module, used to acquire input information, wherein the input information is text information or voice information; A facial key point module, used to generate facial key point information of a virtual object based on the input information; An information determination module, used to determine a target action label and a target playback duration corresponding to the input information; An action video frame sequence module, used to generate an action video frame sequence according to the target playback duration and the action video corresponding to the target action tag; A virtual object video module, used to generate a virtual object video corresponding to the input information according to the action video frame sequence and the facial key point information; Wherein, the virtual object video module includes: A head image generation submodule, used to generate head image information of a virtual object based on the facial key point information; A head image updating submodule, used for updating the head image information of each action video frame in the action video frame sequence according to the head image information of the virtual object, to obtain an updated action video frame sequence; The virtual object video generation submodule is used to generate a virtual object video corresponding to the input information based on the updated action video frame sequence.

15. An electronic device, It is characterized in that It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor is used to implement the steps of the method for generating a virtual object video as described in any one of claims 1 to 13 when executing a program stored in a memory.

16. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method for generating a virtual object video as described in any one of claims 1 to 13 are implemented.

Images