A video generation method and device, and computer readable storage medium

Abstract

The embodiment of the present invention provides a video generation method and device, and a computer-readable storage medium, wherein the method includes: collecting audio and video sequences of real objects; extracting features from the audio and video sequences to determine anthropomorphic features; using a virtual prediction network, preset standard features, and a first feature to predict the anthropomorphic features to generate a video sequence of a virtual object; the preset standard features are features corresponding to the reference object; the first feature represents different attitudes; the video sequence of the virtual object is a video sequence of the virtual object's corresponding reaction generated based on the audio and video sequence of the real object; and presenting the video sequence of the virtual object. The embodiment of the present invention generates a video sequence of a virtual object based on the audio and video sequence of the real object, and the presented video sequence of the virtual object is more vivid and accurate.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction, and in particular to a video generation method and device, and a computer-readable storage medium. Background Art

[0002] From the perspective of human behavior, good communication refers to a two-way communication process, not a one-way information input or output, but an interaction of information. Real communication between people is a process of constantly switching and circulating between the two states of listening and speaking. Among them, listening and speaking are equally important. Both are essential for building anthropomorphic digital humans for human-computer interaction. On the one hand, digital humans are required to express their opinions as clearly, concisely and clearly as possible in a language that the other party understands. On the other hand, anthropomorphic digital humans must also be good at listening to and understanding other people's opinions. The existing technology mainly generates corresponding speaker videos based on the speaker's reference image and time-varying signal. The main approach is to parameterize the speaker using facial key points, 3D facial models, human skeleton models, etc., and then fit these parameters through a deep neural network, and render images with these parameters as the generated results, and the generated images have poor effects. Summary of the invention

[0003] The embodiments of the present invention provide a video generation method and device, and a computer-readable storage medium, which can generate a virtual object video sequence according to an audio and video sequence of a real object, thereby improving the vividness and accuracy of the generated virtual object video sequence.

[0004] The technical solution of the present invention is achieved in this way:

[0005] An embodiment of the present invention provides a video generation method, the method comprising:

[0006] Collect audio and video sequences of real objects;

[0007] Extracting features from the audio and video sequences to determine anthropomorphic features;

[0008] The anthropomorphic features are predicted using a virtual prediction network, a preset standard feature, and a first feature to generate a virtual object video sequence; the preset standard feature is a feature corresponding to a reference object; the first feature represents different attitudes; the virtual object video sequence is a video sequence of a virtual object's corresponding reaction generated based on an audio and video sequence of a real object;

[0009] A video sequence of the virtual object is presented.

[0010] In the above solution, the method of using a virtual prediction network, a preset standard feature, and a first feature to predict the anthropomorphic feature to generate a video sequence of a virtual object includes:

[0011] Acquire preset standard features; the preset standard features include a first posture expression feature and a first identity feature;

[0012] Based on the first posture expression feature, the anthropomorphic feature and the first feature, predicting and decoding through the virtual prediction network to determine the multi-frame posture expression features of the virtual object;

[0013] A video sequence of the virtual object is generated based on the multi-frame posture expression features of the virtual object and the first identity feature.

[0014] In the above solution, the step of obtaining the preset standard features includes:

[0015] Acquire a standard image; the standard image represents an image of a reference object;

[0016] The standard image is subjected to feature extraction through a face reconstruction model to obtain the preset standard features.

[0017] In the above scheme, the anthropomorphic features include: multi-frame anthropomorphic features corresponding to the audio and video sequence; the virtual prediction network includes: a first processing module and a second processing module; the prediction and decoding based on the first posture expression feature, the anthropomorphic feature and the first feature through the virtual prediction network to determine the multi-frame posture expression features of the virtual object include:

[0018] Based on the first frame of anthropomorphic features in the multiple frames of anthropomorphic features, the first posture expression feature and the first feature, the first processing module performs prediction to obtain a next predicted video frame; the first feature is one of a positive attitude, a negative attitude and a normal attitude;

[0019] Decoding the next predicted video frame through the second processing module to determine a next posture expression feature of the virtual object corresponding to the next predicted video frame;

[0020] Based on the next posture expression feature and the next frame of anthropomorphic features in the multi-frame anthropomorphic features, continue to predict and decode until the last posture expression feature of the virtual object corresponding to the last predicted video frame is obtained, thereby obtaining the multi-frame posture expression feature of the virtual object; the first posture expression feature is used as the first frame in the multi-frame posture expression features.

[0021] In the above solution, the step of generating a video sequence of the virtual object based on the multi-frame posture expression features of the virtual object and the first identity feature includes:

[0022] Each frame of the posture expression feature in the multiple frames of posture expression features is fused with the first identity feature to obtain a plurality of second features; the second features represent the fusion result of the identity feature and the posture expression feature;

[0023] The plurality of second features are passed through a renderer to generate a video sequence of the virtual object corresponding to the virtual object; the first identity feature includes a first identity identifier, a first material and first lighting information.

[0024] In the above solution, the audio and video sequence includes: an audio sequence of a real object and a video sequence of a real object; and the step of extracting features from the audio and video sequence to determine anthropomorphic features includes:

[0025] Preprocessing the video sequence of the real object by an encoder to obtain multiple video features;

[0026] Extracting features from the audio sequence of the real object through an encoder to obtain multiple audio features; the audio features include loudness, zero-crossing rate and cepstrum coefficients;

[0027] Based on the multiple video features and the multiple audio features, feature conversion is performed through a feature fusion function to determine the anthropomorphic feature; the anthropomorphic feature is a multi-frame anthropomorphic feature corresponding to an audio and video sequence; the anthropomorphic feature includes video features and audio features.

[0028] In the above scheme, the encoder extracts features from the video sequence of the real object to obtain multiple video features, including:

[0029] Extracting features from each video frame of the video sequence of the real object using a face reconstruction model to obtain a plurality of video frame features; the video frame features include a second identity feature and a second posture and expression feature;

[0030] All the second posture expression features corresponding to the video sequence of the real object are used as the video features.

[0031] In the above solution, before predicting the anthropomorphic feature using the virtual prediction network and the preset standard feature and the first feature to generate a video sequence of the virtual object, the method further includes:

[0032] Collect audio and video sequence samples of the real talking object and the corresponding face images of the real listening object;

[0033] Extracting features from the sample audio and video sequence through an initial encoder to determine anthropomorphic sample features;

[0034] Generate predicted facial features under audio and video sequence samples of real training objects through the initial virtual prediction network and the anthropomorphic sample features; the predicted facial features include predicted posture features and predicted expression features;

[0035] According to the face image of the real listening object, feature extraction is performed through a face reconstruction model to determine real face features, wherein the real face features include real posture features and real expression features;

[0036] Continuously optimizing the initial encoder by using a first loss function and the anthropomorphic sample features until the first loss function value meets a first preset threshold, thereby determining the encoder;

[0037] Based on the real face features and the predicted face features, the initial virtual prediction network is continuously optimized by the second loss function and the third loss function until the second loss function value and the third loss function value meet the second preset threshold, and the virtual prediction network is determined.

[0038] In the above scheme, based on the real face features and the predicted face features, the initial virtual prediction network is continuously optimized by the second loss function and the third loss function until the second loss function value and the third loss function value meet the second preset threshold value, and the virtual prediction network is determined, including:

[0039] Based on the real face features and the predicted face features, determining a second loss function; the second loss function is used to ensure that the predicted expression and the predicted posture are similar to the real expression and the real posture;

[0040] Determining a third loss function based on the change function corresponding to the real face feature and the change function corresponding to the predicted face feature; the third loss function is used to ensure that the inter-frame continuity of the predicted face feature is similar to the real face feature;

[0041] The initial virtual prediction network is continuously optimized by using the second loss function and the third loss function until the second loss function value and the third loss function value meet a second preset threshold, thereby determining the virtual prediction network.

[0042] The embodiment of the present invention provides a video generation device, characterized in that the video generation device includes an acquisition unit, a determination unit and a generation unit; wherein,

[0043] The acquisition unit is used to collect audio and video sequences of real objects;

[0044] The determining unit is used to extract features from the audio and video sequence to determine anthropomorphic features;

[0045] The generation unit is used to predict the anthropomorphic features using a virtual prediction network, preset standard features, and a first feature to generate a video sequence of a virtual object; the preset standard features are features corresponding to a reference object; the first feature represents different attitudes; the video sequence of the virtual object is a video sequence of a virtual object's corresponding reaction generated based on an audio and video sequence of a real object; and the video sequence of the virtual object is presented.

[0046] An embodiment of the present invention provides a video generation device, characterized in that the video generation device includes:

[0047] A memory for storing executable instructions;

[0048] The processor is used to execute the executable instructions stored in the memory. When the executable instructions are executed, the processor executes the video generation method.

[0049] An embodiment of the present invention provides a computer-readable storage medium, characterized in that executable instructions are stored therein, and when the executable instructions are executed by one or more processors, the processors execute the video generating method.

[0050] The embodiments of the present invention provide a video generation method and device, and a computer-readable storage medium, wherein the method includes: collecting audio and video sequences of real objects; extracting features from the audio and video sequences to determine anthropomorphic features; predicting the anthropomorphic features using a virtual prediction network, preset standard features, and a first feature to generate a video sequence of a virtual object; the preset standard features are features corresponding to a reference object; the first feature represents different attitudes; the video sequence of the virtual object is a video sequence of the virtual object's corresponding reaction generated based on the audio and video sequence of the real object; and presenting the video sequence of the virtual object. The embodiments of the present invention generate a video sequence of a virtual object based on the audio and video sequence of the real object, and the presented video sequence of the virtual object is more vivid and accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 An optional terminal operation diagram of a video generation method is provided for an embodiment of the present invention;

[0052] Figure 2 An optional process diagram of a video generation method is provided for an embodiment of the present invention Figure 1 ;

[0053] Figure 3a An optional speaker video generation schematic diagram of a video generation method is provided for an embodiment of the present invention Figure 1 ;

[0054] Figure 3b An optional speaker video generation schematic diagram of a video generation method is provided for an embodiment of the present invention Figure 2 ;

[0055] Figure 3c An optional speaker video generation schematic diagram 3 of a video generation method is provided for an embodiment of the present invention;

[0056] Figure 4 An optional process diagram of a video generation method is provided for an embodiment of the present invention Figure 2 ;

[0057] Figure 5 An optional flow chart of a video generation method is provided for an embodiment of the present invention;

[0058] Figure 6 An optional process diagram of a video generation method is provided for an embodiment of the present invention Figure 4 ;

[0059] Figure 7 An optional process diagram of a video generation method is provided for an embodiment of the present invention Figure 5 ;

[0060] Figure 8a A virtual object video sequence result of a video generation method is provided for an embodiment of the present invention Figure 1 ;

[0061] Figure 8b A virtual object video sequence result of a video generation method is provided for an embodiment of the present invention Figure 2 ;

[0062] Fig. 9 An optional process diagram of a video generation method is provided for an embodiment of the present invention Figure 6 ;

[0063] Fig.10 An optional model architecture diagram of a video generation method is provided for an embodiment of the present invention;

[0064] Fig.11 A schematic diagram of a video generation device provided by an embodiment of the present invention Figure 1 ;

[0065] Fig.12 A schematic diagram of a video generation device provided by an embodiment of the present invention Figure 2 . DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0067] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Figure 1 An optional terminal operation diagram of a video generation method is provided for an embodiment of the present invention, such as Figure 1 As shown, the terminal includes an encoder, a virtual prediction network and a virtual human interface (not shown in the figure). The terminal can extract features of the audio and video of the real object through the speaker encoder (equivalent to the encoder), and input the extracted features, attitude (equivalent to the first feature) and the reference person image (equivalent to the preset standard image) into the listener decoder (equivalent to the virtual prediction network) for prediction, and generate the listener's head movement and expression changes arranged in a timeline, thereby obtaining a virtual object video sequence. Figure 1 There are also images and attitudes of referees, but no corresponding description appears in the manual.

[0068] In some embodiments of the present invention, Figure 2 The embodiment of the present invention provides an optional process diagram of a video generation method. Figure 1 , will combine Figure 2 The steps shown are explained.

[0069] S101, collecting audio and video sequences of real objects.

[0070] In some embodiments of the present invention, according to the concepts of social psychology and anthropology, "listening" is also a functional behavior during communication. Among them, listening behavior styles can be divided into four categories, namely non-listeners, marginal listeners, evaluative listeners and active listeners. Among them, active responsive listening is the most effective one, and it also plays a key role in communication. It requires the listener to fully focus on what a person says, listen carefully, and show some visual reactions to the speaker. These reactions can give feedback to the speaker about whether the listener is interested, understands, and agrees with the content of the speech, so as to adjust the rhythm and progress of the conversation and promote the smooth progress of communication.

[0071] For active listening, listeners will have common visual patterns when expressing their opinions. For example, symmetrical and circular motions are used to indicate "yes", "no" or similar signals; small linear motions are coordinated with emphasized syllables in the other party's speech; and larger linear motions often appear during pauses in the other party's speech. In face-to-face human interactions, even the time when the listener blinks can be regarded as a communication signal. Therefore, based on some audio and video sequences, it is of great significance to generate video sequences under which virtual objects hear audio and video sequences.

[0072] Exemplarily, given a reference image of a speaker and a time-varying signal, a simulated segment of the speaker that can match the time-varying signal is generated. Figure 3a , Figure 3b ,and Figure 3c The embodiment of the present invention provides a video generation method for an optional speaker video generation schematic diagram. Figure 1 , Speaker video generation diagram Figure 2 and the speaker video generation diagram 3, such as Figure 3a As shown in , the speaker video generation task includes the generation of the speaker's body posture; Figure 3b As shown, the speaker video generation task includes the generation of the speaker's lip movements; Figure 3c As shown in , the speaker video generation task includes the motion generation of the speaker’s head (including face). Figure 3a In the figure, the speaker's body posture is generated by processing a time-varying signal input in the dashed box through the body posture generation model to obtain the body posture shown in the dotted box. Figure 3b In the example, the speaker's lip movement is generated by processing a time-varying signal and a general reference image input in the dashed box through the lip movement generation model, and outputting a general lip movement image frame shown in the dotted box. Figure 3c In the figure, the motion generation of the speaker's head (including face) is mainly achieved by processing a time-varying signal and a reference image speaker and emotion input in the dotted box through a head motion generation model, and rendering the processed result through a head rendering model to output a motion image frame of the speaker's head (including face) shown in the dotted box.

[0073] In some embodiments of the present invention, the terminal may collect audio sequences and video sequences of real objects through a collection device.

[0074] Exemplarily, the collection device may be a device capable of collecting video and audio, such as a camera, but the present invention is not limited thereto; the real object may be a person speaking in a scene; the audio sequence and the video sequence may be obtained in the process of tourists asking questions of a self-service consultation device (the carrier of the virtual object) in a tourist attraction scene.

[0075] In some embodiments of the present invention, the present invention is applied to situations requiring human-computer interaction, for example, intelligent consulting equipment in a shopping mall can make corresponding videos based on the videos shown by the shoppers to guide the shoppers.

[0076] S102: Extract features from the audio and video sequences to determine anthropomorphic features.

[0077] In some embodiments of the present invention, the audio-video sequence includes an audio sequence of a real object and a video sequence of a real object.

[0078] In some embodiments of the present invention, feature extraction can be achieved through a neural network model. The process of feature extraction is: input each frame of the video sequence into the neural network, perform feature extraction through multiple convolutional layers and pooling layers, and obtain multiple video features. Anthropomorphic features refer to features obtained by characterizing body movements when people speak in real scenes. Anthropomorphic features are features that include audio features and video features; anthropomorphic features are audio features and video features obtained after feature extraction of audio and video sequences. For example, in a video, the speaker raises his hand while speaking, and the anthropomorphic features can be the features corresponding to the hand-raising action.

[0079] In some embodiments of the present invention, the terminal can perform feature extraction on a video sequence of a real object through an encoder to obtain multiple video features; perform feature extraction on an audio sequence of a real object through an encoder to obtain multiple audio features; the audio features include loudness, zero-crossing rate and inverse spectrum coefficients; and perform feature conversion through a feature fusion function through multiple video features and multiple audio features to determine anthropomorphic features.

[0080] In some embodiments of the present invention, Figure 4 The embodiment of the present invention provides an optional process diagram of a video generation method. Figure 2 ,like Figure 4 As shown, S102 can be implemented through S1021-S1023 as follows:

[0081] S1021. Extract features from a video sequence of a real object through an encoder to obtain multiple video features.

[0082] In some embodiments of the present invention, video features are features obtained by recording head rotation and facial expressions during communication. Video features are features obtained by extracting features from video sequences; video features include posture and expression.

[0083] In some embodiments of the present invention, the terminal can extract features from each video frame of a video sequence of a real object through a face reconstruction model to obtain multiple video frame features; and use all corresponding second posture expression features in the video sequence of the real object as video features.

[0084] In some embodiments of the present invention, Figure 5 FIG. 3 is an optional flow chart of a video generation method provided by an embodiment of the present invention. Figure 5 As shown, S1021 can be implemented by S10211 and S10212 as follows:

[0085] S10211. Perform feature extraction on each video frame of a video sequence of a real object using a face reconstruction model to obtain multiple video frame features.

[0086] In some embodiments of the present invention, the video frame features are obtained by recording the head rotation, facial expressions and various factors of shooting of the characters in each video frame in the video sequence. The video frame features include a second identity feature and a second posture expression feature. The second identity feature is the result of recording the environmental factors of the video sequence shooting and the identity information of the photographed object. The second identity feature includes the identity identification, material and lighting of the real object. The second posture expression feature is the result of recording the real object while speaking, accompanied by head rotation and facial expression changes. The second posture expression feature includes the head posture and facial expression of the real object. The face reconstruction model generally selects a 3D face reconstruction model, but the present invention is not limited to this.

[0087] In some embodiments of the present invention, the terminal can extract features of the person and background in each video frame of the video sequence of the real object through the face reconstruction model, and obtain the person's identity, the material of the video frame, the lighting when shooting, the person's head posture and the expression on the person's face in each video frame. The person's identity, the material of the video frame, and the lighting when shooting are used as the second identity feature, and the person's head posture and the expression on the person's face are used as the second posture expression feature; all the features are used as a video frame feature.

[0088] For example, for a video sequence The 3D face reconstruction model is used to extract the 3D face parameters. For each frame, a five-tuple parameter of the form {α, β, δ, p, γ} can be extracted, which respectively represent the identity of the current face, the facial expression of the person, the material of the video frame, the posture of the person's face, and the lighting when shooting. All these features combined together are a video frame feature.

[0089] S10212. Use all corresponding second posture expression features in the video sequence of the real object as video features.

[0090] In some embodiments of the present invention, the terminal determines all corresponding second posture expression features in the video sequence of the real object as video features corresponding to the video sequence.

[0091] For example, the {α, β, δ, p, γ} parameters are divided into two categories: one is relatively fixed and closely coupled to the identity information. (equivalent to the second identity feature), and the other is relatively dynamic and independent of the identity information m = (β, p) (equivalent to the second posture expression feature). Since only the expression changes and head movements of the real object are considered, and the identity information is ignored, for the input video sequence of the real object, their video features can be extracted as

[0092] It can be understood that in some embodiments of the present invention, the terminal can extract features from each frame of the video sequence of the real object through the face reconstruction model to obtain multiple video frame features; all the corresponding second posture expression features in the video sequence of the real object are used as video features, and the inherent identity features (such as the identity of the person, the material of the video frame, and the lighting during shooting) are removed, thereby improving the effectiveness of feature extraction and providing data support for the subsequent generation of virtual object video sequences.

[0093] S1022. Perform feature extraction on the audio sequence of the real object through an encoder to obtain multiple audio features.

[0094] In some embodiments of the present invention, audio features are some features that accompany a speaker's speech during human communication. Audio features are features obtained after feature extraction of an audio sequence; audio features include loudness, zero crossing rate, and cepstral coefficients corresponding to a real object speaking.

[0095] In some embodiments of the present invention, the terminal can perform feature extraction on the spectrum of the audio sequence of the real object through an encoder to obtain the loudness, zero-crossing rate and cepstral coefficients of the real object at each moment of speaking; the loudness, zero-crossing rate and cepstral coefficients are all regarded as audio features, thereby obtaining multiple audio features.

[0096] For example, for an audio sequence of a real object The energy features, time domain features and frequency domain features are extracted as its features; the loudness, zero crossing rate (ZCR), frequency inverse coefficient (MFCC) features and the corresponding MFCC Delta and Delta-Delta features are extracted as the audio features of the audio segment. i .from The audio features extracted from

[0097] It can be understood that the terminal can perform feature extraction based on the audio and video sequence generated when the real object speaks, and obtain corresponding multiple audio features and multiple video features; it can quickly extract effective features of audio and video, thereby improving the speed of feature extraction and the effectiveness of features.

[0098] S1023. Based on the multiple video features and the multiple audio features, feature conversion is performed through a feature fusion function to determine anthropomorphic features.

[0099] In some embodiments of the present invention, the anthropomorphic features are multi-frame anthropomorphic features corresponding to an audio and video sequence; the anthropomorphic features include video features and audio features; and the feature fusion function can convert nonlinear features into linear features.

[0100] In some embodiments of the present invention, the terminal performs nonlinear feature conversion on multiple audio features and multiple video features through a multimodal feature fusion function in an encoder to obtain feature representations of real objects, that is, to determine anthropomorphic features.

[0101] It can be understood that the terminal can improve the speed of processing audio and video sequences by extracting features from the audio and video sequences; quickly obtain multiple video features and multiple audio features, and convert the multiple video features and multiple audio features into a first feature through a feature fusion function, which can convert the expression of the feature to facilitate the terminal to process it, thereby improving the feasibility of video feature and audio feature processing.

[0102] S103: using a virtual prediction network, preset standard features, and the first feature to predict anthropomorphic features, and generate a video sequence of a virtual object.

[0103] In some embodiments of the present invention, the video sequence of the virtual object is a video sequence of the virtual object's corresponding reaction generated based on the audio and video sequence of the real object; the preset standard feature is the feature corresponding to the reference object; the preset standard feature includes the first posture expression feature and the first identity feature corresponding to the reference object; the first posture expression feature is the head posture and facial expression of the reference object; the first identity feature is the identity-related information corresponding to the reference object. The first feature is the emotional attitude feature of a person when speaking. The first feature represents different attitudes, and the first feature can be a positive attitude, a negative attitude, or a normal attitude.

[0104] In some embodiments of the present invention, the terminal can obtain preset standard features; based on the first posture expression feature, the anthropomorphic feature and the first feature, predict and decode through a virtual prediction network to determine the multi-frame posture expression features of the virtual object; generate a video sequence of the virtual object through the multi-frame posture expression features and the first identity feature of the virtual object.

[0105] For example, given an input video sequence of a real object with a duration from 1 to t and its corresponding audio sequence (Equivalent to the audio and video sequence of the real object). The virtual object generation task aims to generate a virtual object video sequence at time step t+1 which can be expressed by the following formula (1).

[0106]

[0107] in, is the listener's reference image (equivalent to the standard image), e is the listener's attitude, and the total number of generated listener videos is It can be expressed as

[0108] In some embodiments of the present invention, Figure 6 The embodiment of the present invention provides an optional process diagram of a video generation method. Figure 4 ,like Figure 6 As shown, S103 can be implemented through S1031-S1033 as follows:

[0109] S1031. Obtain preset standard features.

[0110] In some embodiments of the present invention, the terminal may obtain a standard image; perform feature extraction on the standard image through a face reconstruction model to obtain a first posture expression feature and a first identity feature, and use the first posture expression feature and the first identity feature as preset standard features.

[0111] In some embodiments of the present invention, S1031 may be implemented by S10311 and S10312 as follows:

[0112] S10311. Obtain a standard image.

[0113] In some embodiments of the present invention, the standard image is an image of a reference object; the standard image is any face image randomly obtained from an image library that is different from a real object.

[0114] S10312. Extract features of the standard image using a face reconstruction model to obtain preset standard features.

[0115] In some embodiments of the present invention, the terminal can extract features from the standard image through the face reconstruction model to obtain the identity of the person in the standard image, the material of the video frame, the lighting during shooting, the head posture of the person, and the facial expression of the person. The identity of the person in the standard image, the material of the video frame, and the lighting during shooting are used as the first identity feature; the head posture of the person in the standard image and the facial expression of the person are used as the first posture expression feature; the first identity feature and the first posture expression feature are used as the preset standard features.

[0116] It can be understood that the terminal can obtain a standard image; by performing feature extraction on the standard image through a face reconstruction model, the first posture expression feature can be quickly and accurately obtained, thereby improving the accuracy and efficiency of standard image processing; the first posture expression feature can be used to generate a video sequence of a virtual object, thereby ensuring the accuracy of generating a video sequence of a virtual object.

[0117] S1032: Based on the first posture expression feature, the anthropomorphic feature and the first feature, prediction and decoding are performed through a virtual prediction network to determine multi-frame posture expression features of the virtual object.

[0118] In some embodiments of the present invention, the terminal can use the first frame of anthropomorphic features, the first posture expression features and the first features in multiple frames of anthropomorphic features to perform prediction through a first processing module to obtain a next predicted video frame; decode the next predicted video frame through a second processing module to determine the next posture expression features of the virtual object corresponding to the next predicted video frame; continue to predict and decode through the next posture expression features and the next frame of anthropomorphic features in multiple frames of anthropomorphic features until the last posture expression features of the virtual object corresponding to the last predicted video frame are obtained, thereby obtaining multiple frames of posture expression features of the virtual object.

[0119] In some embodiments of the present invention, S1032 may be implemented by S10321, S10322, and S10323 as follows:

[0120] S10321. Based on the first frame of anthropomorphic features, the first posture and expression features, and the first feature in the multiple frames of anthropomorphic features, prediction is performed through the first processing module to obtain the next predicted video frame.

[0121] In some embodiments of the present invention, the first feature is one of a positive attitude, a negative attitude and a normal attitude; the first processing module is an encoding module in a virtual prediction network; and the function is to implement encoding in prediction processing.

[0122] In some embodiments of the present invention, the terminal can select any one of a positive attitude, a negative attitude and a normal attitude as the first feature, and the normal attitude is selected here; under the normal attitude, the first frame of anthropomorphic features and the first posture expression features in multiple frames of anthropomorphic features are input into the first processing module in the virtual prediction network for prediction to obtain the next video prediction frame.

[0123] S10322. Decode the next predicted video frame through the second processing module to determine the next posture expression feature of the virtual object corresponding to the next predicted video frame.

[0124] In some embodiments of the present invention, the second processing module is a decoding module in a virtual prediction network, and its function is to implement decoding in prediction processing.

[0125] In some embodiments of the present invention, the terminal can decode the next predicted video frame through the second processing module in the virtual prediction network to obtain the next posture expression feature of the virtual object corresponding to the next predicted video frame; the next posture expression feature includes the next posture feature and the next expression feature.

[0126] S10323. Based on the next posture expression feature and the next frame anthropomorphic feature in the multi-frame anthropomorphic features, continue to predict and decode until the last posture expression feature of the virtual object corresponding to the last predicted video frame is obtained, thereby obtaining the multi-frame posture expression feature of the virtual object.

[0127] In some embodiments of the present invention, the first gesture expression feature is used as the first frame of multiple frames of gesture expression features.

[0128] In some embodiments of the present invention, the terminal inputs the next posture expression feature and the second frame of anthropomorphic features in multiple frames of anthropomorphic features into the first processing module in the virtual prediction network for prediction to obtain the next predicted video frame; decodes through the second processing module in the virtual prediction network to determine the next posture expression feature of the virtual object corresponding to the next predicted video frame; continues to predict and decode through the virtual prediction network until the last posture expression feature of the virtual object corresponding to the last predicted video frame is obtained, thereby determining the multi-frame posture expression features of the virtual object.

[0129] It can be understood that the terminal can make predictions through the first frame anthropomorphic features, the first posture and expression features and the first feature. The first feature can improve the diversity of generated predicted video frames, and the first frame anthropomorphic features and the first posture and expression features can improve the accuracy of generated predicted video frames; based on the first predicted video frame, the next predicted video frame is generated, and the next video frame is predicted by the real-time updated predicted video frame, so as to generate continuous video frames, improve the continuity between video frames, and ensure the integrity of the video sequence; by decoding multiple predicted video frames to obtain multi-frame posture and expression features, the head and facial reactions of the virtual object when hearing the audio and video of the real object can be fully reflected, thereby improving the accuracy of generating the video sequence of the virtual object.

[0130] S1033: Generate a video sequence of the virtual object based on the multi-frame posture expression features and the first identity features of the virtual object.

[0131] In some embodiments of the present invention, the virtual object is a simulated object with simple communication functions, and generally exists in an interactive device.

[0132] In some embodiments of the present invention, the terminal can obtain multiple second features by fusing each frame of posture expression features of multiple frames of posture expression features of the virtual object with the first identity feature; and generate a video sequence of the virtual object corresponding to the virtual object through a renderer using the multiple second features.

[0133] It can be understood that the terminal can obtain preset standard features; determine the multi-frame posture expression features of the virtual object by predicting and decoding the first posture expression features, the anthropomorphic features and the first features among the preset standard features through the virtual prediction network, and improve the accuracy of the multi-frame posture expression features; generate a video sequence of the virtual object based on the multi-frame posture expression features and the first identity feature of the virtual object; and improve the accuracy and vividness of the video sequence of the virtual object.

[0134] In some embodiments of the present invention, S1033 may be implemented by S10331 and S10332 as follows:

[0135] S10331. Each frame of the posture expression feature of the multi-frame posture expression feature of the virtual object is respectively merged with the first identity feature to obtain a plurality of second features.

[0136] In some embodiments of the present invention, the second feature is a fusion feature formed after the posture expression feature is added to the identity feature. The second feature is the fusion result of the identity feature and the posture expression feature.

[0137] In some embodiments of the present invention, the terminal may fuse each frame of the posture expression feature of the multi-frame posture expression feature of the virtual object with the first identity feature to obtain a plurality of second features corresponding to the multi-frame posture expression feature.

[0138] S10332. Pass the multiple second features through a renderer to generate a video sequence of the virtual object corresponding to the virtual object.

[0139] In some embodiments of the present invention, the first identity feature includes a first identity identifier, a first material and first lighting information; the first identity identifier corresponds to the identity identifier of the person in the standard image, the first material corresponds to the material of the video frame and the first lighting information corresponds to the lighting during shooting.

[0140] In some embodiments of the present invention, the terminal may generate a video sequence of a virtual object using a renderer by using the plurality of second features.

[0141] Exemplarily, the task of generating a video sequence of a virtual object can be expressed by formula (2) and formula (3).

[0142]

[0143]

[0144] in, A posture feature of a virtual object predicted by a virtual prediction network; It is a feature of the virtual object associated with its identity, which is used together with the predicted pose features to generate a video sequence of the virtual object through the renderer

[0145] It can be understood that the terminal can obtain multiple second features by fusing each frame of the virtual object's multi-frame posture expression features with the first identity feature; making the second feature have identity attributes can improve the distinctiveness of the second feature, and using the multiple second features through a renderer to generate a video sequence of the virtual object. The video sequence of the virtual object generated in a targeted manner is more vivid and accurate.

[0146] S104: Present a video sequence of a virtual object.

[0147] In some embodiments of the present invention, the terminal may present a video sequence of a virtual object on a virtual human interface.

[0148] It can be understood that the terminal collects the audio and video sequence of the real object; extracts features from the audio and video sequence to determine the anthropomorphic features; uses the virtual prediction network and the preset standard features, as well as the first feature, to predict the anthropomorphic features and generate the video sequence of the virtual object; the preset standard features are the features corresponding to the reference object; the first feature represents different attitudes; the video sequence of the virtual object is a video sequence of the corresponding reaction of the virtual object generated based on the audio and video sequence of the real object; and presents the video sequence of the virtual object. The video sequence of the virtual object is generated based on the audio and video sequence of the real object, and the presented video sequence of the virtual object is more vivid and accurate.

[0149] In some embodiments of the present invention, Figure 7 The embodiment of the present invention provides an optional process diagram of a video generation method. Figure 5 ,like Figure 7 As shown, before executing S103, S105-S1010 are also executed as follows:

[0150] S105, collecting audio and video sequence samples of the real talking object and the corresponding face image of the real listening object.

[0151] In some embodiments of the present invention, the audio and video sequence samples are formed by recording the speech of the real confidant and the corresponding movements and expressions during the speech.

[0152] In some embodiments of the present invention, the terminal can collect audio and video sequence samples of the real confidant and the corresponding face image of the real listening object through a collection device. The collected face image of the real listening object is collected under different first features, and the first features include positive attitude, negative attitude and normal attitude.

[0153] S106: extract features from the audio and video sequence samples through an initial encoder to determine features of the anthropomorphic samples.

[0154] In some embodiments of the present invention, the anthropomorphic sample features are features obtained by characterizing body movements when people speak in scenes corresponding to audio and video sequence samples. The anthropomorphic sample features are obtained by extracting features from audio and video sequence samples, and the anthropomorphic sample features include video sample features and audio sample features.

[0155] In some embodiments of the present invention, the terminal may perform feature extraction on the audio and video sequence samples through an initial encoder to obtain anthropomorphic sample features, wherein the anthropomorphic sample features include multi-frame anthropomorphic sample features.

[0156] S107, generating predicted facial features under sample audio and video sequences of training real objects through the initial virtual prediction network and anthropomorphic sample features.

[0157] In some embodiments of the present invention, the predicted facial features include predicted posture features and predicted expression features.

[0158] In some embodiments of the present invention, the terminal can generate predicted posture features and predicted expression features under audio and video sequence samples of training real objects through an initial virtual prediction network and anthropomorphic sample features; the predicted posture features include multi-frame predicted posture features; the predicted expression features include multi-frame predicted expression features; the predicted posture features and predicted expression features are used as predicted facial features; the predicted facial features include multi-frame predicted facial features.

[0159] S108, extracting features from the real face image of the listening object through a face reconstruction model to determine real face features.

[0160] In some embodiments of the present invention, the real face features include real posture features and real expression features;

[0161] In some embodiments of the present invention, the terminal can extract features from the face image of the real listening object through the face reconstruction model to obtain real posture features and real expression features, and use the real posture features and real expression features as real face features. Among them, the number of face images of the real listening object is consistent with the number of multi-frame anthropomorphic features obtained through audio and video sequence samples, the real posture features include multi-frame real posture features; the real expression features include multi-frame real expression features; the real face features also include multi-frame real face features, and the number of real face features is consistent with the number of predicted face features.

[0162] S109, continuously optimizing the initial encoder through the first loss function and the anthropomorphic sample features until the first loss function value meets the first preset threshold, and determining the encoder.

[0163] In some embodiments of the present invention, the terminal can continuously optimize the initial encoder through a first loss function and anthropomorphic sample features. If the first loss function value is greater than or equal to a first preset threshold, the encoder can be regarded as having been trained and the encoder is determined; if the first loss function value is less than the first preset threshold, the encoder continues to be trained until the first loss function value is greater than or equal to the first preset threshold and the encoder is determined.

[0164] S1010. Based on the real face features and the predicted face features, the initial virtual prediction network is continuously optimized through the second loss function and the third loss function until the second loss function value and the third loss function value both meet the second preset threshold value, and the virtual prediction network is determined.

[0165] In some embodiments of the present invention, the terminal can use real facial features and predicted facial features to continuously optimize the initial virtual prediction network through the second loss function and the third loss function. If the sum of the second loss function value and the third loss function value is greater than or equal to the second preset threshold, the virtual prediction network is determined; if the sum of the second loss function value and the third loss function value is less than the second preset threshold, the virtual prediction network continues to be trained until the sum of the second loss function value and the third loss function value is greater than or equal to the second preset threshold, and the virtual prediction network is determined.

[0166] For example, Figure 8a and Figure 8b The video sequence results of the virtual objects of a video generation method provided by the embodiment of the present invention are respectively Figure 1 The embodiment of the present invention provides a video sequence result of a virtual object of a video generation method Figure 2 ,like Figure 8a and Figure 8b As shown, the horizontal axis represents continuous video frames, including 0-32 frames; Figure 8a The results of generating video sequences of virtual objects are shown in the test results in the domain (the training data contains the speaker or listener in the test data set). Figure 8b The results of the video sequence of generating virtual objects are shown in the test results of out-of-domain (the face data of the speaker and the listener have never appeared in the training set, mainly to test the generalization ability of the model on unseen faces). Figure 8a The true listener attitude is positive. Figure 8b A true listener with a natural attitude. Figure 8a Lines 4, 5, and 6 and Figure 8b The 4th, 5th, and 6th rows in the figure show the video sequence results of virtual objects generated under three different attitudes. Figure 8a and Figure 8b The generated listener in is the virtual object. Among them, frame 0 is the reference frame. The triangles in the figure indicate significant changes: the upper right Gaze changes with eye movement; lower left Head movement; lower right Representative frame for each row; top left Frames with significant differences under different attitudes (column direction).

[0167] It can be seen that the virtual prediction network can capture the patterns of general listeners (equivalent to virtual objects) (such as eye, mouth and head movements, etc.), which may be different from real listeners but still meaningful. In addition, the virtual prediction network can present the visual patterns of virtual objects under different attitudes. Figure 8aFrom the results shown, we can see that although the neutral attitude also smiles (frames 2-8), it lasts shorter than the positive attitude (frames 2-16). For the negative attitude people, the virtual objects do not pay attention to the conversation, and they look to the lower side of the screen at frames 10, 16, 22, and 30.

[0168] Finally, in Figure 8b The generated results are relatively good on out-of-domain data. The virtual object with a positive attitude smiles in frames 6-14, while the listener with a negative attitude frowns and shows negative mouth shapes throughout the process. The virtual object with a negative attitude has small changes in movements and wandering eyes, while the neutral virtual object maintains a relatively calm expression with regular head movements.

[0169] To evaluate the generated virtual object video sequence results, 10 volunteers performed the following two tests:

[0170] Best match test. Given an attitude, a real object audio sequence, a real object video sequence, a real listener video, and a generated virtual object video sequence, volunteers were asked to select the listener that was most perceptually appropriate and best matched the given attitude.

[0171] Attitude classification test. Given a generated video sequence of a virtual object, volunteers need to determine its emotion (positive, negative, neutral). It should be noted that neutral is equivalent to a general attitude.

[0172] Both tests were performed in a double-blind format and the results are shown in Tables 1 and 2.

[0173] Table 1 Best match test results

[0174]

[0175] The mean and variance of the two types of “best listeners” are statistically shown in Table 1. In the in-domain data, volunteers voted that nearly 20% of the generated virtual objects looked more reasonable than real listeners, which verified that the model can generate responsive listeners that are consistent with human subjective perception. Moreover, the results generated in the out-of-domain data were liked by more volunteers.

[0176] Table 2 Expression classification test results

[0177]

[0178] As can be seen from Table 2, for each attitude, the mean and variance of the classification accuracy of all volunteers are calculated, and it can be found that the model can generate videos with the specified attitude to a certain extent.

[0179] It can be understood that the terminal can collect audio and video sequence samples of the real speaking object and the corresponding facial images of the real listening object, and the facial images of the real listening object are collected under different second features, which can ensure the diversity of training samples; through the audio and video sequence samples and the corresponding facial images of the real listening object, the initial encoder and the initial virtual prediction network are optimized and trained to determine the encoder and the virtual prediction network, which can improve the accuracy of the output results of the encoder and the virtual prediction network.

[0180] In some embodiments of the present invention, Fig. 9 The embodiment of the present invention provides an optional process diagram of a video generation method. Figure 6 ,like Fig. 9 As shown, before executing S1010, S1011-S1013 are also executed as follows:

[0181] S1011. Determine a second loss function based on real facial features and predicted facial features.

[0182] In some embodiments of the present invention, the second loss function is used to ensure that the predicted expression and predicted posture are similar to the real expression and real posture.

[0183] In some embodiments of the present invention, the terminal may perform a difference modulo operation based on the real facial features and the predicted facial features to determine the second loss function.

[0184] Exemplarily, the second loss function can be obtained by the following formula (4).

[0185]

[0186] in, represents the second loss function; Represents the predicted expression features in the predicted face features; Represents the real expression features in real face features; Represents the predicted posture features in the predicted face features; Represents the real posture features in the real face features. For the optimization process, there are real face features of real listeners. Due to the lack of supervision signals in frame T+1, the generated results of the last frame of the initial virtual prediction network are discarded. That is, the posture and expression features of the virtual object corresponding to the last predicted video frame (equivalent to the last frame of predicted facial features).

[0187] S1012. Determine a third loss function based on the change function corresponding to the real facial features and the change function corresponding to the predicted facial features.

[0188] In some embodiments of the present invention, the third loss function is used to ensure that the inter-frame continuity of the predicted facial features is similar to the real facial features.

[0189] In some embodiments of the present invention, the terminal may determine the third loss function by performing a difference modulo operation on a change function corresponding to the real facial features and a change function corresponding to the predicted facial features.

[0190] Exemplarily, the third loss function can be obtained by the following formula (5).

[0191]

[0192] in, represents the third loss function; Represents the change function corresponding to the predicted expression feature in the predicted facial feature; Represents the change function corresponding to the real expression features in the real face features; Represents the change function corresponding to the predicted posture feature in the predicted face feature; represents the change function corresponding to the real posture feature in the real face feature; μ is a function that measures the inter-frame change between the current frame and its adjacent previous frame, that is,

[0193] S1013. Continuously optimize the initial virtual prediction network through the second loss function and the third loss function until the sum of the second loss function value and the third loss function value meets the second preset threshold, and determine the virtual prediction network.

[0194] In some embodiments of the present invention, the terminal can determine the loss function of the virtual prediction network by summing the second loss function and the third loss function, and continuously optimize the initial virtual prediction network through the loss function until the loss function value (equivalent to the sum of the second loss function value and the third loss function value) meets the second preset threshold, thereby determining the virtual prediction network.

[0195] Exemplarily, the loss function of the virtual prediction network can be obtained by the following formula (6).

[0196]

[0197] in, represents the loss function of the virtual prediction network; represents the second loss function; represents the third loss function; w is the scale used to balance the two loss functions.

[0198] It can be understood that the terminal can determine the second loss function based on real facial features and predicted facial features; determine the third loss function through the change function corresponding to the real facial features and the change function corresponding to the predicted facial features, thereby improving the effectiveness of the loss function of the virtual prediction network; continuously optimize the initial virtual prediction network through the second loss function and the third loss function until the sum of the second loss function value and the third loss function value meets the second preset threshold, and determine the virtual prediction network; improve the accuracy of the virtual prediction network prediction and the prediction effect of the virtual prediction network.

[0199] The following describes an exemplary application of an embodiment of the present invention in a practical application scenario.

[0200] In some embodiments of the present invention, Fig.10 An optional model architecture diagram of a video generation method provided by an embodiment of the present invention is shown in FIG. Fig.10 As shown, the terminal obtains the speaker's video (equivalent to the video sequence of the real object), pre-process the speaker video through the speaker encoder (equivalent to the encoder) to obtain continuous video frames; extract features from each frame of the speaker video through the face reconstruction model to obtain the identity, material, lighting, expression and posture corresponding to each frame; expression and posture are used as video features Will get speaker audio (equivalent to the audio sequence of the real object), the speaker's audio is processed by the encoder to obtain audio at multiple times; the features of the audio at each moment in the multiple moments are extracted to obtain the loudness, zero-crossing rate and inverse spectrum coefficient corresponding to each moment; the loudness, zero-crossing rate and inverse spectrum coefficient are used as audio features Video Features and audio features Through the feature fusion function f am The listener decoder (equivalent to the virtual prediction network) in the terminal obtains the listener's reference image (equivalent to the standard image), extracts features from the reference image through the face reconstruction model, and obtains the identity, material, lighting, expression and posture (expression and posture are as follows Fig.10 In and ); the identity mark, material, and lighting are used as the first identity feature, and the expression and posture are used as the first posture expression feature. The first posture expression feature is input into the long short-term memory network encoder (equivalent to the first processing module), and the attitude e (equivalent to the first feature) is combined with the anthropomorphic feature to generate a multi-frame posture expression feature (equivalent to and and as well as and ), there are t+1 frames in total. The first identity feature is shared to the decoder, and the decoder (equivalent to the second processing module) fuses each frame of the multi-frame posture expression feature with the first identity feature to obtain a virtual object video sequence.

[0201] For example, for the speaker encoder, at each time step t, first extract the audio feature s t and the speaker's audio characteristics Then use a multimodal feature fusion function f am Perform nonlinear feature transformation to obtain anthropomorphic features.

[0202] To ensure that the virtual object can react with a certain attitude and produce more natural head movements and expression changes, the characteristics of the attitude e and the reference image of the listener are combined. (equivalent to the first gesture expression feature) as the first frame of the virtual object video sequence. Then, at each time step t, the speaker’s fusion feature (equivalent to anthropomorphic features) as input to generate the predicted video frame of step t+1. Finally, the listener decoder is used to decode the predicted video frame into It contains two eigenvectors, namely Expressing facial expressions, Indicates posture (rotation and translation). The terminal supports speaker input of any length. The process can be expressed as:

[0203]

[0204] in, Expressions that indicate prediction, represents the predicted pose (rotation and translation); D m Both LSTM and LSTM are components of the listener decoder, which are used to generate multi-frame gesture expression features; represents fusion features; h t Represents the predicted video frame; c t Represents the storage of predicted video frames.

[0205] It is understandable that the terminal can generate a virtual object video sequence based on the collected audio and video sequence of the real object through the speaker encoder and the listener decoder, so that the video sequence of the virtual object is more vivid and accurate.

[0206] Based on the video generation method of the above embodiment, the embodiment of the present invention also provides a video generation device, such as Fig.11 As stated, Fig.11 A schematic diagram of a video generation structure provided by an embodiment of the present invention Figure 1The device 11 includes: an acquisition unit 1101, a determination unit 1102 and a generation unit 1103; wherein,

[0207] The acquisition unit 1101 is used to collect audio and video sequences of real objects;

[0208] The determining unit 1102 is used to extract features from the audio and video sequence to determine anthropomorphic features;

[0209] The generation unit 1103 is used to predict the anthropomorphic features using a virtual prediction network, preset standard features, and a first feature to generate a virtual object video sequence; the preset standard features are features corresponding to a reference object; the first features represent different attitudes; the video sequence of the virtual object is a video sequence of a virtual object's corresponding reaction generated based on an audio and video sequence of a real object; and the video sequence of the virtual object is presented.

[0210] In some embodiments of the present invention, the acquisition unit 1101 is used to acquire a preset standard feature; the preset standard feature includes a first posture expression feature and a first identity feature;

[0211] The determining unit 1102 is used to determine the multi-frame posture expression features of the virtual object by predicting and decoding through the virtual prediction network based on the first posture expression feature, the anthropomorphic feature and the first feature;

[0212] The generating unit 1103 is configured to generate a video sequence of the virtual object based on the multi-frame posture expression features of the virtual object and the first identity feature.

[0213] In some embodiments of the present invention, the acquisition unit 1101 is used to acquire a standard image; the standard image represents an image of a reference object; and feature extraction is performed on the standard image through a face reconstruction model to obtain the preset standard feature.

[0214] In some embodiments of the present invention, the anthropomorphic features include: multiple frames of anthropomorphic features corresponding to the audio and video sequence; the virtual prediction network includes: a first processing module and a second processing module; the determination unit 1102 is used to predict through the first processing module based on the first frame of anthropomorphic features in the multiple frames of anthropomorphic features, the first posture expression feature and the first feature, to obtain a next predicted video frame; the first feature is one of a positive attitude, a negative attitude and a normal attitude; the next predicted video frame is decoded through the second processing module to determine the next posture expression feature of the virtual object corresponding to the next predicted video frame; based on the next posture expression feature and the next frame of anthropomorphic features in the multiple frames of anthropomorphic features, continue to predict and decode until the last posture expression feature of the virtual object corresponding to the last predicted video frame is obtained, thereby obtaining a multiple-frame posture expression feature of the virtual object; the first posture expression feature is used as the first frame in the multiple-frame posture expression feature.

[0215] In some embodiments of the present invention, the acquisition unit 1101 is used to fuse each frame of the multiple frames of posture expression features with the first identity feature to obtain multiple second features; the second feature represents the fusion result of the identity feature and the posture expression feature;

[0216] The generating unit 1103 is used to generate a video sequence of the virtual object corresponding to the virtual object by using the multiple second features through a renderer; the first identity feature includes a first identity identifier, a first material and first lighting information.

[0217] In some embodiments of the present invention, the audio and video sequence includes: an audio sequence of a real object and a video sequence of a real object; the acquisition unit 1101 is used to extract features from the video sequence of the real object through an encoder to obtain multiple video features; extract features from the audio sequence of the real object through an encoder to obtain multiple audio features; the audio features include loudness, zero crossing rate and cepstral coefficients;

[0218] The determination unit 1102 is used to perform feature conversion through a feature fusion function based on the multiple video features and the multiple audio features to determine the anthropomorphic features; the anthropomorphic features are multiple frames of anthropomorphic features corresponding to the audio and video sequence; the anthropomorphic features include video features and audio features.

[0219] In some embodiments of the present invention, the acquisition unit 1101 is used to extract features from each video frame of the video sequence of the real object through a face reconstruction model to obtain the multiple video frame features; the video frame features include a second identity feature and a second posture expression feature;

[0220] The determining unit 1102 is configured to use all the second posture expression features corresponding to the video sequence of the real object as the video features.

[0221] In some embodiments of the present invention, the acquisition unit 1101 is used to collect audio and video sequence samples of a real talking object and a face image of a real listening object corresponding thereto;

[0222] The determining unit 1102 is used to extract features from the audio and video sequence samples through an initial encoder to determine anthropomorphic sample features;

[0223] The generating unit 1103 is used to generate predicted facial features under the sample audio and video sequence of the training real object through the initial virtual prediction network and the anthropomorphic sample features; the predicted facial features include predicted posture features and predicted expression features;

[0224] The determination unit 1102 is used to perform feature extraction through a face reconstruction model based on the face image of the real listening object to determine real face features, wherein the real face features include real posture features and real expression features; continuously optimize the initial encoder through a first loss function and the anthropomorphic sample features until the first loss function value meets a first preset threshold, and then determine the encoder; based on the real face features and the predicted face features, continuously optimize the initial virtual prediction network through a second loss function and a third loss function until the sum of the second loss function value and the third loss function value meets a second preset threshold, and then determine the virtual prediction network.

[0225] In some embodiments of the present invention, the determination unit 1102 is used to determine a second loss function based on the real facial features and the predicted facial features; the second loss function is used to ensure that the predicted expression and the predicted posture are similar to the real expression and the real posture; based on the change function corresponding to the real facial features and the change function corresponding to the predicted facial features, a third loss function is determined; the third loss function is used to ensure that the inter-frame continuity of the predicted facial features is similar to the real facial features; through the second loss function and the third loss function, the initial virtual prediction network is continuously optimized until the sum of the second loss function value and the third loss function value meets the second preset threshold, and the virtual prediction network is determined.

[0226] It should be noted that, when performing video generation, only the division of the above-mentioned program modules is used as an example for illustration. In actual applications, the above-mentioned processing can be assigned to different program modules as needed, that is, the internal structure of the device is divided into different program modules to complete all or part of the above-described processing. In addition, the video generation device and the video generation method embodiment provided in the above embodiment belong to the same concept, and their specific implementation process and beneficial effects are detailed in the method embodiment, which will not be repeated here. For technical details not disclosed in the embodiment of this device, please refer to the description of the method embodiment of the present invention for understanding.

[0227] Based on the video generation method of the above embodiment, the embodiment of the present invention also provides a video generation device, such as Fig.12 As shown, Fig.12 A schematic diagram of a video generation device provided by an embodiment of the present invention Figure 2 The device 12 includes: a processor 1201 and a memory 1202; the memory 1202 stores one or more programs executable by the processor, and when one or more programs are executed, any one of the video generation methods in the above-mentioned embodiments is executed by the processor 1201.

[0228] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0229] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0230] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0231] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0232] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention.

Claims

1. A video generation method, characterized in that: include: Collect audio and video sequences of real objects; Extracting features from the audio and video sequence to determine anthropomorphic features, wherein the anthropomorphic features include audio features and video features, and the audio features include loudness, zero-crossing rate, and cepstrum coefficients; The anthropomorphic feature is predicted by using a virtual prediction network and a preset standard feature under a first feature to generate a video sequence of a virtual object; the preset standard feature is a feature corresponding to a reference object; the first feature represents different attitudes; the video sequence of the virtual object is a video sequence of a virtual object's corresponding reaction generated based on an audio and video sequence of a real object; Video sequences of the virtual object in different attitudes are presented.

2. The method according to claim 1, characterized in that The method of using a virtual prediction network, a preset standard feature, and a first feature to predict the anthropomorphic feature to generate a video sequence of a virtual object includes: Acquire preset standard features; the preset standard features include a first posture expression feature and a first identity feature; Based on the first posture expression feature, the anthropomorphic feature and the first feature, predicting and decoding through the virtual prediction network to determine the multi-frame posture expression features of the virtual object; A video sequence of the virtual object is generated based on the multi-frame posture expression features of the virtual object and the first identity feature.

3. The method according to claim 2, characterized in that The obtaining of preset standard features includes: Acquire a standard image; the standard image represents an image of a reference object; The standard image is subjected to feature extraction through a face reconstruction model to obtain the preset standard features.

4. The method according to claim 2, characterized in that: The anthropomorphic features include: multiple frames of anthropomorphic features corresponding to the audio and video sequence; the virtual prediction network includes: a first processing module and a second processing module; The method of predicting and decoding the multi-frame posture expression features of the virtual object through the virtual prediction network based on the first posture expression feature, the anthropomorphic feature and the first feature, comprises: Based on the first frame of anthropomorphic features in the multiple frames of anthropomorphic features, the first posture expression features and the first feature, the first processing module performs prediction to obtain a next predicted video frame; wherein the first feature is one of a positive attitude, a negative attitude and a normal attitude; Decoding the next predicted video frame through the second processing module to determine a next posture expression feature of the virtual object corresponding to the next predicted video frame; Based on the next posture expression feature and the next frame of anthropomorphic features in the multi-frame anthropomorphic features, continue to predict and decode until the last posture expression feature of the virtual object corresponding to the last predicted video frame is obtained, thereby obtaining the multi-frame posture expression feature of the virtual object; the first posture expression feature is used as the first frame in the multi-frame posture expression features.

5. The method according to claim 2, characterized in that: The step of generating a video sequence of the virtual object based on the multi-frame posture expression features of the virtual object and the first identity feature comprises: The gesture expression feature of each frame in the plurality of frames of gesture expression features is respectively fused with the first identity feature to obtain a plurality of second features; the second features represent the fusion result of the identity feature and the gesture expression feature; The plurality of second features are passed through a renderer to generate a video sequence of the virtual object corresponding to the virtual object; the first identity feature includes a first identity identifier, a first material and first lighting information.

6. The method according to any one of claims 1 to 5, characterized in that: The audio and video sequence includes: an audio sequence of a real object and a video sequence of a real object; The step of extracting features from the audio and video sequence to determine anthropomorphic features includes: Extracting features from the video sequence of the real object by an encoder to obtain multiple video features; Extracting features from the audio sequence of the real object by an encoder to obtain multiple audio features; Based on the multiple video features and the multiple audio features, feature conversion is performed through a feature fusion function to determine the anthropomorphic feature; the anthropomorphic feature is a multi-frame anthropomorphic feature corresponding to the audio and video sequence.

7. The method according to claim 6, characterized in that The encoder extracts features from the video sequence of the real object to obtain multiple video features, including: Extracting features from each video frame of the video sequence of the real object using a face reconstruction model to obtain a plurality of video frame features; the video frame features include a second identity feature and a second posture and expression feature; All the second posture expression features corresponding to the video sequence of the real object are used as the video features.

8. The method according to any one of claims 1 to 5, characterized in that: Before predicting the anthropomorphic feature using the virtual prediction network, the preset standard feature, and the first feature to generate a video sequence of the virtual object, the method further includes: Collect audio and video sequence samples of the real talking object and the corresponding face images of the real listening object; Extracting features from the audio and video sequence samples through an initial encoder to determine features of anthropomorphic samples; Generate predicted facial features under audio and video sequence samples of real training objects through the initial virtual prediction network and the anthropomorphic sample features; the predicted facial features include predicted posture features and predicted expression features; According to the face image of the real listening object, feature extraction is performed through a face reconstruction model to determine real face features, wherein the real face features include real posture features and real expression features; Continuously optimizing the initial encoder by using a first loss function and the anthropomorphic sample features until the first loss function value meets a first preset threshold, thereby determining the encoder; Based on the real face features and the predicted face features, the initial virtual prediction network is continuously optimized through the second loss function and the third loss function until the second loss function value and the third loss function value both meet the second preset threshold value, and the virtual prediction network is determined.

9. The method according to claim 8, characterized in that The method of continuously optimizing the initial virtual prediction network based on the real face features and the predicted face features through the second loss function and the third loss function until the second loss function value and the third loss function value meet the second preset threshold value, and determining the virtual prediction network includes: Based on the real face features and the predicted face features, determining a second loss function; the second loss function is used to ensure that the predicted expression and the predicted posture are similar to the real expression and the real posture; Determining a third loss function based on the change function corresponding to the real face feature and the change function corresponding to the predicted face feature; the third loss function is used to ensure that the inter-frame continuity of the predicted face feature is similar to the real face feature; The initial virtual prediction network is continuously optimized by using the second loss function and the third loss function until the second loss function value and the third loss function value meet a second preset threshold, thereby determining the virtual prediction network.

10. A video generating device, characterized in that: It includes an acquisition unit, a determination unit and a generation unit; wherein, The acquisition unit is used to collect audio and video sequences of real objects; The determining unit is used to extract features from the audio and video sequence to determine anthropomorphic features, where the anthropomorphic features include audio features and video features, and the audio features include loudness, zero-crossing rate, and cepstrum coefficients; The generation unit is used to use a virtual prediction network and a preset standard feature to predict the anthropomorphic feature under a first feature to generate a video sequence of a virtual object; the preset standard feature is a feature corresponding to a reference object; the first feature represents different attitudes; the video sequence of the virtual object is a video sequence of the corresponding reaction of the virtual object generated based on the audio and video sequence of the real object; and the video sequence of the virtual object under different attitudes is presented.

11. A video generating device, characterized in that: include: A memory for storing executable instructions; The processor is used to implement the video generation method described in any one of claims 1 to 9 when executing the executable instructions stored in the memory.

12. A computer-readable storage medium, characterized in that: The storage medium stores executable instructions, which, when executed, are used to cause a processor to execute the video generation method according to any one of claims 1 to 9.

Images