Video generation method and device, electronic equipment and computer program product

Abstract

The invention provides a video generation method and device, electronic equipment and a computer program product, and the method can generate a virtual image video of a duplicated object based on an audio and a reference image of the duplicated object. Specifically, according to an audio and a reference image, a dynamic image of a lip area of a duplicated object is generated, and a dynamic image of the duplicated object is obtained by combining the dynamic image of the lip area, the reference image and an emotional state corresponding to the audio; or, the recarving features are extracted from the audio and the reference image, the dynamic image of the recarving object is generated according to the recarving features and the emotional state corresponding to the audio, then the virtual image video of the recarving object is generated according to the audio and the dynamic image of the recarving object, and the purpose of recarving the virtual image of the specific object is achieved.

Description

Technical Field

[0001] The present application relates to the field of virtual image technology, and in particular to a video generation method, device, electronic device and computer program product. Background Art

[0002] In modern society, virtual image technology is increasingly widely used, playing an important role in accompanying the elderly, children and other scenarios. If the virtual image of a specific object can be reproduced and interacted with the accompanying object, the accompanying effect can be effectively improved. Therefore, how to reproduce the virtual image of a specific object has become a technical problem that technicians in this field need to solve urgently. Summary of the invention

[0003] In view of this, the present application proposes a video generation method, device, electronic device and computer program product, which can replicate a virtual image of a specific object.

[0004] The technical solutions proposed in this application are as follows:

[0005] In a first aspect, an embodiment of the present application provides a video generation method, comprising:

[0006] Get audio and reference images of the replica object;

[0007] Generate a dynamic image of the lip region of the replica object according to the audio and the reference image, and obtain the dynamic image of the replica object by combining the dynamic image of the lip region, the reference image and the emotional state corresponding to the audio; or extract a replica feature from the audio and the reference image, and generate the dynamic image of the replica object according to the replica feature and the emotional state corresponding to the audio;

[0008] A virtual image video of the replica object is generated according to the audio and the dynamic image of the replica object.

[0009] Furthermore, in the above method, generating a dynamic image of the lip area of ​​the replica object according to the audio and the reference image includes:

[0010] Extracting an image of the lip region of the replica object from the reference image;

[0011] A dynamic image of the lip area is generated based on the image of the lip area and the audio.

[0012] Furthermore, in the above method, the combining the dynamic image of the lip area, the reference image and the emotional state corresponding to the audio to obtain the dynamic image of the replica object includes:

[0013] fusing the dynamic image of the lip area and the reference image to obtain a dynamic image of the facial area of ​​the replica object;

[0014] Obtaining the body movements of the replica object according to the emotional state corresponding to the audio;

[0015] A dynamic image of the replica object is generated according to the dynamic image of the facial area and the body movements of the replica object.

[0016] Furthermore, in the above method, the dynamic image of the lip area includes feature points of the lip area and position information of the feature points; and fusing the dynamic image of the lip area and the reference image to obtain the dynamic image of the facial area of ​​the replica object includes:

[0017] According to the position information of the feature points, the dynamic image of the lip area is fused with the reference image to obtain a dynamic image of the facial area of ​​the replica object.

[0018] Furthermore, in the above method, extracting the replica features from the audio and the reference image includes:

[0019] Extracting temporal features from the audio, and extracting spatial features from the reference image; the spatial features include feature point information and muscle texture information of the face of the replica object;

[0020] The time feature and the space feature are integrated to obtain the replica feature.

[0021] Furthermore, in the above method, generating a dynamic image of the replica object according to the replica feature and the emotional state corresponding to the audio includes:

[0022] The emotional state corresponding to the audio is used as a prompt word, and the prompt word and the replica feature are input into a pre-trained generation model, so that the generation model generates a dynamic image of the replica object according to the replica feature and the prompt word.

[0023] Furthermore, the above method further includes:

[0024] Obtaining the requirements of the user and generating a response to the requirements;

[0025] Based on the reply content, reply content audio having a timbre consistent with that of the replica object is generated as the audio of the replica object.

[0026] Furthermore, the above method further includes:

[0027] Get the image information of the object being used;

[0028] Based on the image information of the user, the health data of the user is determined, and the health data is sent to a designated terminal; the health data includes the heart rate of the user.

[0029] In a second aspect, an embodiment of the present application provides a video generating device, including:

[0030] An acquisition module, used to acquire audio and reference images of the replica object;

[0031] A first generation module is used to generate a dynamic image of the lip area of ​​the replica object according to the audio and the reference image, and obtain the dynamic image of the replica object by combining the dynamic image of the lip area, the reference image and the emotional state corresponding to the audio; or extract replica features from the audio and the reference image, and generate the dynamic image of the replica object according to the replica features and the emotional state corresponding to the audio;

[0032] The second generating module is used to generate a virtual image video of the replica object according to the audio and the dynamic image of the replica object.

[0033] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0034] A memory and a processor; wherein the memory is used to store programs; and the processor is used to implement any of the methods described above by running the programs in the memory.

[0035] In a fourth aspect, an embodiment of the present application provides a computer program product, the computer program product comprising a computer program, and when the computer program is executed by a processor, the computer program implements any one of the above methods. Optionally, the computer program can be stored in a readable storage medium of a computer device or in the cloud; the processor of the computer device reads the computer program from the readable storage medium or the cloud.

[0036] The video generation method proposed in the present application can generate a virtual image video of the replica object based on the audio and reference image of the replica object. Specifically, a dynamic image of the lip area of ​​the replica object can be generated based on the audio and reference image, and the dynamic image of the lip area, the reference image and the emotional state corresponding to the audio can be combined to obtain the dynamic image of the replica object; or, the replica features can be extracted from the audio and reference image, and the dynamic image of the replica object can be generated based on the replica features and the emotional state corresponding to the audio, and then the virtual image video of the replica object can be generated based on the audio and the dynamic image of the replica object, so as to achieve the purpose of replicating the virtual image of a specific object. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0038] Figure 1 It is a flowchart of a video generation method provided in an embodiment of the present application.

[0039] Figure 2 It is a schematic diagram of the process of generating a dynamic image of the lip area provided in an embodiment of the present application.

[0040] Figure 3 It is a flowchart of another video generation method provided in an embodiment of the present application.

[0041] Figure 4 It is a structural schematic diagram of a video generating device provided in an embodiment of the present application.

[0042] Figure 5 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0044] In modern society, virtual image technology is increasingly widely used, and it plays an important role in many scenarios such as accompanying the elderly and children. For example, existing elderly care equipment can have daily conversations with the elderly through intelligent voice assistants or animated characters, providing emotional support and life assistance; companion robots for children use artificial intelligence and machine learning technology to simulate human movements, voices and expressions, interact with children, and provide companionship and entertainment. If the virtual image of a specific object can be replicated to interact with the companion object, the companionship function can be fully utilized and the companionship effect can be effectively improved. For example, the virtual image of the elderly’s children can be replicated to accompany the elderly, and the virtual image of the child’s parents can be replicated to accompany the child.

[0045] How to replicate a virtual image of a specific object has become a technical problem that needs to be solved urgently by those skilled in the art. Based on this, the present application proposes a video generation method, device, electronic device and computer program product. The technical solution processes the audio and reference image of the replicated object to achieve the effect of replicating a virtual image of a specific object.

[0046] The following examples provide a detailed description.

[0047] The embodiment of the present application provides a video generation method, which can be executed by an electronic device, which can be any device with data and instruction processing functions, for example, a notebook computer, a tablet computer, a desktop computer, a mobile device (for example, a mobile phone, a personal digital assistant, a dedicated messaging device) and other types of user terminals, or a combination of any two or more of these electronic devices, or a server. Figure 1 As shown, the method includes:

[0048] S101: Acquire audio and reference images of a replica object.

[0049] The above-mentioned replica object refers to an object for generating a virtual image. The specific identity of the replica object is not limited in the embodiments of the present application, and any object that requires virtual image replica can be used as the replica object of this embodiment.

[0050] The audio of the replica object refers to the audio that needs to be output when the virtual object interacts with the user when the virtual object video is generated. The voice content of the replica object can be directly obtained as the audio of the replica object, and the audio of the replica object can also be synthesized according to the timbre of the replica object, which is not limited in this embodiment.

[0051] In some embodiments, when synthesizing the audio of the replica object according to the timbre of the replica object, a speech replication model can be constructed using models such as VITS / WaveNet, and then the speech replication model can be trained to enable it to process the input reference audio and text and convert the text into audio consistent with the timbre of the reference audio.

[0052] The speech of the replicated object and the input text are input into the speech replication model, wherein the speech of the replicated object serves as the reference audio, and the input text serves as the text that needs to be synthesized according to the timbre of the replicated object. The speech replication model can extract the timbre of the replicated object from the speech of the replicated object, and then convert the above-mentioned input text into audio consistent with the timbre of the replicated object as the audio of the replicated object.

[0053] The reference image of the replica object refers to an image containing the original appearance or features of the replica object. The reference image of the replica object can be in picture format or video format, which is not limited in this embodiment.

[0054] In an embodiment of the present application, audio and reference images of a replicated object are obtained.

[0055] S102. Generate a dynamic image of the lip area of ​​the replica object based on the audio and the reference image, and obtain the dynamic image of the replica object by combining the dynamic image of the lip area, the reference image and the emotional state corresponding to the audio; or, extract replica features from the audio and the reference image, and generate the dynamic image of the replica object based on the replica features and the emotional state corresponding to the audio.

[0056] In some embodiments, a dynamic image of the lip region of the replica object is generated based on the audio and the reference image, and the dynamic image of the lip region, the reference image, and the emotional state corresponding to the audio are combined to obtain the dynamic image of the replica object.

[0057] In this embodiment, the reference image may be processed first to obtain an image of the lip region of the replica object.

[0058] Exemplarily, the image of the lip area of ​​the replica object can be extracted from the reference image by training the lip area extraction model. A large number of reference images are obtained as training samples, and the lip areas in the training samples are marked as training labels. During training, the training samples are input into the lip area extraction model to obtain the prediction results output by the lip area extraction model. By comparing the above prediction results and the training labels, the loss of the lip area extraction model is determined. With the goal of reducing the loss of the lip area extraction model, the parameters of the lip area extraction model are adjusted. Then the above training process is repeated until the loss function value, accuracy and other indicators of the lip area extraction model meet the requirements, and the lip area extraction model training is completed. The above lip area extraction model can be obtained by training based on any neural network model, or based on a pre-trained model, such as a pre-trained large model similar to ChatGPT. The reference image is input into the trained lip area extraction model to obtain the image of the lip area of ​​the replica object output by the lip area extraction model.

[0059] Then, based on the audio and the image of the lip area, a dynamic image of the lip area of ​​the replica object is generated.

[0060] The wave2lip model can be used to input audio and images of the lip area into the wave2lip model to obtain a dynamic image of the lip area of ​​the replica object output by the wave2lip model.

[0061] The dynamic image of the lip area is pasted back to the original reference image to obtain a dynamic image of the facial area of ​​the replica object. The virtual image of the replica object thus obtained only has the movements of the facial area. In order to improve the authenticity of the virtual image, body movements can be further added to the dynamic image of the facial area of ​​the replica object.

[0062] Specifically, the emotional state reflected by the audio can be determined, and the body movements of the replicated object can be determined in combination with the emotional state. For example, when the emotional state is aggrieved, the body movement is squatting; when the emotional state is happy, the body movement is jumping up, etc.

[0063] In some embodiments, the emotional state reflected by the audio can be determined by an emotional state model. Specifically, a large amount of audio can be obtained as training samples, and the emotional state corresponding to each training sample can be marked as a training label. During training, the training sample is input into the emotional state model to obtain the prediction result output by the emotional state model, and the loss of the emotional state model is determined by comparing the above prediction result and the training label. With the goal of reducing the loss of the emotional state model, the parameters of the emotional state model are adjusted. Then repeat the above training process until the loss function value, accuracy and other indicators of the emotional state model meet the requirements, and the emotional state model training is completed. The above emotional state model can be obtained based on any neural network model training, or based on a pre-trained model training, such as a pre-trained large model similar to ChatGPT. The above audio is input into the trained emotional state model to obtain the emotional state output by the emotional state model.

[0064] The mapping relationship between different emotional states and actions can be set in advance, and the corresponding body movement videos can be recorded and stored. For example, aggrieved corresponds to squatting, happy corresponds to jumping up, helpless corresponds to waving hands, etc., which are not limited in this embodiment. After determining the emotional state corresponding to the audio, the corresponding body movement video can be retrieved based on the pre-set mapping relationship. In this way, a dynamic image including facial movements and body movements can be obtained.

[0065] In other embodiments, replica features are extracted from the audio and the reference image, and a dynamic image of the replica object is generated according to the replica features and the emotional state corresponding to the audio.

[0066] Specifically, feature extraction may be performed on the audio and the reference image to obtain replica features. The replica features are used to generate a virtual image video of the replica object. In some embodiments, feature extraction may be performed on the audio and the reference image respectively, audio features may be extracted from the audio, and image features may be extracted from the reference image. The audio features and image features are then fused to obtain replica features containing the audio features and the image features.

[0067] Based on the above-mentioned replica features and the emotional state corresponding to the audio, a dynamic image of the replica object is generated.

[0068] Using the replica features, a dynamic image of the face of the replica object can be obtained. In some embodiments, a pre-trained facial generation model is used to generate a dynamic image of the face of the replica object. Specifically, a large number of reference images and audios can be obtained as training samples, and the dynamic image corresponding to the object shown in the reference image speaking the audio is used as a training label. During training, the training sample is input into the facial generation model to obtain the prediction result output by the facial generation model. By comparing the above prediction result and the training label, the loss of the facial generation model is determined, and the parameters of the facial generation model are adjusted with the goal of reducing the loss of the facial generation model. Then the above training process is repeated until the loss function value, accuracy and other indicators of the facial generation model meet the requirements, and the facial generation model training is completed. The above facial generation model can be obtained by training based on any neural network model, or based on a pre-trained model, such as a pre-trained large model similar to ChatGPT. The above audio and reference image are input into the trained facial generation model to obtain the facial dynamic image of the replica object output by the facial generation model.

[0069] The virtual image of the replica object thus obtained only has the movements of the facial area. In order to improve the authenticity of the virtual image, body movements can be further added to the dynamic image of the facial area of ​​the replica object. According to the records of the above embodiment, the emotional state model can be used to determine the emotional state according to the audio, and the mapping relationship between different emotional states and movements can be set in advance, and the corresponding body movement video can be recorded and stored. After determining the emotional state corresponding to the audio, the corresponding body movement video can be retrieved based on the preset mapping relationship. The specific implementation can refer to the records of the above embodiment, which will not be repeated here. In this way, a dynamic image including facial movements and body movements can be obtained.

[0070] S103: Generate a virtual image video of the replica object according to the audio and the dynamic image of the replica object.

[0071] Finally, the audio and the dynamic image of the replica object are combined together to obtain a virtual image video of the replica object containing sound and action.

[0072] In the above embodiments, a virtual image video of the replica object can be generated based on the audio and reference image of the replica object. Specifically, a dynamic image of the lip area of ​​the replica object can be generated based on the audio and reference image, and the dynamic image of the lip area, the reference image and the emotional state corresponding to the audio can be combined to obtain the dynamic image of the replica object; or, replica features can be extracted from the audio and reference image, and a dynamic image of the replica object can be generated based on the replica features and the emotional state corresponding to the audio, and then a virtual image video of the replica object can be generated based on the audio and the dynamic image of the replica object, so as to achieve the purpose of replicating the virtual image of a specific object.

[0073] As an optional implementation, Figure 2 As shown, in addition to generating a dynamic image of the lip area according to the description of the above embodiment, another embodiment of the present application discloses that the steps of the above embodiment generate a dynamic image of the lip area of ​​the replica object according to the audio and the reference image, and specifically may further include the following steps:

[0074] S201, extracting an image of the lip region of the replica object from a reference image.

[0075] According to the reference image, an image of the lip region of the replica object can be extracted from the reference image. Specifically, an image of the facial region of the replica object, that is, a face image of the replica object, can be first extracted from the reference image.

[0076] In some embodiments, a pre-trained face recognition model is used to extract the face image of the replica object from the reference image. Specifically, a large number of video images can be obtained as training samples, and the face images in each training sample can be extracted as training labels. During training, the training samples are input into the face recognition model to obtain the prediction results output by the face recognition model. By comparing the above prediction results and training labels, the loss of the face recognition model is determined. With the goal of reducing the loss of the face recognition model, the parameters of the face recognition model are adjusted. Then the above training process is repeated until the loss function value, accuracy and other indicators of the face recognition model meet the requirements, and the face recognition model training is completed. The above face recognition model can be obtained by training based on any neural network model, or can be obtained by training based on a pre-trained model, such as a pre-trained large model similar to ChatGPT. The above reference image is input into the trained face recognition model to obtain the face image of the replica object output by the face recognition model.

[0077] In some embodiments, the face recognition model is trained based on a speech2Face model, and the face recognition model can output a face image with a size that is consistent with the original face size.

[0078] Then, an image of the lip area of ​​the replica object is captured from the above-mentioned face image.

[0079] S202: Generate a dynamic image of the lip area according to the image and audio of the lip area.

[0080] After obtaining the image of the lip area, the image and audio of the lip area are used to generate a dynamic image of the lip area. The wave2lip model can be used to input the audio and the image of the lip area into the pre-trained wave2lip model to obtain the dynamic image of the lip area of ​​the replica object output by the wave2lip model.

[0081] More specifically, the dynamic image of the lip area includes the lip area image, feature points of the lip area, and position information of the feature points at different times. The feature points of the lip area refer to key coordinate points used to identify the shape, position, and dynamic changes of the lips. The number and position of the feature points of the lip area can be set according to actual conditions, and this embodiment does not limit this. For example, 20 feature points of the lip area can be set. The dynamic image of the lip area is obtained by changing the position of the feature points at different times, and the dynamic image of the lip area can be restored to the face image by using the feature points of the lip area.

[0082] In the above embodiment, a dynamic image of the lip region of the replica object can be extracted to achieve the purpose of replicating the virtual image of a specific object.

[0083] As an optional implementation, another embodiment of the present application discloses that the steps of the above embodiment are combined with the dynamic image of the lip area, the reference image and the emotional state corresponding to the audio to obtain the dynamic image of the replica object, which may specifically include the following steps:

[0084] The dynamic image of the lip area and the reference image are integrated to obtain the dynamic image of the facial area of ​​the replica object; the body movements of the replica object are obtained according to the emotional state corresponding to the audio; and the dynamic image of the replica object is generated according to the dynamic image of the facial area and the body movements of the replica object.

[0085] The dynamic image of the lip area is fused with the reference image. According to the position information of the feature points, the dynamic image of the lip area is fused with the reference image to obtain a dynamic image of the facial area of ​​the replica object.

[0086] Specifically, the feature points corresponding to the initial image before the action starts in the dynamic image of the lip area can be restored one by one to the positions of the same feature points in the reference image, so that the position corresponding to the dynamic image of the lip area is the real position of the lip area of ​​the replica object. As for the position corresponding to the original lip area in the reference image, the grayscale image of the mask can be used and an "AND" operation can be performed to quickly hollow out the position corresponding to the original lip area, leaving only the dynamic image of the lip area. In this way, a dynamic image of the facial area of ​​the replica object can be obtained.

[0087] Further, the body movements of the replica object are obtained according to the emotional state corresponding to the audio. According to the records of the above embodiment, the emotional state model can be used to determine the emotional state reflected by the audio, and then the body movements of the replica object can be determined in combination with the emotional state, for example, when the emotional state is aggrieved, the body movement is squatting, and when the emotional state is happy, the body movement is jumping up, etc.

[0088] The mapping relationship between different emotional states and actions can be set in advance, and the corresponding body movement videos and transition videos can be recorded and stored in advance. For example, aggrieved corresponds to squatting down, happy corresponds to jumping up, helpless corresponds to waving hands, etc., which are not limited in this embodiment. After determining the emotional state corresponding to the audio, the corresponding body movement video and transition video can be retrieved based on the pre-set mapping relationship. In some embodiments, after determining the emotional state, the state machine retrieves the corresponding body movement video and transition video, maintains the body movement, and obtains the body movement of the replica object.

[0089] The dynamic image of the facial area and the body movements of the replica object are combined to obtain a dynamic image of the replica object.

[0090] In this embodiment, since the body movement image is generated by the state machine using a pre-recorded video, it is only necessary to generate a dynamic image of the lip area, the synthesis area is small, and the computing power requirement is low.

[0091] As an optional implementation, another embodiment of the present application discloses that the steps of the above embodiment extract the replica features from the audio and the reference image, which may specifically include the following steps:

[0092] Temporal features are extracted from audio, and spatial features are extracted from reference images; temporal features and spatial features are fused to obtain replica features.

[0093] Specifically, feature extraction may be performed on audio to obtain temporal features, and feature extraction may be performed on reference images to obtain spatial features.

[0094] Among them, the spatial features include feature point information and muscle texture information of the face of the replica object. The feature point information of the face of the replica object includes the position of 68 feature points on the face, which can reflect the overall distribution of the facial body, including facial contours, facial features, etc. The spatial features of this embodiment include not only the feature point information of the face, but also the local muscle texture conditions of the left and right faces, because the distribution of facial muscles is different in different states, such as muscles are tight when angry and muscles are relaxed when happy. These detailed information cannot be obtained only through feature points. Therefore, the spatial features obtained by coupling feature point information and muscle texture information in this embodiment can contain emotional information, so the virtual image obtained based on the spatial features is more consistent with the image of the replica object.

[0095] For the above audio, the temporal features in the audio can be obtained in combination with the context. The temporal features can be understood as high-dimensional representations of features. According to the temporal features of the audio, a temporal graph is constructed, with each audio frame as a node in the graph. In the temporal graph of the audio, graph convolution operations are used to capture the temporal dependencies between audio frames. The graph convolutional network can calculate the representation of each node through the local neighborhood of the node, and at the same time, the graph attention mechanism is used to strengthen the feature learning of key moments, such as the moment of tone change.

[0096] The temporal features and spatial features are fused to obtain the replica features. Specifically, the feature entanglement technology can be used to perform multimodal data fusion on the spatial features and temporal features to construct a feature map. Gaussian filtering is used to smooth the feature map and filter out high-noise nodes, making the method more robust, obtaining a multimodal feature map with audio and image, and determining the above multimodal feature map as a replica feature.

[0097] In the above embodiment, the replica features of the replica object can be obtained, so as to accurately restore the virtual image of the replica object based on the replica features.

[0098] As an optional implementation, another embodiment of the present application discloses that the steps of the above embodiment generate a dynamic image of the replica object according to the replica features and the emotional state corresponding to the audio, which may specifically include the following steps:

[0099] The emotional state corresponding to the audio is used as a prompt word, and the prompt word and the replica features are input into a pre-trained generation model, so that the generation model generates a dynamic image of the replica object according to the replica features and the prompt word.

[0100] In this embodiment, a generation model is used to generate a dynamic image of a replica object. Specifically, the emotional state corresponding to the audio can be used as a prompt word, that is, the audio is used as a conditional constraint and the replica features are input into the generation model, so that the generation model generates a dynamic image of the replica object according to the replica features and the prompt word. The dynamic image of the replica object includes a virtual image of the replica object, wherein the facial and body movements of the virtual image are consistent with the emotional state, and the movement of the lip area of ​​the virtual image is the movement when the content corresponding to the audio is spoken.

[0101] Among them, the emotional state can be extracted from the audio using the emotion perception model according to the description of the above embodiment. Those skilled in the art can refer to the description of the above embodiment, which will not be elaborated here.

[0102] The generative model needs to be trained or fine-tuned in advance. Specifically, a large number of videos of people speaking can be recorded in advance, and the replica features and emotional states can be extracted from them as training samples according to the records of the above embodiments. The videos corresponding to each training sample are used as training labels. During training, the training samples are input into the generative model to obtain the prediction results output by the generative model. By comparing the above prediction results and training labels, the loss of the generative model is determined. With the goal of reducing the loss of the generative model, the parameters of the generative model are adjusted. Then repeat the above training process until the loss function value, accuracy and other indicators of the generative model meet the requirements, and the generative model training is completed. The above-mentioned generative model can be obtained based on any neural network model training, or based on a pre-trained model training, such as a pre-trained large model similar to ChatGPT. Input the reference image into the trained generative model to obtain a dynamic image of the replica object output by the generative model.

[0103] In some embodiments, a cross-modal attention mechanism can be used to dynamically adjust the weights of temporal features, spatial features, and emotional states to help the generative model better learn the relationship between different conditions.

[0104] As an optional implementation, Figure 3 As shown, in another embodiment of the present application, the method of the above embodiment may specifically include the following steps:

[0105] S301, detecting whether the current network status meets the set requirements; if the current network status meets the set requirements, executing S302; if the current network status does not meet the set requirements, executing S303.

[0106] The above setting requirements refer to the requirements for the network status, generally the requirements for the network speed. The above setting requirements can be set according to actual conditions, and this embodiment does not limit them.

[0107] It can be detected whether the current network status meets the set requirements. If the current network status meets the set requirements, S302 can be executed. If the current network status does not meet the set requirements, S303 can be executed.

[0108] S302, initiate a generation request to the cloud, the generation request is used to trigger the cloud to extract replica features from the audio and reference images, generate a dynamic image of the replica object according to the replica features and the emotional state corresponding to the audio, and generate a virtual image video of the replica object according to the audio and the dynamic image of the replica object.

[0109] The above generation request carries audio and reference images.

[0110] Specifically, when the network status meets the set requirements, that is, the network status is good, a virtual image video of the replica object can be generated through the cloud. Specifically, a generation request can be sent to the cloud, so that the cloud can extract the replica features from the audio and the reference image according to the generation request, generate a dynamic image of the replica object according to the replica features and the emotional state corresponding to the audio, and generate a virtual image video of the replica object according to the audio and the dynamic image of the replica object.

[0111] It should be noted that the cloud extracts replica features from audio and reference images, generates a dynamic image of the replica object based on the replica features and the emotional state corresponding to the audio, and generates a virtual image video of the replica object based on the audio and the dynamic image of the replica object. The specific process is the same as the description in the above embodiment. Those skilled in the art may refer to the above description and will not elaborate here.

[0112] By extracting the obtained replica features and emotional states and directly generating a dynamic image of the replica object, the entire human face can be generated. The resulting virtual image has more details, stronger and more accurate emotional expression, and is closer to the replica object. Therefore, greater computing power resources are required. In this embodiment, when the network status is good, a cloud with more sufficient computing power resources can be requested for processing to obtain a virtual image that is closer to the replica object.

[0113] S303, generating a dynamic image of the lip area of ​​the replica object based on the audio and the reference image, and combining the dynamic image of the lip area, the reference image and the emotional state corresponding to the audio to obtain the dynamic image of the replica object, and generating a virtual image video of the replica object based on the audio and the dynamic image of the replica object.

[0114] When the network state meets the unset requirements, that is, the network state is poor, a virtual image video of the replica object can be generated at the terminal. Specifically, a dynamic image of the lip area of ​​the replica object can be generated based on the audio and the reference image, and the dynamic image of the lip area, the reference image and the emotional state corresponding to the audio can be combined to obtain the dynamic image of the replica object, and the virtual image video of the replica object can be generated based on the audio and the dynamic image of the replica object.

[0115] It should be noted that the terminal generates a dynamic image of the lip area of ​​the replica object based on the audio and the reference image, and obtains the dynamic image of the replica object by combining the dynamic image of the lip area, the reference image and the emotional state corresponding to the audio. The specific process of generating a virtual image video of the replica object based on the audio and the dynamic image of the replica object is the same as the description in the above embodiment. Those skilled in the art may refer to the above description and will not elaborate here.

[0116] The method recorded in this step only needs to generate a dynamic image of the lip area. The synthesis area is small and the computing power requirement is low. Therefore, when the network conditions are poor, a virtual image video can be generated on the terminal to meet user needs.

[0117] This embodiment can select a method for generating a virtual image video according to the network status, and improve the degree of closeness between the virtual image video and the replica object on the basis of meeting the user's needs.

[0118] As an optional implementation, another embodiment of the present application discloses that the method of the above embodiment may specifically include the following steps:

[0119] The demand content of the user object is obtained, and the reply content for the demand content is generated; based on the reply content, the reply content audio with the same timbre as the replica object is generated as the audio of the replica object.

[0120] In this embodiment, the audio of the replica object is synthesized according to the timbre of the replica object. Specifically, the demand content of the user object is first obtained, and then the corresponding reply content is generated based on the demand content of the user object. Exemplarily, a large language model can be used to generate the reply content based on the demand content of the user object. The large language model can adopt a mature large language model in the prior art, such as Wenxin Yiyan, ChatGPT, etc., which is not limited in this embodiment.

[0121] More specifically, the user generally uses voice to express his or her needs. In this embodiment, the voice recognition model can be used to convert the voice needs into text, and the text needs are input into the large language model to obtain the response output by the large language model. The voice recognition model can use a hidden Markov model (HMM) or a deep neural network (DNN).

[0122] In some embodiments, when synthesizing the audio of the replica object according to the timbre of the replica object, a speech replication model can be constructed using models such as VITS / WaveNet, and then the speech replication model can be trained to enable it to process the input reference audio and text and convert the text into audio consistent with the timbre of the reference audio.

[0123] The speech and reply content of the replica object are input into the speech replication model, wherein the speech of the replica object is used as the reference audio, and the reply content is used as the text that needs to be synthesized according to the timbre of the replica object. The speech replication model can extract the timbre of the replica object from the speech of the replica object, and then convert the above reply content into audio consistent with the timbre of the replica object as the audio of the replica object.

[0124] In the embodiments of the present application, audio having the same timbre as the replica object can be automatically generated, so as to generate a virtual image video of the replica object based on the audio.

[0125] As an optional implementation, another embodiment of the present application discloses that the method of the above embodiment may specifically include the following steps:

[0126] Acquire image information of the user; determine health data of the user based on the image information of the user, and send the health data to a designated terminal; the health data includes the heart rate of the user.

[0127] In the embodiments of the present application, the physical condition of the user can also be monitored. Specifically, the image information of the user can be obtained, and then the physical condition of the user can be analyzed based on the image information of the user to obtain the health data of the user, and the health data of the user can also be sent to a designated terminal. Other users can view the health data of the user at the designated terminal to understand the physical condition of the user.

[0128] In addition, when the health data of the user is abnormal and is not within the normal data range, an alarm can be issued, or the above-mentioned designated terminal can be controlled to issue an alarm.

[0129] In some embodiments, the health data includes the heart rate of the user. This embodiment uses a non-contact heart rate monitoring remote photoplethysmography (rPPG) method to detect heart rate. When the heart beats, it causes periodic changes in blood volume in the blood vessels. This change causes a slight change in the intensity of light reflected from the skin surface. By capturing this change with a camera and using signal processing techniques such as independent component analysis, signals related to heart rate can be extracted, thereby achieving heart rate measurement. Specifically, the DLIB model is used to detect key points of the face in the image, and then the heart rate data is obtained by analyzing the face image.

[0130] In a specific embodiment, the video generation method of this embodiment can be applied to an intelligent companion robot. The intelligent companion robot can interact with a user object.

[0131] Specifically, the requirements of the user can be obtained, and the reply content for the requirements can be generated; based on the reply content, the reply content audio with the same timbre as the replica object is generated as the audio of the replica object. A dynamic image of the lip area of ​​the replica object is generated based on the audio and reference image of the replica object, and the dynamic image of the lip area, the reference image and the emotional state corresponding to the audio are combined to obtain the dynamic image of the replica object; or, the replica features are extracted from the audio and reference image of the replica object, and the dynamic image of the replica object is generated based on the replica features and the emotional state corresponding to the audio; based on the audio and the dynamic image of the replica object, a virtual image video of the replica object is generated.

[0132] The intelligent companion robot can also detect the user's physical condition. Specifically, it can obtain the user's image information; based on the user's image information, determine the user's health data and send the health data to a designated terminal; the health data includes the user's heart rate. The designated terminal can be a mobile phone, watch or other device of the user's family, which is not limited in this embodiment.

[0133] The intelligent companion robot can be used to accompany the elderly or children, etc., which is not limited in this embodiment.

[0134] Corresponding to the above-mentioned video generation method, the embodiment of the present application also discloses a video generation device, see Figure 5 As shown, the device comprises:

[0135] An acquisition module 100 is used to acquire audio and reference images of a replica object;

[0136] The first generation module 110 is used to generate a dynamic image of the lip area of ​​the replica object according to the audio and the reference image, and obtain the dynamic image of the replica object by combining the dynamic image of the lip area, the reference image and the emotional state corresponding to the audio; or extract the replica features from the audio and the reference image, and generate the dynamic image of the replica object according to the replica features and the emotional state corresponding to the audio;

[0137] The second generating module 120 is used to generate a virtual image video of the replica object according to the audio and the dynamic image of the replica object.

[0138] As an optional implementation method, another embodiment of the present application discloses that the first generation module 110 of the above embodiment is specifically used to extract the image of the lip area of ​​the replica object from the reference image; and generate a dynamic image of the lip area based on the image and audio of the lip area.

[0139] As an optional implementation method, another embodiment of the present application discloses that the first generation module 110 of the above embodiment is specifically used to fuse the dynamic image of the lip area and the reference image to obtain the dynamic image of the facial area of ​​the replica object; obtain the body movements of the replica object according to the emotional state corresponding to the audio; and generate the dynamic image of the replica object according to the dynamic image of the facial area and the body movements of the replica object.

[0140] As an optional implementation method, another embodiment of the present application discloses that the dynamic image of the lip area of ​​the above embodiment includes feature points of the lip area and position information of the feature points; the first generation module 110 of the above embodiment is specifically used to fuse the dynamic image of the lip area with the reference image according to the position information of the feature points to obtain the dynamic image of the facial area of ​​the replica object.

[0141] As an optional implementation method, another embodiment of the present application discloses that the first generation module 110 of the above embodiment is specifically used to extract time features from audio and extract spatial features from reference images; the spatial features include feature point information and muscle texture information of the face of the replica object; and the time features and spatial features are fused to obtain the replica features.

[0142] As an optional implementation method, disclosed in another embodiment of the present application, the first generation module 110 of the above embodiment takes the emotional state corresponding to the audio as a prompt word, and inputs it into a pre-trained generation model together with the replica features, so that the generation model generates a dynamic image of the replica object according to the replica features and the prompt words.

[0143] As an optional implementation, another embodiment of the present application discloses that the device in the above embodiment further includes:

[0144] The first acquisition module is used to acquire the demand content of the user and generate the reply content according to the demand content;

[0145] The third generating module is used to generate, based on the reply content, reply content audio having a timbre consistent with that of the replica object as the audio of the replica object.

[0146] As an optional implementation, another embodiment of the present application discloses that the device in the above embodiment further includes:

[0147] A second acquisition module is used to acquire image information of the object;

[0148] The determination module is used to determine the health data of the user based on the image information of the user, and send the health data to a designated terminal; the health data includes the heart rate of the user.

[0149] Specifically, the device provided in this embodiment belongs to the same application concept as the method provided in the above embodiment of this application, can execute the method provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of the execution method. For the technical details not fully described in this embodiment, please refer to the specific processing content of the method provided in the above embodiment of this application, which will not be repeated here.

[0150] The functions implemented by the above modules can be implemented by the same or different processors respectively, and the embodiments of the present application are not limited thereto.

[0151] It should be understood that the units in the above devices can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit of the device, wherein the processor can be a general-purpose processor, such as a CPU or a microprocessor, etc., and the memory can be a memory in the device or a memory outside the device. Alternatively, the units in the device can be implemented in the form of hardware circuits, and the functions of some or all units can be realized by designing the hardware circuits. The hardware circuit can be understood as one or more processors; for example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be implemented by PLD, taking FPGA as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by the configuration file, so as to realize the functions of some or all of the above units. All units of the above devices can be implemented in the form of a processor calling software, or in the form of hardware circuits, or in part by a processor calling software, and the remaining part is implemented in the form of hardware circuits.

[0152] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and run instructions, such as a CPU, a microprocessor, a GPU, or a DSP; in another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.

[0153] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0154] In addition, all or part of the units in the above device can be integrated together, or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a SOC. The SOC may include at least one processor for implementing any of the above methods or implementing the functions of each unit of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0155] An embodiment of the present application further provides a control device, which includes a processor and an interface circuit. The processor in the control device is connected to an input-output component through the interface circuit of the control device.

[0156] The input-output component specifically refers to a hardware component that enables a user to input information and output information to the user, such as a microphone, keyboard, handwriting tablet, touch screen, display, speaker, printer, etc.

[0157] The above-mentioned interface circuit can be any interface circuit that can realize the data communication function, for example, it can be a USB interface circuit, a Type-C interface circuit, a serial port circuit, a PCIE circuit, etc.

[0158] The processor in the control device is a circuit with signal processing capability, which realizes the purpose of replicating the virtual image of a specific object by executing any one of the video generation methods introduced in the above embodiments. The specific implementation of the processor can refer to the above processor implementation, and the embodiments of this application are not strictly limited.

[0159] When the control device is applied to a device with a human-computer interaction function, the input and output components of the control device may be input components and output components on the device, such as a microphone, a keyboard, a handwriting tablet, a touch screen, a display, an audio player, etc. At the same time, the processor of the control device may be a CPU or GPU, etc. provided by the device, and the interface circuit of the control device may be an interface circuit between the information input component of the device and a processor such as a CPU or GPU.

[0160] Corresponding to the above-mentioned video generation method, the embodiment of the present application further discloses an electronic device, as shown in the figure, the electronic device includes:

[0161] Memory 200 and processor 210;

[0162] The memory 200 is connected to the processor 210 and is used to store programs;

[0163] The processor 210 is used to implement the video generation method disclosed in any of the above embodiments by running the program stored in the memory 200.

[0164] Specifically, the electronic device may further include: a bus, a communication interface 220 , an input device 230 and an output device 240 .

[0165] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are connected to each other via a bus.

[0166] A bus may include a pathway that transfers information between components of a computer system.

[0167] The processor 210 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present application. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0168] The processor 210 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0169] The memory 200 stores a program for executing the technical solution of the present application, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes computer operation instructions. More specifically, the memory 200 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.

[0170] The input device 230 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0171] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0172] The communication interface 220 may include any transceiver or the like to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0173] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement the various steps of the video generation method provided in the above embodiments of the present application.

[0174] In addition to the above methods and devices, the embodiments of the present application may also be a computer program product, which includes a computer program. When the computer program is executed by a processor, the video generation method provided by any of the above embodiments of the present application may be executed. Optionally, the computer program may be stored in a readable storage medium or in the cloud of a computer device; the processor of the computer device reads the computer program from the readable storage medium or the cloud.

[0175] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0176] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK).

[0177] In addition, the embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes each step of the video generation method provided in the above embodiment.

[0178] The computer readable storage medium may adopt any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0179] Specifically, the specific working contents of each part of the above-mentioned electronic device, computer program product and storage medium, as well as the specific processing contents when the computer program product or the computer program on the above-mentioned storage medium is executed by the processor, can all be found in the contents of the various embodiments of the above-mentioned video generation method, and will not be repeated here.

[0180] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.

[0181] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0182] The steps in the methods of each embodiment of the present application can be adjusted in sequence, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0183] The modules and sub-modules in the devices and terminals in the various embodiments of the present application can be combined, divided and deleted according to actual needs.

[0184] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, such as multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0185] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0186] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.

[0187] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0188] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0189] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including a..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0190] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A video generation method, characterized in that: include: Get audio and reference images of the replica object; Generate a dynamic image of the lip region of the replica object according to the audio and the reference image, and obtain the dynamic image of the replica object by combining the dynamic image of the lip region, the reference image and the emotional state corresponding to the audio; or extract a replica feature from the audio and the reference image, and generate the dynamic image of the replica object according to the replica feature and the emotional state corresponding to the audio; A virtual image video of the replica object is generated according to the audio and the dynamic image of the replica object.

2. The method according to claim 1, characterized in that The step of generating a dynamic image of the lip region of the replica object according to the audio and the reference image comprises: Extracting an image of the lip region of the replica object from the reference image; A dynamic image of the lip area is generated based on the image of the lip area and the audio.

3. The method according to claim 1, characterized in that The step of combining the dynamic image of the lip region, the reference image, and the emotional state corresponding to the audio to obtain the dynamic image of the replica object includes: fusing the dynamic image of the lip area and the reference image to obtain a dynamic image of the facial area of ​​the replica object; Obtaining the body movements of the replica object according to the emotional state corresponding to the audio; A dynamic image of the replica object is generated according to the dynamic image of the facial area and the body movements of the replica object.

4. The method according to claim 3, characterized in that The dynamic image of the lip region includes feature points of the lip region and position information of the feature points; the step of fusing the dynamic image of the lip region and the reference image to obtain the dynamic image of the facial region of the replica object includes: According to the position information of the feature points, the dynamic image of the lip area is fused with the reference image to obtain a dynamic image of the facial area of ​​the replica object.

5. The method according to claim 1, characterized in that The extracting the replica features from the audio and the reference image comprises: Extracting temporal features from the audio, and extracting spatial features from the reference image; the spatial features include feature point information and muscle texture information of the face of the replica object; The time feature and the space feature are integrated to obtain the replica feature.

6. The method according to claim 1, characterized in that The step of generating a dynamic image of the replica object according to the replica feature and the emotional state corresponding to the audio includes: The emotional state corresponding to the audio is used as a prompt word, and the prompt word and the replica feature are input into a pre-trained generation model, so that the generation model generates a dynamic image of the replica object according to the replica feature and the prompt word.

7. The method according to any one of claims 1 to 6, characterized in that: Also includes: Obtaining the requirements of the user and generating a response to the requirements; Based on the reply content, reply content audio having a timbre consistent with that of the replica object is generated as the audio of the replica object.

8. The method according to any one of claims 1 to 6, characterized in that: Also includes: Get the image information of the object being used; Based on the image information of the user, the health data of the user is determined, and the health data is sent to a designated terminal; the health data includes the heart rate of the user.

9. A video generating device, characterized in that: include: An acquisition module, used to acquire audio and reference images of the replica object; A first generation module is used to generate a dynamic image of the lip area of ​​the replica object according to the audio and the reference image, and obtain the dynamic image of the replica object by combining the dynamic image of the lip area, the reference image and the emotional state corresponding to the audio; or extract replica features from the audio and the reference image, and generate the dynamic image of the replica object according to the replica features and the emotional state corresponding to the audio; The second generating module is used to generate a virtual image video of the replica object according to the audio and the dynamic image of the replica object.

10. An electronic device, characterized in that: include: Memory and processor; Wherein, the memory is used to store programs; The processor is used to implement the method according to any one of claims 1 to 8 by running the program in the memory.

11. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images