Video generation methods, apparatus, equipment and storage media

Abstract

This invention relates to the fields of artificial intelligence and digital healthcare, providing a video generation method, apparatus, device, and storage medium. The method extracts facial expression features from each source face image in a source video sequence, extracts facial features from the person image, generates multiple target face images based on the facial expression and facial features, and detects the multiple target face images using a face discriminator and each source face image. If the detection passes, a mixed shape-driving parameter sequence is extracted from the multiple target face images and rendered, improving the efficiency of video animation generation. Furthermore, this invention also relates to blockchain technology, allowing the video animation to be stored in the blockchain.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and digital medical technology, and in particular to a video generation method, apparatus, device and storage medium. Background Technology

[0002] With the development of artificial intelligence, 3D virtual human-driven migration solutions have emerged. In existing 3D virtual human-driven migration solutions, the process of migrating virtual characters typically requires artists to adjust the animation using motion tools. This results in the inability to reuse animations created on previous virtual characters when changing them, leading to low efficiency in generating new motion videos. With the rise of 3D virtual human-driven migration technology, solutions can support functions such as disease-assisted diagnosis, health management, and remote consultations. Summary of the Invention

[0003] In view of the above, it is necessary to provide a video generation method, apparatus, device, and storage medium that can solve the technical problem of how to improve the generation efficiency of action videos.

[0004] On one hand, the present invention proposes a video generation method, the video generation method comprising:

[0005] Obtain the source video sequence of the source person and the image of the target person;

[0006] An image synthesis model is obtained, the image synthesis model including a face generator and a face discriminator, the face generator including an expression extraction network and a facial feature extraction network;

[0007] Based on the expression extraction network, extract the expression features of each source face image in the source video sequence;

[0008] The facial features of the person image are extracted using the facial feature extraction network.

[0009] Multiple target face images are generated based on the facial expression features and the facial features;

[0010] The multiple target face images are detected based on the face discriminator and each source face image;

[0011] If all of the multiple target face images pass the detection of the face discriminator, then based on the pre-trained driving parameter recognition model, a mixed shape driving parameter sequence is extracted from the multiple target face images;

[0012] The blended shape driving parameter sequence is rendered to obtain the video animation of the target character.

[0013] According to a preferred embodiment of the present invention, the face discriminator includes a discriminant extraction network and a discriminant output network, and the step of detecting the plurality of target face images based on the face discriminator and each source face image includes:

[0014] Based on the discriminative extraction network, source expression information is extracted from each source face image, and target expression information is extracted from each target face image;

[0015] By comparing the target facial expression information with the source facial expression information, the difference facial expression information is obtained;

[0016] Based on the discriminant output network, the differential facial expression information is discriminated to obtain a discrimination result;

[0017] If all the discrimination results are preset results, then it is determined that all of the multiple target face images have passed the detection of the face discriminator; or

[0018] If the discrimination results are not all the preset results, then it is determined that there is a target face image among the multiple target face images that does not pass the detection of the face discriminator.

[0019] According to a preferred embodiment of the present invention, after detecting the plurality of target face images based on the face discriminator and each source face image, the video generation method further includes:

[0020] If any of the multiple target face images fails the detection of the face discriminator, then the target face image whose discrimination result is not the preset result is determined as an image to be processed;

[0021] Based on the difference in facial expression information, the facial expression of the image to be processed is adjusted to obtain an adjusted image.

[0022] According to a preferred embodiment of the present invention, the driving parameter recognition model includes a key point recognition network, a category recognition network, and a convolutional network and parameter output network corresponding to each face category. The extraction of a mixed shape driving parameter sequence from the multiple target face images based on the pre-trained driving parameter recognition model includes:

[0023] Based on the key point recognition network, facial key points are identified in each target face image;

[0024] Based on the category recognition network, category key points corresponding to each face category are identified from the facial key points;

[0025] Based on the location information of the key points of the category on the target face image, key point encoding information for each face category is generated;

[0026] The key point encoding information is input into the corresponding convolutional network to obtain the positional features of each face category;

[0027] By concatenating multiple positional features, a concatenated feature is obtained;

[0028] The stitched features are mapped based on the parameter output network to obtain the output driving parameters for each target face image;

[0029] The output driving parameters are spliced ​​according to the splicing order of multiple source face images in the source video sequence to obtain the hybrid shape driving parameter sequence.

[0030] According to a preferred embodiment of the present invention, the step of mapping the stitched features based on the parameter output network to obtain the output driving parameters for each target face image includes:

[0031] Extract the weight matrix and weight offset from the parameter output network;

[0032] Calculate the dot product of the weight matrix and the concatenated features;

[0033] The target matrix is ​​generated based on the sum of the weight offsets of each element in the dot product result.

[0034] The target matrix is ​​mapped based on the mapping table in the parameter output network to obtain the output driving parameters.

[0035] According to a preferred embodiment of the present invention, the expression extraction network includes multiple deconvolution layers, and the step of extracting expression features of each source face image in the source video sequence based on the expression extraction network includes:

[0036] An image vector is generated based on the pixel information of each source face image;

[0037] The image vector is transformed to obtain the reconstructed vector;

[0038] Based on the multiple deconvolutional layers, the reshaped vector is upsampled multiple times to obtain the facial expression features.

[0039] According to a preferred embodiment of the present invention, generating multiple target face images based on the expression features and the facial features includes:

[0040] By fusing the facial expression features and the facial features, the target features are obtained;

[0041] The target features are decoded to obtain the multiple target face images.

[0042] On the other hand, the present invention also proposes a video generation apparatus, the video generation apparatus comprising:

[0043] The acquisition unit is used to acquire the source video sequence of the source person and the image of the target person.

[0044] The acquisition unit is also used to acquire an image synthesis model, the image synthesis model including a face generator and a face discriminator, the face generator including an expression extraction network and a facial feature extraction network;

[0045] An extraction unit is used to extract facial expression features from each source face image in the source video sequence based on the facial expression extraction network.

[0046] The extraction unit is also used to extract facial features of the person image based on the facial feature extraction network;

[0047] The generation unit is used to generate multiple target face images based on the expression features and the face features;

[0048] The detection unit is used to detect the multiple target face images based on the face discriminator and each source face image;

[0049] The extraction unit is further configured to extract a mixed shape driving parameter sequence from the multiple target face images based on a pre-trained driving parameter recognition model if all of the multiple target face images pass the detection of the face discriminator.

[0050] The rendering unit is used to render the blended shape driving parameter sequence to obtain the video animation of the target character.

[0051] On the other hand, the present invention also proposes an electronic device, the electronic device comprising:

[0052] Memory, which stores computer-readable instructions; and

[0053] The processor executes computer-readable instructions stored in the memory to implement the video generation method.

[0054] On the other hand, the present invention also proposes a computer-readable storage medium storing computer-readable instructions, which are executed by a processor in an electronic device to implement the video generation method.

[0055] As can be seen from the above technical solutions, this application, through the face generator, can fuse the expression features of the source face image and the face features of the person image, thereby generating a target face image with the expression features. Then, by using the face discriminator and each source face image to detect the multiple target face images, it can ensure that the target expression information in the multiple target face images is as close as possible to the source expression information, improving the generation accuracy of the multiple target face images. When the target person changes, only the person image of the target person needs to be input into the image synthesis model to directly obtain the corresponding target face image, thereby improving the generation efficiency of the video animation. Furthermore, when all multiple target face images pass the detection of the face discriminator, this application directly extracts the mixed shape driving parameter sequence from the multiple target face images through the driving parameter recognition model, which can improve the generation efficiency of the mixed shape driving parameter sequence, thereby further improving the generation efficiency of the video animation. Furthermore, since the driving parameter recognition model is a self-supervised model, no data needs to be acquired when training the driving parameter recognition model, thus reducing the cost of data acquisition. Attached Figure Description

[0056] Figure 1 This is a flowchart of a preferred embodiment of the video generation method of the present invention.

[0057] Figure 2 This is a network structure diagram of the image synthesis model in the video generation method of this invention.

[0058] Figure 3 This is a functional block diagram of a preferred embodiment of the video generation device of the present invention.

[0059] Figure 4 This is a schematic diagram of the structure of an electronic device that implements the video generation method of the present invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0061] like Figure 1 The diagram shown is a flowchart of a preferred embodiment of the video generation method of the present invention. The order of the steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements.

[0062] The video generation method described above can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0063] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0064] The video generation method is applied to one or more electronic devices. The electronic device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored computer-readable instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0065] The electronic device can be any electronic product that can interact with the user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), game console, interactive network television (IPTV), smart wearable device, etc.

[0066] The electronic devices may include network devices and / or user devices. The network devices include, but are not limited to, single network electronic devices, groups of multiple network electronic devices, or cloud computing-based systems consisting of a large number of hosts or network electronic devices.

[0067] The network in which the electronic device is located includes, but is not limited to: the Internet, wide area network, metropolitan area network, local area network, virtual private network (VPN), etc.

[0068] 101. Obtain the source video sequence of the source person and the image of the target person.

[0069] In at least one embodiment of the present invention, the source person can be any broadcaster, and the source video sequence refers to a video segment of the broadcaster making a voice announcement. In medical applications, the source person can also be any patient, and the source video sequence refers to a video segment of the patient undergoing a remote consultation. The source video sequence includes multiple source facial images.

[0070] The target character can be any digital virtual character, for example, the target character can be a virtual character in the shape of a mouse, etc.

[0071] The image of the person refers to a two-dimensional facial image of the target person.

[0072] In at least one embodiment of the present invention, the electronic device acquires the source video sequence of the source person by including:

[0073] The initial source video is located based on the character identifier of the source character;

[0074] The video frames that need to be used for character migration are extracted from the initial source video and used as the source video sequence.

[0075] 102. Obtain an image synthesis model, the image synthesis model including a face generator and a face discriminator, the face generator including an expression extraction network and a facial feature extraction network.

[0076] In at least one embodiment of the present invention, the image synthesis model is used to synthesize the expression features of each source face image and the facial features of the person image into the same image.

[0077] In at least one embodiment of the present invention, the facial expression extraction network includes multiple deconvolutional layers.

[0078] like Figure 2 The diagram shown is a network structure diagram of the image synthesis model in the video generation method of this invention. Figure 2 In the image synthesis model, a face generator and a face discriminator are included. The face generator includes an expression extraction network and a facial feature extraction network, and the face discriminator includes a discrimination extraction network and a discrimination output network.

[0079] 103. Based on the expression extraction network, extract the expression features of each source face image in the source video sequence.

[0080] In at least one embodiment of the present invention, the facial expression features refer to the features on each source face image used to express mood. For example, the facial expression features may include, but are not limited to, blinking, mouth raising, mouth movement posture, etc.

[0081] In at least one embodiment of the present invention, the electronic device extracts facial expression features of each source face image in the source video sequence according to the facial expression extraction network, including:

[0082] An image vector is generated based on the pixel information of each source face image;

[0083] The image vector is transformed to obtain the reconstructed vector;

[0084] Based on the multiple deconvolutional layers, the reshaped vector is upsampled multiple times to obtain the facial expression features.

[0085] The reshaped vector refers to the vector generated after dimensional transformation of the image vector. For example, if the image vector has 1*100 dimensions, then the reshaped vector can have 4*4*1024 dimensions.

[0086] The multiple deconvolutional layers refer to deconvolutional networks with different kernel sizes. For example, if the multiple deconvolutional layers include three deconvolutional layers, namely a first deconvolutional layer, a second deconvolutional layer, and a third deconvolutional layer, then the reshaped vector is input into the first deconvolutional layer for upsampling, the sampled features output by the first deconvolutional layer are input into the second deconvolutional layer for upsampling, and the sampled features output by the second deconvolutional layer are input into the third deconvolutional layer for upsampling to obtain the facial expression features.

[0087] By reshaping and transforming the image vector, it is possible to ensure that the number of rows and columns of the reshaping vector is the same, thereby improving the analysis efficiency of the reshaping vector. By performing multiple upsampling processes on the reshaping vector, the receptive field of the facial expression features can be increased, which is beneficial to the combination of the facial expression features and the facial features.

[0088] 104. Extract facial features from the person image based on the facial feature extraction network.

[0089] In at least one embodiment of the present invention, the network structure of the facial feature extraction network is the same as that of the expression extraction network, but the network parameters of the facial feature extraction network are different from those of the expression extraction network.

[0090] The facial features refer to the characteristics used to identify the target person.

[0091] In at least one embodiment of the present invention, the way in which the electronic device extracts the facial features of the person image based on the facial feature extraction network is similar to the way in which the electronic device extracts the expression features of each source face image in the source video sequence based on the expression extraction network, and will not be described in detail here.

[0092] 105. Generate multiple target face images based on the expression features and the face features.

[0093] In at least one embodiment of the present invention, each target face image refers to an image having the representational features of the corresponding source face image and the face features of the person image.

[0094] In at least one embodiment of the present invention, the electronic device generates multiple target face images based on the expression features and the facial features, including:

[0095] By fusing the facial expression features and the facial features, the target features are obtained;

[0096] The target features are decoded to obtain the multiple target face images.

[0097] By fusing the facial expression features and the facial features, the target features combine the facial expression features and the facial features, thereby generating a target face image that has the representational features of the corresponding source face image and the facial features of the person image.

[0098] 106. Detect the multiple target face images based on the face discriminator and each source face image.

[0099] In at least one embodiment of the present invention, the face discriminator is used to determine the difference between the target expression information of each target face image and the source expression information of each source face image.

[0100] In at least one embodiment of the present invention, the electronic device detects the plurality of target face images based on the face discriminator and each source face image, including:

[0101] Based on the discriminative extraction network, source expression information is extracted from each source face image, and target expression information is extracted from each target face image;

[0102] By comparing the target facial expression information with the source facial expression information, the difference facial expression information is obtained;

[0103] Based on the discriminant output network, the differential facial expression information is discriminated to obtain a discrimination result;

[0104] If all the discrimination results are preset results, then it is determined that all of the multiple target face images have passed the detection of the face discriminator; or

[0105] If the discrimination results are not all the preset results, then it is determined that there is a target face image among the multiple target face images that does not pass the detection of the face discriminator.

[0106] The discriminant extraction network includes multiple convolutional layers.

[0107] The source expression information refers to the expression information extracted from each source face image based on the discriminant extraction network. The target expression information refers to the expression information extracted from each target face image based on the discriminant extraction network.

[0108] The preset result can be that the target face image belongs to a real image.

[0109] The discriminant extraction network simultaneously extracts corresponding source expression information and target expression information from each source face image and each target face image, enabling the quantification of the source expression information and the target expression information from the same perspective, thereby improving the accuracy of the differential expression information. The discriminant output network can accurately discriminate the differential expression information, improving the accuracy of the discrimination result.

[0110] Specifically, the discrimination output network includes a discrimination matrix, and the electronic device, based on the discrimination output network, discriminates the differential expression information to obtain the discrimination result, including:

[0111] Calculate the product of the discrimination matrix and the differential facial expression information;

[0112] Compare the product with the network threshold in the discriminative output network;

[0113] If the product is less than or equal to the network threshold, then the discrimination result is determined to be the preset result.

[0114] In at least one embodiment of the present invention, after detecting the plurality of target face images based on the face discriminator and each source face image, the video generation method further includes:

[0115] If any of the multiple target face images fails the detection of the face discriminator, then the target face image whose discrimination result is not the preset result is determined as an image to be processed;

[0116] Based on the difference in facial expression information, the facial expression of the image to be processed is adjusted to obtain an adjusted image.

[0117] By using the differential expression information, the facial expressions of target faces whose discrimination results are not the preset results are adjusted to avoid large errors in the subsequently extracted hybrid shape driving parameter sequence due to excessive differences in the differential expression information.

[0118] 107. If all the target face images pass the detection of the face discriminator, then based on the pre-trained driving parameter recognition model, the mixed shape driving parameter sequence is extracted from the multiple target face images.

[0119] In at least one embodiment of the present invention, the driving parameter recognition model includes a key point recognition network, a category recognition network, and a convolutional network and parameter output network corresponding to each face category.

[0120] The hybrid shape-driven parameter sequence refers to the parameter sequence corresponding to the face shape that has the same emotion as the multiple source face images.

[0121] In at least one embodiment of the present invention, the electronic device extracts a sequence of mixed shape driving parameters from the plurality of target face images based on a pre-trained driving parameter recognition model, including:

[0122] Based on the key point recognition network, facial key points are identified in each target face image;

[0123] Based on the category recognition network, category key points corresponding to each face category are identified from the facial key points;

[0124] Based on the location information of the key points of the category on the target face image, key point encoding information for each face category is generated;

[0125] The key point encoding information is input into the corresponding convolutional network to obtain the positional features of each face category;

[0126] By concatenating multiple positional features, a concatenated feature is obtained;

[0127] The stitched features are mapped based on the parameter output network to obtain the output driving parameters for each target face image;

[0128] The output driving parameters are spliced ​​according to the splicing order of multiple source face images in the source video sequence to obtain the hybrid shape driving parameter sequence.

[0129] The keypoint recognition network is used to compare the pixel information of each target face image with the pixel information of configured keypoints to identify the facial keypoints. The configured keypoints may include, but are not limited to, the eyeballs and the tip of the nose.

[0130] The various facial categories mentioned include, but are not limited to: left eye, right eye, nose, mouth, cheek, etc.

[0131] The category recognition network is used to match the facial key points with each facial category to identify the category key points corresponding to each facial category.

[0132] By using the key point recognition network and the category recognition network, the category key points corresponding to each face category can be accurately identified from the facial key points. Then, by using the convolutional network corresponding to each face category to extract features from the key point encoding information, the positional features of each face category can be accurately generated, thereby improving the generation accuracy of the hybrid shape driving parameter sequence.

[0133] Specifically, the electronic device performs mapping processing on the stitched features based on the parameter output network to obtain the output driving parameters for each target face image, including:

[0134] Extract the weight matrix and weight offset from the parameter output network;

[0135] Calculate the dot product of the weight matrix and the concatenated features;

[0136] The target matrix is ​​generated based on the sum of the weight offsets of each element in the dot product result.

[0137] The target matrix is ​​mapped based on the mapping table in the parameter output network to obtain the output driving parameters.

[0138] The splicing features can be adjusted using the weight matrix and the weight offset to improve the accuracy of generating the target matrix, thereby improving the mapping accuracy of the output driving parameters.

[0139] 108. Render the blended shape driving parameter sequence to obtain the video animation of the target character.

[0140] It should be emphasized that, to further ensure the privacy and security of the aforementioned video animations, they can also be stored in a blockchain node.

[0141] In at least one embodiment of the present invention, the video animation refers to a video sequence that has the same audio content and facial expressions as the source video sequence, and the character in the video animation is the target character.

[0142] In at least one embodiment of the present invention, the electronic device invokes a rendering tool to render the hybrid shape driving parameter sequence to obtain the video animation.

[0143] The rendering tool can be a tool such as 3DMAX or Maya.

[0144] The rendering tool can quickly convert the blended shape-driven parameter sequence into the video animation, thereby improving the generation efficiency of the video animation.

[0145] As can be seen from the above technical solutions, this application, through the face generator, can fuse the expression features of the source face image and the face features of the person image, thereby generating a target face image with the expression features. Furthermore, by using the face discriminator and each source face image to detect the multiple target face images, it can ensure that the target expression information in the multiple target face images is as close as possible to the source expression information, improving the generation accuracy of the multiple target face images. When the target person changes, only the person image of the target person needs to be input into the image synthesis model to directly obtain the corresponding target face image, thereby improving the generation efficiency of the video animation. Further, when all multiple target face images pass the detection of the face discriminator, this application directly extracts the mixed shape driving parameter sequence from the multiple target face images through the driving parameter recognition model, which can improve the generation efficiency of the mixed shape driving parameter sequence, thereby further improving the generation efficiency of the video animation. Furthermore, since the driving parameter recognition model is a self-supervised model, no data needs to be acquired when training the driving parameter recognition model, thus reducing the cost of data acquisition.

[0146] like Figure 3 The diagram shown is a functional block diagram of a preferred embodiment of the video generation apparatus of the present invention. The video generation apparatus 11 includes an acquisition unit 110, an extraction unit 111, a generation unit 112, a detection unit 113, a rendering unit 114, a determination unit 115, and an adjustment unit 116. The module / unit referred to in this invention refers to a series of computer-readable instruction segments that can be acquired by the processor 13 and perform a fixed function, and are stored in the memory 12. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0147] The acquisition unit 110 is used to acquire the source video sequence of the source person and the image of the target person.

[0148] The acquisition unit 110 is also used to acquire an image synthesis model, the image synthesis model including a face generator and a face discriminator, the face generator including an expression extraction network and a facial feature extraction network;

[0149] Extraction unit 111 is used to extract facial expression features of each source face image in the source video sequence according to the facial expression extraction network;

[0150] The extraction unit 111 is also used to extract facial features of the person image based on the facial feature extraction network;

[0151] The generation unit 112 is used to generate multiple target face images based on the expression features and the face features;

[0152] Detection unit 113 is used to detect the multiple target face images based on the face discriminator and each source face image;

[0153] The extraction unit 111 is further configured to extract a mixed shape driving parameter sequence from the multiple target face images based on a pre-trained driving parameter recognition model if all of the multiple target face images pass the detection of the face discriminator.

[0154] The rendering unit 114 is used to render the blended shape driving parameter sequence to obtain the video animation of the target character.

[0155] In at least one embodiment of the present invention, the face discriminator includes a discriminant extraction network and a discriminant output network, and the detection unit 113 is further configured to extract source expression information from each source face image and extract target expression information from each target face image based on the discriminant extraction network.

[0156] By comparing the target facial expression information with the source facial expression information, the difference facial expression information is obtained;

[0157] Based on the discriminant output network, the differential facial expression information is discriminated to obtain a discrimination result;

[0158] If all the discrimination results are preset results, then it is determined that all of the multiple target face images have passed the detection of the face discriminator; or

[0159] If the discrimination results are not all the preset results, then it is determined that there is a target face image among the multiple target face images that does not pass the detection of the face discriminator.

[0160] In at least one embodiment of the present invention, after detecting the plurality of target face images according to the face discriminator and each source face image, the determining unit 115 is used to determine the target face image whose discrimination result is not the preset result as an image to be processed if there is a target face image in the plurality of target face images that fails the detection of the face discriminator.

[0161] The adjustment unit 116 is used to adjust the expression of the image to be processed based on the difference expression information to obtain an adjusted image.

[0162] In at least one embodiment of the present invention, the driving parameter recognition model includes a key point recognition network, a category recognition network, and a convolutional network and parameter output network corresponding to each face category. The extraction unit 111 is also used to identify the facial key points of each target face image based on the key point recognition network.

[0163] Based on the category recognition network, category key points corresponding to each face category are identified from the facial key points;

[0164] Based on the location information of the key points of the category on the target face image, key point encoding information for each face category is generated;

[0165] The key point encoding information is input into the corresponding convolutional network to obtain the positional features of each face category;

[0166] By concatenating multiple positional features, a concatenated feature is obtained;

[0167] The stitched features are mapped based on the parameter output network to obtain the output driving parameters for each target face image;

[0168] The output driving parameters are spliced ​​according to the splicing order of multiple source face images in the source video sequence to obtain the hybrid shape driving parameter sequence.

[0169] In at least one embodiment of the present invention, the extraction unit 111 is further configured to extract the weight matrix and weight offset from the parameter output network;

[0170] Calculate the dot product of the weight matrix and the concatenated features;

[0171] The target matrix is ​​generated based on the sum of the weight offsets of each element in the dot product result.

[0172] The target matrix is ​​mapped based on the mapping table in the parameter output network to obtain the output driving parameters.

[0173] In at least one embodiment of the present invention, the expression extraction network includes multiple deconvolution layers, and the extraction unit 111 is further configured to generate an image vector based on the pixel information of each source face image.

[0174] The image vector is transformed to obtain the reconstructed vector;

[0175] Based on the multiple deconvolutional layers, the reshaped vector is upsampled multiple times to obtain the facial expression features.

[0176] In at least one embodiment of the present invention, the generation unit 112 is further configured to fuse the expression features and the facial features to obtain target features;

[0177] The target features are decoded to obtain the multiple target face images.

[0178] As can be seen from the above technical solutions, this application, through the face generator, can fuse the expression features of the source face image and the face features of the person image, thereby generating a target face image with the expression features. Then, by using the face discriminator and each source face image to detect the multiple target face images, it can ensure that the target expression information in the multiple target face images is as close as possible to the source expression information, improving the generation accuracy of the multiple target face images. When the target person changes, only the person image of the target person needs to be input into the image synthesis model to directly obtain the corresponding target face image, thereby improving the generation efficiency of the video animation. Furthermore, when all multiple target face images pass the detection of the face discriminator, this application directly extracts the mixed shape driving parameter sequence from the multiple target face images through the driving parameter recognition model, which can improve the generation efficiency of the mixed shape driving parameter sequence, thereby further improving the generation efficiency of the video animation. Furthermore, since the driving parameter recognition model is a self-supervised model, no data needs to be acquired when training the driving parameter recognition model, thus reducing the cost of data acquisition.

[0179] like Figure 4 The diagram shown is a schematic representation of the structure of an electronic device that implements the video generation method of the present invention.

[0180] In one embodiment of the present invention, the electronic device 1 includes, but is not limited to, a memory 12, a processor 13, and computer-readable instructions, such as a video generation program, stored in the memory 12 and executable on the processor 13.

[0181] Those skilled in the art will understand that the schematic diagram is merely an example of electronic device 1 and does not constitute a limitation on electronic device 1. It may include more or fewer components than shown in the diagram, or combine certain components, or different components. For example, electronic device 1 may also include input / output devices, network access devices, buses, etc.

[0182] The processor 13 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 13 is the computing core and control center of the electronic device 1, connecting various parts of the electronic device 1 through various interfaces and lines, and executing the operating system of the electronic device 1, as well as various installed application programs and program code.

[0183] For example, the computer-readable instructions can be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete the present invention. The one or more modules / units can be a series of computer-readable instruction segments capable of performing a specific function, which describe the execution process of the computer-readable instructions in the electronic device 1. For example, the computer-readable instructions can be divided into an acquisition unit 110, an extraction unit 111, a generation unit 112, a detection unit 113, a rendering unit 114, a determination unit 115, and an adjustment unit 116.

[0184] The memory 12 can be used to store the computer-readable instructions and / or modules. The processor 13 implements various functions of the electronic device 1 by running or executing the computer-readable instructions and / or modules stored in the memory 12 and calling the data stored in the memory 12. The memory 12 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. The memory 12 may include non-volatile and volatile memory, such as: hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other storage devices.

[0185] The memory 12 can be the external memory and / or internal memory of the electronic device 1. Furthermore, the memory 12 can be a physical memory, such as a memory module, a TF card (Trans-flash Card), etc.

[0186] If the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by instructing related hardware through computer-readable instructions. The computer-readable instructions can be stored in a computer-readable storage medium, and when executed by a processor, the computer-readable instructions can implement the steps of the various method embodiments described above.

[0187] The computer-readable instructions include computer-readable instruction code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer-readable instruction code, recording medium, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), and random access memory (RAM).

[0188] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed video generation, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0189] Combination Figure 1 The memory 12 in the electronic device 1 stores computer-readable instructions to implement a video generation method, and the processor 13 can execute the computer-readable instructions to implement:

[0190] Obtain the source video sequence of the source person and the image of the target person;

[0191] An image synthesis model is obtained, the image synthesis model including a face generator and a face discriminator, the face generator including an expression extraction network and a facial feature extraction network;

[0192] Based on the expression extraction network, extract the expression features of each source face image in the source video sequence;

[0193] The facial features of the person image are extracted using the facial feature extraction network.

[0194] Multiple target face images are generated based on the facial expression features and the facial features;

[0195] The multiple target face images are detected based on the face discriminator and each source face image;

[0196] If all of the multiple target face images pass the detection of the face discriminator, then based on the pre-trained driving parameter recognition model, a mixed shape driving parameter sequence is extracted from the multiple target face images;

[0197] The blended shape driving parameter sequence is rendered to obtain the video animation of the target character.

[0198] Specifically, the specific implementation method of the processor 13 for the above-mentioned computer-readable instructions can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0199] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0200] The computer-readable storage medium stores computer-readable instructions, which, when executed by the processor 13, are used to perform the following steps:

[0201] Obtain the source video sequence of the source person and the image of the target person;

[0202] An image synthesis model is obtained, the image synthesis model including a face generator and a face discriminator, the face generator including an expression extraction network and a facial feature extraction network;

[0203] Based on the expression extraction network, extract the expression features of each source face image in the source video sequence;

[0204] The facial features of the person image are extracted using the facial feature extraction network.

[0205] Multiple target face images are generated based on the facial expression features and the facial features;

[0206] The multiple target face images are detected based on the face discriminator and each source face image;

[0207] If all of the multiple target face images pass the detection of the face discriminator, then based on the pre-trained driving parameter recognition model, a mixed shape driving parameter sequence is extracted from the multiple target face images;

[0208] The blended shape driving parameter sequence is rendered to obtain the video animation of the target character.

[0209] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0210] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0211] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0212] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. The multiple units or devices described may also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0213] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A video generation method, characterized in that, The video generation method includes: Obtain the source video sequence of the source person and the image of the target person; An image synthesis model is obtained, the image synthesis model including a face generator and a face discriminator, the face generator including an expression extraction network and a facial feature extraction network, and the face discriminator including a discrimination extraction network and a discrimination output network; Based on the expression extraction network, extract the expression features of each source face image in the source video sequence; The facial features of the person image are extracted using the facial feature extraction network. Multiple target face images are generated based on the facial expression features and the facial features; Detecting multiple target face images based on the face discriminator and each source face image includes: extracting source expression information from each source face image and extracting target expression information from each target face image based on the discriminative extraction network; comparing the target expression information with the source expression information to obtain difference expression information; discriminating the difference expression information based on the discriminative output network to obtain a discrimination result; if all discrimination results are preset results, then it is determined that all multiple target face images have passed the detection of the face discriminator; or if the discrimination results are not all preset results, then it is determined that there is a target face image among the multiple target face images that has not passed the detection of the face discriminator. If all the target face images pass the detection of the face discriminator, then based on the pre-trained driving parameter recognition model, a mixed shape driving parameter sequence is extracted from the target face images. The driving parameter recognition model includes a keypoint recognition network, a category recognition network, and a convolutional network and parameter output network corresponding to each face category. Extracting the mixed shape driving parameter sequence from the target face images based on the pre-trained driving parameter recognition model includes: identifying facial keypoints in each target face image based on the keypoint recognition network; identifying category keypoints corresponding to each face category from the facial keypoints based on the category recognition network; generating keypoint encoding information for each face category based on the position information of the category keypoints on the target face image; and inputting the keypoint encoding information into the corresponding convolutional network. In the network, the positional features of each face category are obtained; multiple positional features are concatenated to obtain concatenated features; the concatenated features are mapped based on the parameter output network to obtain output driving parameters for each target face image; the output driving parameters are concatenated according to the concatenation order of multiple source face images in the source video sequence to obtain the mixed shape driving parameter sequence; wherein, the step of mapping the concatenated features based on the parameter output network to obtain the output driving parameters for each target face image includes: extracting a weight matrix and weight offsets from the parameter output network; calculating the dot product result of the weight matrix and the concatenated features; generating a target matrix based on the sum of each element in the dot product result and the weight offset; and mapping the target matrix based on the mapping table in the parameter output network to obtain the output driving parameters. The blended shape driving parameter sequence is rendered to obtain the video animation of the target character.

2. The video generation method as described in claim 1, characterized in that, After detecting the multiple target face images based on the face discriminator and each source face image, the video generation method further includes: If any of the multiple target face images fails the detection of the face discriminator, then the target face image whose discrimination result is not the preset result is determined as an image to be processed; Based on the difference in facial expression information, the facial expression of the image to be processed is adjusted to obtain an adjusted image.

3. The video generation method as described in claim 1, characterized in that, The expression extraction network includes multiple deconvolutional layers, and the step of extracting expression features for each source face image in the source video sequence based on the expression extraction network includes: An image vector is generated based on the pixel information of each source face image; The image vector is transformed to obtain the reconstructed vector; Based on the multiple deconvolutional layers, the reshaped vector is upsampled multiple times to obtain the facial expression features.

4. The video generation method as described in claim 1, characterized in that, The step of generating multiple target face images based on the expression features and the facial features includes: By fusing the facial expression features and the facial features, the target features are obtained; The target features are decoded to obtain the multiple target face images.

5. A video generation apparatus, characterized in that, The video generation device includes: The acquisition unit is used to acquire the source video sequence of the source person and the image of the target person. The acquisition unit is also used to acquire an image synthesis model, the image synthesis model including a face generator and a face discriminator, the face generator including an expression extraction network and a facial feature extraction network, and the face discriminator including a discrimination extraction network and a discrimination output network; An extraction unit is used to extract facial expression features from each source face image in the source video sequence based on the facial expression extraction network. The extraction unit is also used to extract facial features of the person image based on the facial feature extraction network; The generation unit is used to generate multiple target face images based on the expression features and the face features; The detection unit is configured to detect the plurality of target face images based on the face discriminator and each source face image, including: extracting source expression information from each source face image and extracting target expression information from each target face image based on the discriminative extraction network; comparing the target expression information with the source expression information to obtain difference expression information; discriminating the difference expression information based on the discriminative output network to obtain a discrimination result; if the discrimination results are all preset results, then it is determined that the plurality of target face images have all passed the detection of the face discriminator; or if the discrimination results are not all preset results, then it is determined that there is a target face image among the plurality of target face images that has not passed the detection of the face discriminator. The extraction unit is further configured to, if all the multiple target face images pass the detection of the face discriminator, extract a mixed shape driving parameter sequence from the multiple target face images based on a pre-trained driving parameter recognition model. The driving parameter recognition model includes a keypoint recognition network, a category recognition network, and a convolutional network and parameter output network corresponding to each face category. Extracting the mixed shape driving parameter sequence from the multiple target face images based on the pre-trained driving parameter recognition model includes: identifying facial keypoints in each target face image based on the keypoint recognition network; identifying category keypoints corresponding to each face category from the facial keypoints based on the category recognition network; generating keypoint encoding information for each face category based on the position information of the category keypoints on the target face image; and inputting the keypoint encoding information into the... In the corresponding convolutional network, positional features for each face category are obtained; multiple positional features are concatenated to obtain concatenated features; the concatenated features are mapped based on the parameter output network to obtain output driving parameters for each target face image; the output driving parameters are concatenated according to the concatenation order of multiple source face images in the source video sequence to obtain the mixed shape driving parameter sequence; wherein, the step of mapping the concatenated features based on the parameter output network to obtain output driving parameters for each target face image includes: extracting a weight matrix and weight offsets from the parameter output network; calculating the dot product result of the weight matrix and the concatenated features; generating a target matrix based on the sum of each element in the dot product result and the weight offset; and mapping the target matrix based on the mapping table in the parameter output network to obtain the output driving parameters. The rendering unit is used to render the blended shape driving parameter sequence to obtain the video animation of the target character.

6. An electronic device, characterized in that, The electronic device includes: Memory, which stores computer-readable instructions; and The processor executes computer-readable instructions stored in the memory to implement the video generation method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions that are executed by a processor in an electronic device to implement the video generation method as described in any one of claims 1 to 4.

Images