Specific character video generation method, device and equipment and medium

Abstract

The invention discloses a video generation method and device for a specific person, equipment and a medium. The method comprises the following steps: respectively obtaining prompt word text generation features and abstract face features according to video prompt words of a to-be-generated video input by a user and a face image of a target person; performing multi-round processing on the cue word text generation features and the abstract human face features through a multi-mode transformer diffusion generation model to obtain a target person related image; wherein in the multi-mode transformer diffusion generation model, feature fusion is realized through a trans-attention mechanism; and generating a specific figure video according to the target figure related image. By adopting the above technical scheme, the specific figure video can be generated only through a single figure image, the computing power consumption is low, the consumed time is short, the specific figure video can be efficiently and conveniently obtained, and the use experience of a user is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device, equipment and medium for generating a video of a specific person. Background Art

[0002] Artificial intelligence generated content (AIGC) is a method of using artificial intelligence technology to automatically generate various forms of content such as text, images, audio, and video. Although generating random face videos that match the text description has certain application value, if video content that matches the characteristics of a specified person can be generated, it will bring significant efficiency improvements, both in the industrialized process of film production and in the efficient creation of self-media content.

[0003] At present, most generative models can only generate specified face images and do not have the ability to generate dynamic videos, which limits their application scenarios. Some generative models need to rely on a large number of character images for fine-tuning training. Although they can also generate videos of specific characters, the fine-tuning training process is time-consuming and requires high computing resources. In actual applications, it leads to a poor user experience and is not conducive to efficient and convenient generation needs.

[0004] Therefore, the existing general generative models still have certain limitations in imitating specific characters, and it is difficult to fully meet the sophisticated needs in actual production and life applications. Summary of the invention

[0005] The present invention provides a method, device, equipment and medium for generating a video of a specific person, which can generate a video of a specific person with only a single image of the person, with low computing power consumption and short time consumption, and can efficiently and conveniently obtain the video of the specific person, thereby improving the user experience.

[0006] According to one aspect of the present invention, a method for generating a video of a specific person is provided, comprising:

[0007] According to the video prompt words of the video to be generated and the face image of the target person input by the user, the prompt word text generation features and the abstract face facial features are respectively obtained;

[0008] Through the multimodal transformer diffusion generation model, the prompt word text generation features and the abstract face facial features are processed multiple times to obtain the target person related image; wherein, the multimodal transformer diffusion generation model realizes feature fusion through the cross-attention mechanism;

[0009] Generate a specific person video based on the target person related images.

[0010] According to another aspect of the present invention, there is provided a device for generating a video of a specific person, comprising:

[0011] A feature extraction module is used to obtain the prompt word text generation features and the abstract face facial features according to the video prompt words of the video to be generated and the face image of the target person input by the user;

[0012] A target person related image acquisition module is used to perform multiple rounds of processing on the prompt word text generation features and abstract face facial features through a multi-modal transformer diffusion generation model to obtain target person related images; wherein, the multi-modal transformer diffusion generation model realizes feature fusion through a cross-attention mechanism;

[0013] The character video generation module is used to generate a specific character video based on the target character related images.

[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the video generation method for a specific person described in any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for generating a video of a specific person described in any embodiment of the present invention when executed.

[0019] The technical solution of the embodiment of the present invention obtains the prompt word text generation features and the abstract face facial features respectively according to the video prompt words of the video to be generated input by the user and the face image of the target person, performs multiple rounds of processing on the prompt word text generation features and the abstract face facial features through a multi-modal transformer diffusion generation model, obtains the target person related image, and generates a specific person video according to the target person related image. In this way, a complete specific person video with smooth visual effects and rich dynamic expressiveness can be obtained, which meets the user's demand for generating specific person videos, can be widely used in various video generation scenarios, quickly and efficiently provides high-quality specific person videos, and the computing power consumption of generating specific person videos is low, the time consumption is short, and the user experience is effectively improved.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 is a flowchart of a method for generating a video of a specific person provided in accordance with Embodiment 1 of the present invention;

[0023] Figure 2 is a schematic diagram of a method for acquiring facial features of an abstract human face provided according to an embodiment of the present invention;

[0024] Figure 3 1 is a schematic diagram of a feature processing flow of a multi-mode transformer diffusion generation model provided according to an embodiment of the present invention;

[0025] Figure 4 is a flowchart of another method for generating a video of a specific person provided in accordance with Embodiment 2 of the present invention;

[0026] Figure 5 is a schematic diagram of the structure of a device for generating a video of a specific person provided in accordance with Embodiment 3 of the present invention;

[0027] Figure 6 It is a schematic diagram of the structure of an electronic device for implementing the method for generating a video of a specific person according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] Embodiment 1

[0031] Figure 1 This is a flowchart of a method for generating a video of a specific person provided in the first embodiment of the present invention. This embodiment is applicable to the case of generating a dynamic video of a specific person. The method can be executed by a video generating device for a specific person. The video generating device for a specific person can be implemented in the form of hardware and / or software and can generally be configured in a computer or processor with data and image processing functions. Figure 1 As shown, the method includes:

[0032] S110, obtaining prompt word text generation features and abstract face facial features respectively according to the video prompt words of the to-be-generated video and the face image of the target person input by the user.

[0033] Optionally, the video prompt word may refer to text information provided by the user that describes the content of the video to be generated. The video prompt word may convey key elements of the video such as the theme, scene, action, emotional atmosphere, etc., and is used to guide the direction and content details of the video generation.

[0034] Optionally, when using the video generation system, the user can input video prompt words through a specific input interface, such as a text box.

[0035] Optionally, after obtaining the video prompt words, the input texts may be preprocessed, including text cleaning, vocabulary parsing, and semantic understanding, and then prompt word text generation features may be generated according to the preprocessing results.

[0036] Optionally, the target person may refer to the person in the video that the user wishes to generate, and the target person's facial image may refer to the facial photo or image of the video person. The facial image may include information such as the person's facial features, contours, expressions, etc., and may be a clear frontal, bare-headed image of the target person.

[0037] Figure 2FIG. 1 is a schematic diagram of an optional method for obtaining facial features of an abstract face. Figure 2 As shown, through the text-image pre-training model and the face image recognition model, the features of the face image can be extracted respectively, and then the extracted features are input into the multi-layer perceptron respectively, and the features output by the multi-layer perceptron are used as abstract facial features.

[0038] Optionally, the abstract facial features may include unique information, detailed features, and deep features of the face. The features output by the text-image pre-training model can help generate facial expressions and postures that conform to the semantics of the scene, while the features output by the face image recognition model can ensure that the generated face maintains the identity characteristics of the target person.

[0039] The advantage of this setting is that users only need to input a single character image, without having to rely on a large number of images for fine-tuning training, reducing dependence on computing resources and time costs, improving the user experience of related application products, and at the same time reducing the demand for computing resources in large-scale generation processes and reducing costs.

[0040] S120, through the multi-modal transformer diffusion generation model, multiple rounds of processing are performed on the prompt word text generation features and the abstract face facial features to obtain the target person related image.

[0041] Among them, feature fusion is achieved through the cross-attention mechanism in the multimodal transformer diffusion generation model.

[0042] It is understandable that the existing generative model architecture is incompatible with the transformer model. It generally relies on the U-network-based generative architecture to design the generative model, which lacks adaptability to the latest transformer architecture.

[0043] Optionally, the multimodal transformer diffusion generation model proposed in the present invention is a deep learning model based on the transformer architecture and combined with the diffusion generation principle, which is used to process and fuse data from different modalities and generate videos of specific people. The multimodal transformer diffusion generation model can capture the complex correlation between different modal data through a cross-attention mechanism, and gradually generate high-quality, high-resolution video image frames using a diffusion process. It can effectively combine the semantic information of text prompt words and the visual features of the target person's facial image to generate video content with a specific character image and consistent with the text description.

[0044] Figure 3 It is a schematic diagram of the feature processing flow of the multi-modal transformer diffusion generation model. Figure 3As shown in the figure, the initial image generation feature is the pre-loaded feature in the multimodal transformer diffusion generation model. The initial image generation feature is fused with the prompt word text generation feature to obtain the image text generation feature. Then, through the cross-attention mechanism, the abstract face facial feature is fused with the image text generation feature to generate the image text joint feature. The image text joint feature is then split to update the image generation feature and the prompt word text generation feature. Figure 3 The subsequent process is not shown later, but it is understandable that the subsequent Figure 3 The whole process is repeated many times, that is, the updated image generation features and prompt word text generation features are fused to generate image text generation features, and then the image text generation features are fused with the abstract face facial features through the cross-attention mechanism, and then the updated image generation features and prompt word text generation features are split according to the fused image text joint features, and such operations are repeated until the final end condition is met, and the image generation features obtained by the last split are used as the target person related image.

[0045] Optionally, the initial image generation feature is noise. After multiple rounds of feature processing, the final output image generation feature is the target person related image. Each feature processing will reduce the noise in the previous round of image generation features and increase the person feature details.

[0046] S130: Generate a specific person video based on the target person related images.

[0047] Optionally, the target person related image includes the appearance features of the target person and incorporates semantic information extracted from the video prompt words. The target person related image may be a frame image in a video of a specific person.

[0048] Optionally, in order to convert the target person-related images into video content with dynamic expressiveness and temporal coherence, the target person-related images can be input into a dedicated image-driven video generation model, which can be used to achieve temporal prediction and image driving.

[0049] Optionally, time series prediction can refer to inferring the image content of subsequent frames based on the logic and rules of the time series based on the input single-frame image related to the target person. For example, if the input image related to the target person shows that the person is in the starting posture of walking, the model can predict the person's leg movement, body center of gravity transfer, arm swing and other movement changes in the next few frames by analyzing the person's body structure, movement trends and common walking patterns, thereby generating a series of coherent image sequences that conform to the laws of physical motion, making the person's movements look natural and smooth, rather than abrupt image switching.

[0050] Optionally, image-driven may mean that in the process of generating a video, the input image of the target person is always used as the basis and core to maintain the consistency and accuracy of the person's appearance features. No matter what the scene or action changes, the target person's facial features, body proportions and other key appearance elements will not deviate or distort, thereby ensuring that the generated video always shows the target person performing various activities.

[0051] The technical solution of the embodiment of the present invention obtains the prompt word text generation features and the abstract face facial features respectively according to the video prompt words of the video to be generated input by the user and the face image of the target person, performs multiple rounds of processing on the prompt word text generation features and the abstract face facial features through a multi-modal transformer diffusion generation model, obtains the target person related image, and generates a specific person video according to the target person related image. In this way, a complete specific person video with smooth visual effects and rich dynamic expressiveness can be obtained, which meets the user's demand for generating specific person videos, can be widely used in various video generation scenarios, quickly and efficiently provides high-quality specific person videos, and the computing power consumption of generating specific person videos is low, the time consumption is short, and the user experience is effectively improved.

[0052] Embodiment 2

[0053] Figure 4 This is a flow chart of a method for generating a video of a specific person provided in the second embodiment of the present invention. Based on the above embodiments, this embodiment specifically illustrates the method for generating a video of a specific person.

[0054] like Figure 4 As shown, the method includes:

[0055] S210, obtaining prompt word text generation features and abstract face facial features respectively according to the video prompt words of the to-be-generated video and the face image of the target person input by the user.

[0056] Wherein, obtaining the abstract facial features may include:

[0057] Inputting the facial image into a text-image pre-training model and a facial image recognition model respectively, and obtaining a first facial feature and a second facial feature outputted by the text-image pre-training model and the facial image recognition model respectively;

[0058] Processing the first facial feature and the second facial feature through a multi-layer perceptron respectively to obtain abstract facial features output by the multi-layer perceptron;

[0059] Wherein, the multi-layer perceptron uses the GELU activation function.

[0060] Optionally, the text-image pre-training model is a deep learning model. The text-image pre-training model can learn the association pattern between text and image, and associate the object category in the image with the corresponding text description. The text-image pre-training model can be a CLIP model, which maps the text and image to the same semantic space through contrastive learning. The features extracted by the text-image pre-training model can be used as the first facial features, which can include the visual features of the face itself, and semantic association information related to the text.

[0061] Optionally, a face image recognition model is a model for recognizing faces, which can extract unique information and detailed features of the face that can represent the identity of an individual from a face image. The features extracted by the face image recognition model can be used as a second face feature, which can be used to distinguish different people and is robust to changes in posture, lighting, etc. to a certain extent.

[0062] Optionally, a multilayer perceptron can be used to perform complex nonlinear mapping on input features to obtain more abstract and advanced feature representations. Using the GELU activation function in a multilayer perceptron can take advantage of the smooth nonlinear characteristics of the GELU activation function to better fit the data. When processing facial features, the data may have complex distributions and changes. GELU can handle these changes more flexibly, allowing the model to learn more detailed feature representations.

[0063] S220, acquiring image text generation features according to the prompt word text generation features.

[0064] Wherein, obtaining the image text generation feature according to the prompt word text generation feature may include:

[0065] The initial image generation feature in the multimodal transformer diffusion generation model is obtained, and the initial image generation feature is fused with the prompt word text generation feature to generate an image text generation feature.

[0066] Among them, the initial image generation feature is a noise image.

[0067] S230, performing multiple rounds of fusion and splitting on the image text generation features and the abstract face facial features, and obtaining the target person related image according to the last splitting result.

[0068] The step of fusing and splitting the image text generation features and the abstract face features in multiple rounds may include:

[0069] Through the cross-attention mechanism, the image-text generation features and the abstract face features are integrated to obtain the image-text joint features;

[0070] Splitting the image-text joint feature, and updating the image generation feature and the prompt word text generation feature according to the splitting result;

[0071] Based on the updated image generation features and prompt word text generation features, multiple rounds of feature fusion and feature splitting operations are repeated until the end condition is met.

[0072] Optionally, the cross-attention mechanism allows the model to dynamically focus on relevant information in another modality when processing data in one modality, thereby establishing connections and interactions between data in different modalities. Specifically, the image text generation features and abstract facial features can be fused, so that the model can learn the correspondence between text descriptions and facial features of characters, thereby achieving more accurate feature fusion.

[0073] The advantage of this setting is that the cross-attention mechanism can dynamically allocate attention weights according to the content of text and image features, so that the feature information most relevant to the current task receives more attention and utilization. By fusing the features of two different modalities, text and image, it can enrich the semantic connotation and visual details of the features.

[0074] Optionally, the image generation features and the prompt word text generation features are concatenated to obtain the image-text joint features used in each round.

[0075] Optionally, the number of rounds of feature processing may be set, and when it is determined that the current execution round is the last round of feature processing, it is determined that the end condition is satisfied.

[0076] Among them, according to the last split result, obtaining the target person related image may include:

[0077] When the end condition is met, the image generation features obtained by the last round of splitting are determined as the target person related image.

[0078] S240: Input the target person-related image into the image-driven video generation model, and obtain the specific person video output by the image-driven video generation model.

[0079] The technical solution of the embodiment of the present invention obtains the prompt word text generation features and the abstract face facial features respectively according to the video prompt words of the video to be generated input by the user and the face image of the target person, performs multiple rounds of processing on the prompt word text generation features and the abstract face facial features through a multi-modal transformer diffusion generation model, obtains the target person related image, and generates a specific person video according to the target person related image. In this way, a complete specific person video with smooth visual effects and rich dynamic expressiveness can be obtained, which meets the user's demand for generating specific person videos, can be widely used in various video generation scenarios, quickly and efficiently provides high-quality specific person videos, and the computing power consumption of generating specific person videos is low, the time consumption is short, and the user experience is effectively improved.

[0080] Embodiment 3

[0081] Figure 5 This is a schematic diagram of the structure of a video generation device for a specific person provided by Embodiment 3 of the present invention. Figure 5 As shown, the device includes: a feature extraction module 310, a target person related image acquisition module 320 and a person video generation module 330.

[0082] The feature extraction module 310 is used to obtain the prompt word text generation features and the abstract face facial features respectively according to the video prompt words of the video to be generated and the face image of the target person input by the user.

[0083] The target person related image acquisition module 320 is used to perform multiple rounds of processing on the prompt word text generation features and abstract face facial features through a multimodal transformer diffusion generation model to obtain the target person related image; wherein, the multimodal transformer diffusion generation model realizes feature fusion through a cross-attention mechanism.

[0084] The character video generation module 330 is used to generate a specific character video based on the target character related image.

[0085] The technical solution of the embodiment of the present invention obtains the prompt word text generation features and the abstract face facial features respectively according to the video prompt words of the video to be generated input by the user and the face image of the target person, performs multiple rounds of processing on the prompt word text generation features and the abstract face facial features through a multi-modal transformer diffusion generation model, obtains the target person related image, and generates a specific person video according to the target person related image. In this way, a complete specific person video with smooth visual effects and rich dynamic expressiveness can be obtained, which meets the user's demand for generating specific person videos, can be widely used in various video generation scenarios, quickly and efficiently provides high-quality specific person videos, and the computing power consumption of generating specific person videos is low, the time consumption is short, and the user experience is effectively improved.

[0086] Based on the above embodiments, the feature extraction module 310 can be specifically used for:

[0087] Inputting the facial image into a text-image pre-training model and a facial image recognition model respectively, and obtaining a first facial feature and a second facial feature outputted by the text-image pre-training model and the facial image recognition model respectively;

[0088] Processing the first facial feature and the second facial feature through a multi-layer perceptron respectively to obtain abstract facial features output by the multi-layer perceptron;

[0089] Wherein, the multi-layer perceptron uses the GELU activation function.

[0090] Based on the above embodiments, the target person related image acquisition module 320 may include:

[0091] An image text feature generation unit, used to obtain image text generation features according to the prompt word text generation features;

[0092] The feature multi-round fusion unit is used to perform multi-round fusion and splitting of the image text generation features and the abstract face facial features, and obtain the target person related image according to the last splitting result.

[0093] Based on the above embodiments, the image text feature generating unit can be specifically used for:

[0094] The initial image generation feature in the multimodal transformer diffusion generation model is obtained, and the initial image generation feature is fused with the prompt word text generation feature to generate an image text generation feature.

[0095] Based on the above embodiments, the feature multi-round fusion unit can be specifically used for:

[0096] Through the cross-attention mechanism, the image-text generation features and the abstract face features are integrated to obtain the image-text joint features;

[0097] Splitting the image-text joint feature, and updating the image generation feature and the prompt word text generation feature according to the splitting result;

[0098] Based on the updated image generation features and prompt word text generation features, multiple rounds of feature fusion and feature splitting operations are repeated until the end condition is met.

[0099] On the basis of the above embodiments, the characteristic multi-round fusion unit can also be specifically used for:

[0100] When the end condition is met, the image generation features obtained by the last round of splitting are determined as the target person related image;

[0101] Wherein, the initial image generation feature is a noise image.

[0102] Based on the above embodiments, the character video generation module 330 can be specifically used for:

[0103] The target person-related image is input into the image-driven video generation model, and the specific person video output by the image-driven video generation model is obtained.

[0104] The video generation device for a specific person provided in the embodiment of the present invention can execute the video generation method for a specific person provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0105] Embodiment 4

[0106] Figure 6 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0107] like Figure 6As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0108] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0109] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as the video generation method of a specific person as described in an embodiment of the present invention. That is:

[0110] According to the video prompt words of the video to be generated and the face image of the target person input by the user, the prompt word text generation features and the abstract face facial features are respectively obtained;

[0111] Through the multimodal transformer diffusion generation model, the prompt word text generation features and the abstract face facial features are processed multiple times to obtain the target person related image; wherein, the multimodal transformer diffusion generation model realizes feature fusion through the cross-attention mechanism;

[0112] Generate a specific person video based on the target person related images.

[0113] In some embodiments, the method for generating a video of a specific person may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the method for generating a video of a specific person described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the method for generating a video of a specific person in any other appropriate manner (e.g., by means of firmware).

[0114] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0115] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0116] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0117] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0118] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0119] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0120] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0121] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating a video of a specific person, characterized in that: include: According to the video prompt words of the video to be generated and the face image of the target person input by the user, the prompt word text generation features and the abstract face facial features are respectively obtained; Through the multimodal transformer diffusion generation model, the prompt word text generation features and the abstract face facial features are processed multiple times to obtain the target person related image; wherein, the multimodal transformer diffusion generation model realizes feature fusion through the cross-attention mechanism; Generate a specific person video based on the target person related images.

2. The method according to claim 1, characterized in that Get abstract facial features, including: Inputting the facial image into a text-image pre-training model and a facial image recognition model respectively, and obtaining a first facial feature and a second facial feature outputted by the text-image pre-training model and the facial image recognition model respectively; Processing the first facial feature and the second facial feature through a multi-layer perceptron respectively to obtain abstract facial features output by the multi-layer perceptron; Wherein, the multi-layer perceptron uses the GELU activation function.

3. The method according to claim 1, characterized in that The prompt word text generation features and the abstract face facial features are processed multiple times to obtain the target person related image, including: Acquire image text generation features according to the prompt word text generation features; The image text generation features and the abstract face facial features are fused and split for multiple rounds, and the target person related image is obtained according to the last splitting result.

4. The method according to claim 3, characterized in that According to the prompt word text generation feature, the image text generation feature is obtained, including: The initial image generation feature in the multimodal transformer diffusion generation model is obtained, and the initial image generation feature is fused with the prompt word text generation feature to generate an image text generation feature.

5. The method according to claim 4, characterized in that The image text generation features and the abstract face facial features are subjected to multiple rounds of fusion and splitting, including: Through the cross-attention mechanism, the image-text generation features and the abstract face features are integrated to obtain the image-text joint features; Splitting the image-text joint feature, and updating the image generation feature and the prompt word text generation feature according to the splitting result; Based on the updated image generation features and prompt word text generation features, multiple rounds of feature fusion and feature splitting operations are repeated until the end condition is met.

6. The method according to claim 5, characterized in that According to the last split result, obtain the target person's related images, including: When the end condition is met, the image generation features obtained by the last round of splitting are determined as the target person related image; Wherein, the initial image generation feature is a noise image.

7. The method according to claim 1, characterized in that Generating a specific person video according to the target person related image, including: The target person-related image is input into the image-driven video generation model, and the specific person video output by the image-driven video generation model is obtained.

8. A video generation device for a specific person, characterized in that: include: A feature extraction module is used to obtain the prompt word text generation features and the abstract face facial features according to the video prompt words of the video to be generated and the face image of the target person input by the user; A target person related image acquisition module is used to perform multiple rounds of processing on the prompt word text generation features and abstract face facial features through a multi-modal transformer diffusion generation model to obtain target person related images; wherein, the multi-modal transformer diffusion generation model realizes feature fusion through a cross-attention mechanism; The character video generation module is used to generate a specific character video based on the target character related images.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for generating a video of a specific person according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for generating a video of a specific person according to any one of claims 1 to 7 when executed.

Images