A method, apparatus, device and medium for generating a video of a specific character

Abstract

The application discloses a specific character video generation method and device, equipment and medium. The method comprises the following steps: acquiring a prompt word text generation feature and an abstract face feature according to a video prompt word of a video to be generated and a face image of a target character input by a user; performing multi-round processing on the prompt word text generation feature and the abstract face feature through a multi-modal transformer diffusion generation model to acquire a target character related image; wherein, the multi-modal transformer diffusion generation model realizes feature fusion through a cross-attention mechanism; and generating a specific character video according to the target character related image. By using the above technical scheme, a specific character video can be generated only by using a single character image, the power consumption is low, the time consumption is short, the specific character video can be efficiently and conveniently obtained, and the use experience of the user is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a specific person video generation method, device, equipment and medium. BACKGROUND

[0002] Artificial intelligence generated content (AIGC) is a technology that uses artificial intelligence to automatically generate text, images, audio, video and other forms of content. Although generating random face videos that meet the text description has certain application value, if we can generate video content that meets the specified characteristics of a person, whether in the industrialization process of film production or in the efficient creation of self-media content, it will bring significant efficiency improvement.

[0003] At present, most generation models can only generate specified face images and do not have the ability to generate dynamic videos, resulting in limited application scenarios. Another part of the generation model needs to rely on a large number of image fine-tuning training, although it can also generate specific person videos, but the fine-tuning training process is time-consuming and has high demand for computing resources, resulting in poor user experience in actual application, which is not conducive to efficient and convenient generation needs.

[0004] Therefore, the existing general generation model still has certain limitations in imitating specific persons, and it is difficult to fully meet the fine needs in actual production and life applications. SUMMARY

[0005] The present application provides a specific person video generation method, device, equipment and medium, which can generate specific person videos only with a single person image, has low computing power consumption, short time consumption, can efficiently and conveniently obtain specific person videos, and improves user experience.

[0006] According to one aspect of the present application, a specific person video generation method is provided, comprising:

[0007] According to the video prompt word input by the user and the face image of the target person, the prompt word text generation feature and the abstract face feature are obtained respectively;

[0008] The multi-modal transformer diffusion generation model is used to process the prompt word text generation feature and the abstract face feature in multiple rounds to obtain the target person related image, wherein the multi-modal transformer diffusion generation model realizes feature fusion through a cross-attention mechanism;

[0009] According to the target person related image, a specific person video is generated.

[0010] According to another aspect of the present application, a specific person video generation device is provided, comprising:

[0011] a feature extraction module configured to obtain a prompt text generation feature and an abstract face feature according to a video prompt word of a video to be generated input by a user and a face image of a target person, respectively;

[0012] a target person related image acquisition module configured to perform multi-round processing on the prompt text generation feature and the abstract face feature by a multi-modal transformer diffusion generation model to obtain a target person related image, wherein feature fusion is achieved by a cross-attention mechanism in the multi-modal transformer diffusion generation model;

[0013] a person video generation module configured to generate a specific person video according to the target person related image.

[0014] According to another aspect of the present application, an electronic device is provided, which comprises:

[0015] at least one processor; and a memory in communication connection with the at least one processor; wherein

[0016] the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the specific person video generation method according to any one of the embodiments of the present application.

[0017] According to another aspect of the present application, a computer readable storage medium is provided, which stores computer instructions for enabling a processor to implement the specific person video generation method according to any one of the embodiments of the present application when executed by the processor.

[0018] The technical solution of the embodiments of the present application can obtain a complete specific person video with smooth visual effect and rich dynamic performance by obtaining a prompt text generation feature and an abstract face feature according to a video prompt word of a video to be generated input by a user and a face image of a target person, respectively, performing multi-round processing on the prompt text generation feature and the abstract face feature by a multi-modal transformer diffusion generation model to obtain a target person related image, and generating a specific person video according to the target person related image, thereby meeting the user's demand for generating a specific person video, and the specific person video can be widely applied to various video generation scenarios to provide high-quality specific person videos quickly and efficiently, and the power consumption for generating the specific person video is low, the time consumption is short, and the user's use experience is effectively improved.

[0019]

[0020] It is to be understood that the details set forth herein do not limit the scope of the embodiments of the application to the specific embodiments described. Rather, the scope of the embodiments of the application is to be defined by the appended claims. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort based on these drawings.

[0022] Figure 1 is a flow chart of a specific person video generation method according to the first embodiment of the present application;

[0023] Figure 2 is a schematic diagram of an abstract face feature acquisition method according to the embodiment of the present application;

[0024] Figure 3 is a feature processing flowchart of a multi-modal transformer diffusion generation model according to the embodiment of the present application;

[0025] Figure 4 is a flow chart of another specific person video generation method according to the second embodiment of the present application;

[0026] Figure 5 is a structural schematic diagram of a specific person video generation device according to the third embodiment of the present application;

[0027] Figure 6 is a structural schematic diagram of an electronic device for implementing the specific person video generation method according to the embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the technical personnel in the art better understand the present application scheme, the following will combine the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort should be within the scope of protection of the present application.

[0029] It should be noted that the terms "first", "second", and the like in the description and claims of the application and the above drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0030] Embodiment one

[0031] Figure 1 A specific person's video generation method flowchart is provided for the first embodiment of the application. The embodiment can be applied to the generation of dynamic video of a specific person. The method can be executed by a specific person's video generation device, which can be realized in the form of hardware and / or software, and can generally be configured in a computer or processor with data and image processing functions. As shown in the figure, the method comprises: Figure 1

[0032] S110, according to the video prompt word of the video to be generated input by the user and the face image of the target person, respectively obtain the prompt word text generation feature and the abstract face feature.

[0033] Optionally, the video prompt word can refer to the text information provided by the user to describe the content of the video to be generated. The video prompt word can convey the theme, scene, action, emotional atmosphere and other key elements of the video, and be used to guide the direction and content details of video generation.

[0034] Optionally, when using the video generation system, the user can input the video prompt word through a specific input interface, such as a text box.

[0035] Optionally, after obtaining the video prompt word, the input text can be preprocessed, including text cleaning, vocabulary parsing and semantic understanding, and then generating the prompt word text generation feature according to the preprocessing result.

[0036] Optionally, the target person can refer to the person in the video that the user expects to generate, and the face image of the target person can refer to the face photo or image of the video character. The face image can contain the facial feature, contour and expression of the character, and can be a clear and clean image of the target person.

[0037] Figure 2 ​It is an optional schematic diagram of an abstract face feature acquisition method. As shown in Figure 2 The text-image pre-training model and the face image recognition model can be used to extract features from the face image respectively, and then the extracted features are input into the multi-layer perceptron, and the features output by the multi-layer perceptron are used as the abstract face feature.

[0038] Optionally, the abstract face feature can include unique information, detailed features and deep features of the face. The features output by the text-image pre-training model can help generate face expressions and poses that conform to the scene semantics, and the features output by the face image recognition model can ensure that the generated face retains the identity features of the target person.

[0039] The advantage of such a setting is that only a single image of the user needs to be input, without relying on a large number of images for fine-tuning training, reducing the dependence on computing resources and time cost, improving the user experience of related application products, and reducing the demand for computing resources in large-scale generation process and cost.

[0040] In S120, the prompt word text generation feature and the abstract face feature are processed in multiple rounds by a multi-modal transformer diffusion generation model to obtain a target person related image.

[0041] In the multi-modal transformer diffusion generation model, feature fusion is achieved through a cross-attention mechanism.

[0042] It can be understood that the existing generation model architecture is not compatible with the transformer model. Generally, the generation model is designed based on a U-shaped network, which lacks adaptability to the latest transformer architecture.

[0043] Optionally, the multi-modal transformer diffusion generation model proposed in the present application is a deep learning model based on the transformer architecture and combined with the diffusion generation principle, which is used to process and fuse data from different modalities and generate videos of specific persons. The multi-modal transformer diffusion generation model can capture the complex correlation between different modal data through a cross-attention mechanism, and gradually generate high-quality and high-resolution video image frames using a diffusion process. It can effectively combine the semantic information of the text prompt and the visual features of the target person face image to generate videos with specific person images and conforming to the text description.

[0044] Figure 3 It is a feature processing flowchart of a multi-modal transformer diffusion generation model. As shown in Figure 3As shown, the initial image generation features are pre-loaded features in the multimodal transformer diffusion generation model. These initial image generation features are fused with the prompt text generation features to obtain image-text generation features. Then, through a cross-attention mechanism, abstract facial features are fused with the image-text generation features to generate joint image-text features. These joint features are then split, and the image generation features and prompt text generation features are updated. Although... Figure 3 The subsequent process was not shown, but it is understandable that what follows is a... Figure 3 The entire process is repeated multiple times. Specifically, based on the updated image generation features and the prompt text generation features, image text generation features are fused together. Then, the image text generation features are fused with the abstract facial features through a cross-attention mechanism. Based on the fused image text joint features, the updated image generation features and prompt text generation features are split to obtain the updated image generation features and prompt text generation features. This process is repeated until the final termination condition is met. The image generation features obtained from the last split are then used as the relevant image of the target person.

[0045] Optionally, the initial image generation features are noise. After multiple rounds of feature processing, the final output image generation features are the relevant images of the target person. Each feature processing reduces the noise in the previous round of image generation features and increases the details of the person's features.

[0046] S130. Generate a video of the target person based on relevant images of the target person.

[0047] Optionally, the relevant images of the target person contain the target person's physical features and incorporate semantic information extracted from video prompts. The relevant images of the target person can be a frame from a video of a specific person.

[0048] Optionally, in order to transform images related to the target person into video content with dynamic expressiveness and temporal coherence, the images related to the target person can be input into a specialized image-driven video generation model. The image-driven video generation model can be used to achieve temporal prediction and image-driven processing.

[0049] Optionally, temporal prediction can refer to inferring the image content of subsequent frames based on the logic and patterns in the time sequence of the input single-frame image of the target person. For example, if the input image of the target person shows that the person is in the initial walking posture, the model can predict the changes in the person's leg movements, the shift of the body's center of gravity, and the swinging of the arms in the next few frames by analyzing the person's body structure, movement trends, and common walking patterns. This generates a series of coherent image sequences that conform to the laws of physical movement, making the person's movements look natural and smooth, rather than abrupt image transitions.

[0050] Optionally, the image driving can refer to that, in the process of generating the video, the target person-related image is always taken as the basis and core, the consistency and accuracy of the appearance characteristics of the target person are maintained, and the key appearance elements such as the facial features and the body proportions of the target person will not be deviated or distorted under any scene and action change, so as to ensure that the target person is always performing various activities in the generated video.

[0051] The technical scheme of the embodiment of the present application can obtain complete specific person video with smooth visual effect and rich dynamic performance by acquiring the prompt word text generation feature and the abstract face feature of the target person according to the video prompt word input by the user and the face image of the target person, performing multi-round processing on the prompt word text generation feature and the abstract face feature of the target person by the multi-modal transformer diffusion generation model, acquiring the target person-related image, and generating the specific person video according to the target person-related image, thereby meeting the user's demand for generating the specific person video, and the specific person video generation method can be widely applied to various video generation scenes, and high-quality specific person video can be quickly and efficiently provided, and the power consumption of generating the specific person video is low, the time consumption is short, and the user's use experience is effectively improved.

[0052] Embodiment two

[0053] Figure 4 A flowchart of a specific person video generation method provided by the embodiment two of the present application is provided, and the specific person video generation method is specifically described based on the above-mentioned embodiment.

[0054] As Figure 4 shown, the method comprises:

[0055] S210, acquiring prompt word text generation feature and abstract face feature according to the video prompt word input by the user and the face image of the target person.

[0056] The abstract face feature can be acquired by:

[0057] inputting the face image into a text-image pre-training model and a face image recognition model respectively, and acquiring the first face feature and the second face feature output by the text-image pre-training model and the face image recognition model respectively;

[0058] processing the first face feature and the second face feature by a multi-layer perception machine respectively, and acquiring the abstract face feature output by the multi-layer perception machine;

[0059] The multi-layer perception machine uses a GELU activation function.

[0060] Optionally, the text-image pre-training model is a deep learning model that can learn the association pattern between text and image, and associate the object class in the image with the corresponding text description. The text-image pre-training model can be a CLIP model. Through contrastive learning, the text and image are mapped to the same semantic space. The features extracted by the text-image pre-training model can be used as the first face feature. The first face feature can include visual features of the face itself and semantic association information related to the text.

[0061] Optionally, the face image recognition model is a model for recognizing faces. It can extract unique information and detailed features of a person's face from a face image. The features extracted by the face image recognition model can be used as the second face feature. The second face feature can be used to distinguish different people and is robust to changes in posture and lighting to some extent.

[0062] Optionally, the multi-layer perceptron can be used to perform complex nonlinear mapping on the input features, thereby obtaining more abstract and higher-level feature representations. In the multi-layer perceptron, the GELU activation function is used. The smooth nonlinear characteristics of the GELU activation function can be used to better fit the data. When processing face features, the data may have complex distribution and changes. GELU can handle these changes more flexibly, allowing the model to learn more detailed feature representations.

[0063] S220, generating features according to the prompt word text, and obtaining image text generation features.

[0064] Wherein, according to the prompt word text generation feature, the image text generation feature can be obtained, which can include:

[0065] Obtain the initial image generation feature in the multi-modal transformer diffusion generation model, and fuse the initial image generation feature with the prompt word text generation feature to generate image text generation feature.

[0066] Wherein, the initial image generation feature is a noise image.

[0067] S230, the image text generation feature and the abstract face feature are fused and split for multiple rounds, and the target person related image is obtained according to the last split result.

[0068] Wherein, the image text generation feature and the abstract face feature are fused and split for multiple rounds, which can include:

[0069] Through the cross-attention mechanism, the image text generation feature and the abstract face feature are fused to obtain the image text joint feature.

[0070] split the image-text joint feature, and update the image generation feature and the prompt word text generation feature according to the split result;

[0071] According to the updated image generation feature and the prompt word text generation feature, the feature fusion and the feature splitting operation are repeatedly performed for multiple rounds until the end condition is met.

[0072] Optionally, the cross-attention mechanism can enable the model to dynamically focus on relevant information in another modality when processing data in one modality, thereby establishing a connection and interaction between different modal data. Specifically, the image-text generation feature and the abstract face feature can be fused to enable the model to learn the correspondence between the text description and the face feature of the character, thereby achieving more accurate feature fusion.

[0073] The advantage of this setting is that the cross-attention mechanism can dynamically allocate attention weights according to the content of the text and image features, so that the most relevant feature information for the current task is given more attention and utilized. By fusing features of two different modalities of text and image, the semantic connotation and visual details of the features can be enriched.

[0074] Optionally, the image generation feature and the prompt word text generation feature can be spliced to obtain the image-text joint feature used in each round.

[0075] Optionally, the number of rounds of feature processing can be set, and when it is determined that the current execution round is the last round of feature processing, it is determined that the end condition is met.

[0076] According to the last split result, the target character related image can be obtained, which can include:

[0077] When the end condition is met, the image generation feature obtained by the last round of splitting is determined as the target character related image.

[0078] S240, input the target character related image into the image driven video generation model, and obtain the specific character video output by the image driven video generation model.

[0079] The technical scheme of the embodiment of the present application can obtain complete specific person videos with fluent visual effects and rich dynamic performance by generating specific person videos according to the video prompt word of the video to be generated input by the user and the face image of the target person, and by respectively obtaining prompt word text generation features and abstract face features, and by performing multi-round processing on the prompt word text generation features and the abstract face features through a multi-modal transformer diffusion generation model, and by obtaining target person related images, and by generating specific person videos according to the target person related images, thereby meeting the needs of users for generating specific person videos, and the specific person videos can be widely applied to various video generation scenes, high-quality specific person videos can be quickly and efficiently provided, and the consumption of computing power for generating specific person videos is low, the time consumption is short, and the use experience of users is effectively improved.

[0080] Embodiment three

[0081] Figure 5 A structural schematic diagram of a specific person video generation device provided by the third embodiment of the present application is shown in FIG. 3. Figure 5 As shown in the figure, the device comprises a feature extraction module 310, a target person related image acquisition module 320 and a person video generation module 330.

[0082] The feature extraction module 310 is configured to respectively obtain prompt word text generation features and abstract face features according to the video prompt word of the video to be generated input by the user and the face image of the target person.

[0083] The target person related image acquisition module 320 is configured to perform multi-round processing on the prompt word text generation features and the abstract face features through a multi-modal transformer diffusion generation model to obtain target person related images, wherein the multi-modal transformer diffusion generation model realizes feature fusion through a cross-attention mechanism.

[0084] The person video generation module 330 is configured to generate specific person videos according to the target person related images.

[0085] The technical scheme of the embodiment of the present application can obtain a complete specific character video with fluent visual effect and rich dynamic performance by generating the specific character video according to the video prompt word input by the user and the face image of the target character, and can meet the demand of the user for generating the specific character video, can be widely applied to various video generation scenes, and can provide high-quality specific character videos quickly and efficiently, and the specific character video has low computing power consumption and short time consumption, and the use experience of the user is effectively improved.

[0086] On the basis of the above embodiments, the feature extraction module 310 can be specifically used for:

[0087] The face image is input into a text-image pre-training model and a face image recognition model respectively, and first face features and second face features output by the text-image pre-training model and the face image recognition model are obtained;

[0088] The first face features and the second face features are processed by a multi-layer perception machine respectively, and abstract face features output by the multi-layer perception machine are obtained;

[0089] The multi-layer perception machine uses a GELU activation function.

[0090] On the basis of the above embodiments, the target character related image acquisition module 320 can include:

[0091] An image text feature generation unit is configured to obtain image text generation features according to the prompt word text generation features;

[0092] A feature multi-round fusion unit is configured to perform multi-round fusion and splitting on the image text generation features and the abstract face features, and obtain target character related images according to the last splitting result.

[0093] On the basis of the above embodiments, the image text feature generation unit can be specifically used for:

[0094] The initial image generation features in the multi-modal transformer diffusion generation model are obtained, and the initial image generation features are fused with the prompt word text generation features to generate image text generation features.

[0095] On the basis of the above embodiments, the feature multi-round fusion unit can be specifically used for:

[0096] Through the cross-attention mechanism, the image text generation feature and the abstract human face feature are fused to obtain an image text joint feature;

[0097] The image text joint feature is split, and an image generation feature and a prompt word text generation feature are updated according to a split result;

[0098] According to the updated image generation feature and the prompt word text generation feature, the feature fusion and the feature split operation are repeatedly performed for multiple rounds until a termination condition is met.

[0099] On the basis of the above-mentioned embodiments, the feature multi-round fusion unit can also be specifically used for:

[0100] When the termination condition is met, the image generation feature obtained through the last round of splitting is determined as the target person-related image.

[0101] The initial image generation feature is a noise image.

[0102] On the basis of the above-mentioned embodiments, the person video generation module 330 can be specifically used for:

[0103] The target person-related image is input into an image-driven video generation model, and a specific person video output by the image-driven video generation model is obtained.

[0104] The specific person video generation device provided in the embodiments of the present application can execute the specific person video generation method provided in any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0105] Embodiment Four

[0106] Figure 6 A structural schematic diagram of an electronic device 10 that can be used to implement the embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (such as headsets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0107] As Figure 6As shown, the electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11, wherein the memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0108] Various components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc., an output unit 17, such as various types of displays, a speaker, etc., a storage unit 18, such as a magnetic disk, an optical disk, etc., and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0109] The processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the specific character video generation method as described in embodiments of the present application. That is:

[0110] According to the video prompt word of the video to be generated input by the user and the face image of the target character, a prompt word text generation feature and an abstract face feature are respectively acquired;

[0111] The prompt word text generation feature and the abstract face feature are processed in multiple rounds by a multi-modal transformer diffusion generation model to acquire a target character related image, wherein feature fusion is achieved in the multi-modal transformer diffusion generation model through a cross-attention mechanism;

[0112] According to the target character related image, a specific character video is generated.

[0113] In some embodiments, the specific person video generation method can be implemented as a computer program tangibly embodied in a computer readable storage medium, e.g., storage unit 18. In some embodiments, parts or all of the computer program can be loaded and / or installed onto electronic device 10 via, e.g., ROM 12 and / or communication unit 19. When the computer program is loaded onto RAM 13 and executed by processor 11, one or more steps of the specific person video generation method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the specific person video generation method by other means, e.g., with the aid of firmware.

[0114] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, specially designed application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0115] Computer programs used to implement the methods of the application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program, when executed by the processor of the computer or other programmable data processing apparatus, enables the systems and methods as claimed in the claims to be implemented. The computer program can be executed entirely on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0116] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0117] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0118] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), blockchain network, and the Internet.

[0119] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0120] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0121] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for generating a video of a specific person, characterized in that, include: Based on the video prompt words input by the user for the video to be generated and a single face image of the target person, the text generation features of the prompt words and the abstract facial features of the face are obtained respectively. The multimodal transformer diffusion generation model is used to process the text generation features of the prompt words and the abstract facial features in multiple rounds to obtain images related to the target person; wherein, the multimodal transformer diffusion generation model achieves feature fusion through a cross-attention mechanism; Generate a video of the target person based on the relevant images of the target person; Among these steps, obtaining abstract facial features includes: The single face image is input into the text-image pre-training model and the face image recognition model respectively, and the first face feature and the second face feature output by the text-image pre-training model and the face image recognition model respectively are obtained; The first facial feature and the second facial feature are processed by a multilayer perceptron to obtain the abstract facial features output by the multilayer perceptron. The multilayer perceptron uses the GELU activation function. The first facial feature includes the visual features of the face itself, as well as semantic association information related to the text; the second facial feature contains unique information about personal identity, used to distinguish different people. The text generation features of the prompt words and the abstract facial features are processed in multiple rounds to obtain images related to the target person, including: Based on the text generation features of the prompt words, obtain the image text generation features; The image text generation features and the abstract facial features are fused and split in multiple rounds, and the relevant image of the target person is obtained based on the last split result; The image text generation features and the abstract facial features are fused and split in multiple rounds, including: By fusing the image-text generation features and abstract facial features through a cross-attention mechanism, joint image-text features are obtained; The image-text joint features are split, and the image generation features and prompt word text generation features are updated based on the splitting results; Based on the updated image generation features and prompt text generation features, the feature fusion and feature splitting operations are repeated multiple times until the termination condition is met.

2. The method according to claim 1, characterized in that, Based on the prompt word text generation features, image text generation features are obtained, including: The initial image generation features in the multimodal transformer diffusion generation model are obtained, and the initial image generation features are fused with the prompt word text generation features to generate image text generation features.

3. The method according to claim 1, characterized in that, Based on the final segmentation result, obtain relevant images of the target person, including: When the termination condition is met, the image generation features obtained from the last round of splitting are determined as images related to the target person; The initial image generation feature is a noisy image.

4. The method according to claim 1, characterized in that, Based on the images related to the target person, generate a video of that specific person, including: Input images of the target person into the image-driven video generation model, and obtain the specific person video output by the image-driven video generation model.

5. A video generation device for a specific person, characterized in that, include: The feature extraction module is used to obtain the text generation features of the prompt words and the abstract facial features of the target person based on the video prompt words input by the user and a single face image of the target person. The target person-related image acquisition module is used to perform multiple rounds of processing on the prompt word text generation features and abstract facial features through a multimodal transformer diffusion generation model to acquire target person-related images; wherein, the multimodal transformer diffusion generation model achieves feature fusion through a cross-attention mechanism; The character video generation module is used to generate a specific character video based on the relevant images of the target character; The feature extraction module is specifically used for: The single face image is input into the text-image pre-training model and the face image recognition model respectively, and the first face feature and the second face feature output by the text-image pre-training model and the face image recognition model respectively are obtained; The first facial feature and the second facial feature are processed by a multilayer perceptron to obtain the abstract facial features output by the multilayer perceptron. The multilayer perceptron uses the GELU activation function. The first facial feature includes the visual features of the face itself, as well as semantic association information related to the text; the second facial feature contains unique information about personal identity, used to distinguish different people. The target person related image acquisition module includes: The image text feature generation unit is used to obtain image text generation features based on the prompt word text generation features; The feature multi-round fusion unit is used to perform multi-round fusion and splitting of the image text generation features and the abstract facial features, and obtain the relevant image of the target person based on the last splitting result; The feature multi-round fusion unit is specifically used for: By fusing the image-text generation features and abstract facial features through a cross-attention mechanism, joint image-text features are obtained; The image-text joint features are split, and the image generation features and prompt word text generation features are updated based on the splitting results; Based on the updated image generation features and prompt text generation features, the feature fusion and feature splitting operations are repeated multiple times until the termination condition is met.

6. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the video generation method for a specific person as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the video generation method for a specific person as described in any one of claims 1-4.

Images