Video processing method and apparatus, and device

Abstract

Embodiments of the present disclosure provide a video processing method and apparatus, and a device. The method comprises: acquiring a reference image and text information; determining a target action video corresponding to the text information; if first human body key point information of the reference image fails to match second human body key point information of a target image frame in the target action video, performing inpainting processing on the reference image to obtain a target reference image; and generating a target video on the basis of the target reference image, the text information, and the target action video. In this way, an electronic device can automatically determine the target action video on the basis of the text information, and a user does not need to manually upload a guiding action video; and additionally, automatic inpainting processing for the reference image can be achieved, avoiding limitations on image integrity, enriching interaction forms of the user, improving the flexibility of user interaction, and meeting diversified requirements of the user.

Description

Video processing method, device and equipment

[0001] The present disclosure claims priority to Chinese Patent Application No. 202410526115.8, filed on April 28, 2024, with the Chinese Patent Office, entitled "Video processing method, device and equipment", the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0002] The present disclosure relates to the field of computer technology, and particularly relates to a video processing method, device and equipment. BACKGROUND

[0003] In recent years, the rapid progress of Artificial Intelligence Generated Content (AIGC) technology has promoted the vigorous development of the digital industry, and the intelligent generation of images and video content has greatly improved the work efficiency of related industries. Among them, human-centered video generation has a wide range of application scenarios in daily entertainment, digital virtual humans, etc.

[0004] In related technologies, in the video processing process such as generation of a character video, a user usually needs to input a complete character image and a guided action video. This interaction mode is relatively single and has low flexibility, and cannot meet the diversified needs of users. SUMMARY

[0005] Aspects of the present disclosure provide a video processing method, device and equipment to enrich the input mode of users, improve the flexibility of user interaction, and meet the diversified needs of users.

[0006] In a first aspect, an embodiment of the present disclosure provides a video processing method, comprising:

[0007] obtaining a reference image and text information;

[0008] determining a target action video corresponding to the text information;

[0009] if first human body key point information of the reference image fails to match second human body key point information of a target image frame in the target action video, performing completion processing on the reference image to obtain a target reference image;

[0010] generating a target video according to the target reference image, the text information and the target action video.

[0011] In a possible implementation, the determining of the target action video corresponding to the text information comprises:

[0012] determine an action description text corresponding to the text information, and extract a first text feature from the action description text;

[0013] match the first text feature with a second text feature corresponding to each preset action video in a preset action video library;

[0014] determine, as the target action video, the preset action video corresponding to the second text feature matched with the first text feature.

[0015] In a possible implementation, the method further includes:

[0016] obtaining at least one preset action video;

[0017] determining, for each preset action video, text description information of the preset action video, and extracting a second text feature corresponding to the text description information;

[0018] storing the preset action video and the second text feature corresponding to the preset action video, to obtain the preset action video library.

[0019] In a possible implementation, the method further includes:

[0020] extracting the first human body key point information from the reference image;

[0021] determining a target image frame corresponding to the target action video, and extracting the second human body key point information from the target image frame;

[0022] calculating a similarity between the first human body key point information and the second human body key point information;

[0023] if the similarity is less than a preset similarity threshold, determining that the first human body key point information and the second human body key point information fail to match.

[0024] In a possible implementation, the completing processing on the reference image to obtain a target reference image includes:

[0025] obtaining a target action image corresponding to the target action video; the target action image includes an action posture corresponding to the target image frame;

[0026] performing alignment processing on the reference image and the target action image to obtain an intermediate image corresponding to the reference image;

[0027] extracting, by a target image completion model, a first image feature corresponding to the intermediate image and a second image feature corresponding to the target action image;

[0028] inputting the intermediate image, the first image feature, and the second image feature into the target image completion model to generate the target reference image.

[0029] In a possible implementation, the method further includes:

[0030] obtaining sample video data;

[0031] extracting, for each sample video in the sample video data, a first image frame and a second image frame in the sample video;

[0032] randomly cropping the first image frame to obtain a sample reference image corresponding to the first image frame;

[0033] extracting a motion pose from the second image frame, and generating a sample motion image corresponding to the second image frame according to the motion pose;

[0034] iteratively training a preset image completion model according to the sample reference image and the sample motion image to obtain the target image completion model.

[0035] In a second aspect, the embodiments of the present disclosure provide a video processing method, including:

[0036] in response to an interactive operation of a user, obtaining a reference image and text information corresponding to the interactive operation;

[0037] determining a target motion video corresponding to the text information;

[0038] if first human body key point information of the reference image fails to match second human body key point information of a target image frame in the target motion video, performing completion processing on the reference image to obtain a target reference image;

[0039] generating a target video according to the target reference image, the text information, and the target motion video, and displaying the target video.

[0040] In a possible implementation, the determining of the target motion video corresponding to the text information includes:

[0041] determining a motion description text corresponding to the text information, and extracting first text features from the motion description text;

[0042] matching the first text features with second text features corresponding to each preset motion video in a preset motion video library;

[0043] determine the preset action video corresponding to the second text feature matching the first text feature as the target action video.

[0044] In a third aspect, the embodiments of the present disclosure provide a video processing apparatus, comprising:

[0045] an acquisition module, configured to acquire a reference image and text information;

[0046] a determination module, configured to determine a target action video corresponding to the text information;

[0047] a completion module, configured to, if first human body key point information of the reference image fails to match second human body key point information of a target image frame in the target action video, perform completion processing on the reference image to obtain a target reference image;

[0048] a generation module, configured to generate a target video according to the target reference image, the text information, and the target action video.

[0049] In a possible implementation, the determination module is specifically configured to:

[0050] determine an action description text corresponding to the text information, and extract a first text feature from the action description text;

[0051] match the first text feature with a second text feature corresponding to each preset action video in a preset action video library;

[0052] determine the preset action video corresponding to the second text feature matching the first text feature as the target action video.

[0053] In a possible implementation, the apparatus is further configured to:

[0054] acquire at least one preset action video;

[0055] for each preset action video, determine text description information of the preset action video, and extract a second text feature corresponding to the text description information;

[0056] store the preset action video and the second text feature corresponding to the preset action video to obtain the preset action video library.

[0057] In a possible implementation, the apparatus is further configured to:

[0058] extract the first human body key point information from the reference image;

[0059] determine a target image frame corresponding to the target action video, and extract the second human key point information from the target image frame;

[0060] calculate a similarity between the first human key point information and the second human key point information;

[0061] if the similarity is less than a preset similarity threshold, determine that the first human key point information and the second human key point information fail to match.

[0062] In a possible implementation, the completion module is specifically configured to:

[0063] obtain a target action image corresponding to the target action video; the target action image includes an action posture corresponding to the target image frame;

[0064] align the reference image and the target action image to obtain an intermediate image corresponding to the reference image;

[0065] extract, by a target image completion model, first image features corresponding to the intermediate image and second image features corresponding to the target action image;

[0066] input the intermediate image, the first image features, and the second image features into the target image completion model to generate the target reference image.

[0067] In a possible implementation, the apparatus is further configured to:

[0068] obtain sample video data;

[0069] extract, for each sample video in the sample video data, a first image frame and a second image frame in the sample video;

[0070] perform random cropping processing on the first image frame to obtain a sample reference image corresponding to the first image frame;

[0071] extract an action posture from the second image frame, and generate a sample action image corresponding to the second image frame according to the action posture;

[0072] iteratively train a preset image completion model according to the sample reference image and the sample action image to obtain the target image completion model.

[0073] In a fourth aspect, an embodiment of the present disclosure provides a video processing apparatus, including:

[0074] an obtaining module configured to, in response to an interactive operation of a user, obtain a reference image and text information corresponding to the interactive operation;

[0075] determining a target action video corresponding to the text information;

[0076] completing the reference image to obtain a target reference image if the first human body key point information of the reference image fails to match the second human body key point information of the target image frame in the target action video;

[0077] generating a target video according to the target reference image, the text information, and the target action video, and displaying the target video.

[0078] In a possible implementation, the determining module is specifically configured to:

[0079] determine an action description text corresponding to the text information, and extract a first text feature from the action description text;

[0080] match the first text feature with a second text feature corresponding to each preset action video in a preset action video library;

[0081] determine, as the target action video, the preset action video corresponding to the second text feature matched with the first text feature.

[0082] In a fifth aspect, an electronic device is provided, and the electronic device includes a memory and a processor.

[0083] The memory stores computer-executable instructions.

[0084] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the video processing method in any one of the first aspect or the second aspect.

[0085] In a sixth aspect, a computer-readable storage medium is provided, and the computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by a processor, the computer-executable instructions are used to implement the video processing method in any one of the first aspect or the second aspect.

[0086] In a seventh aspect, a computer program product is provided, and the computer program product includes a computer program. When the computer program is executed by a processor, the computer program implements the video processing method in any one of the first aspect or the second aspect.

[0087] In the embodiments of the present disclosure, a reference image and text information are acquired, a target action video corresponding to the text information is determined, if first human body key point information of the reference image fails to match second human body key point information of a target image frame in the target action video, the reference image is completed to obtain a target reference image, and the target video is generated according to the target reference image, the text information and the target action video. In the present disclosure, the electronic device automatically determines the target action video corresponding to the text information according to the text information, and when the human body key point information in the reference image is not complete enough, the electronic device can automatically complete the reference image to obtain the target reference image, and then generate the final target video according to the target reference image, the text information and the target action video. In this way, the electronic device can automatically determine the target action video according to the text information, without the need for the user to manually upload the guide action video, and can automatically complete the reference image, avoiding the limitation on the completeness of the image, enriching the user interaction form, improving the flexibility of user interaction, and meeting the diversified needs of users. BRIEF DESCRIPTION OF DRAWINGS

[0088] The accompanying drawings, which are included to provide a further understanding of the present disclosure and constitute a part of the present disclosure, illustrate the exemplary embodiments of the present disclosure and serve to explain the present disclosure together with the descriptions. In the drawings:

[0089] FIG. 1 is a schematic diagram of an application scenario provided by an exemplary embodiment of the present disclosure;

[0090] FIG. 2 is a flowchart of a video processing method provided by an exemplary embodiment of the present disclosure;

[0091] FIG. 3 is a flowchart of another video processing method provided by an exemplary embodiment of the present disclosure;

[0092] FIG. 4 is a schematic diagram of generating a target reference image provided by an exemplary embodiment of the present disclosure;

[0093] FIG. 5 is a schematic diagram of an architecture of a video processing model provided by an embodiment of the present disclosure;

[0094] FIG. 6 is a flowchart of another video processing method provided by an exemplary embodiment of the present disclosure;

[0095] FIG. 7 is a schematic diagram of a structure of a video processing apparatus provided by an exemplary embodiment of the present disclosure;

[0096] FIG. 8 is a schematic diagram of a structure of another video processing apparatus provided by an exemplary embodiment of the present disclosure;

[0097] FIG. 9 is a schematic diagram of a structure of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0098] For the purposes of the present disclosure, technical solutions and advantages, the following will be combined with specific embodiments of the present disclosure and corresponding drawings to clearly and completely describe the technical solutions of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, not all embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present disclosure. The user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present disclosure are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0099] The following first briefly describes the professional terms involved in the present disclosure:

[0100] Variational Autoencoder (VAE) is a probabilistic model for generative model and unsupervised learning, which can learn the latent feature representation of input data.

[0101] Diffusion Models (DMs) is a model for adding and removing noise to images.

[0102] Latent Diffusion Model (LDM) is a model that uses VAE to convert images to a smaller dimensional latent space, and then performs traditional diffusion model inference and optimization.

[0103] Stable Diffusion (SD) is a text-to-image generation model based on Latent Diffusion Model.

[0104] ControlNet is a controllable generation network based on SD, which can generate corresponding images according to different control conditions.

[0105] Content Guider is a plug-in module based on SD, which can map multi-modal input (including text and image) into latent representation suitable for SD.

[0106] The motion module is an SD-based plug-in module for improving continuity between video frames.

[0107] The rapid progress of AIGC technology has driven the vigorous development of the digital industry, and the intelligent generation of image and video content has greatly improved the work efficiency of related industries. Among them, the person-centered video generation has a wide range of application scenarios in daily entertainment, digital virtual people, etc. Although the video generation technology has made great breakthroughs, there are still many difficulties in the controllable video processing of the video generation of the person-centered video.

[0108] In the related art, in the video processing process, the user usually needs to input a complete person image and a guide action video, and then the electronic device can generate a corresponding person video according to the person image and the guide action video. And the quality requirement of the person image in this process is high, such as no occlusion of limbs, clear facial features, etc. Therefore, the person generation process in the related art has a single user interaction mode and low flexibility, which cannot meet the diversified needs of users.

[0109] To solve the above problems, the present disclosure provides a video processing method, device and equipment. The electronic device automatically determines the target action video corresponding to the user according to the text information input by the user, and when the completeness of the reference image does not meet the preset condition, the electronic device can automatically complete the reference image to obtain a target reference image, and then generate a final target video according to the target reference image, the text information and the target action video. In this way, the electronic device can automatically determine the target action video required by the user according to the text information, without the user manually uploading the action video, and can automatically complete the reference image without the user uploading a complete person video, which enriches the user's interaction form, improves the flexibility of user interaction, and can meet the diversified needs of users.

[0110] FIG. 1 is a schematic diagram of an application scenario provided by an exemplary embodiment of the present disclosure. As shown in FIG. 1, in the related art, in the video processing process, the electronic device 101 usually needs to obtain a complete person image and a guide action video uploaded by the user, and then generates a person video based on the complete person image and the guide action video. This video generation method has a single user interaction form and low flexibility, which cannot meet the diversified needs of users.

[0111] In the embodiments of the present disclosure, the electronic device 101 can obtain the reference image and the text information input by the user, and according to the text information, the electronic device 101 can automatically determine the target action video required by the user. In the case that the reference image is incomplete, the electronic device 101 can perform completion processing on the reference image to obtain the completed target reference image, and then generate the target video according to the target reference image, the text information and the target action video. In this way, the embodiments of the present disclosure can realize automatic determination of the target action video, and also realize automatic completion of the reference image, without the need for the user to manually upload complete character images and action videos, and the interactive form of the user is more flexible, which can meet the diversified needs of the user.

[0112] The technical solutions shown in the present disclosure will be described in detail through specific embodiments. It should be noted that the following embodiments can exist independently, or can be combined with each other. For the same or similar content, it will not be repeated in different embodiments.

[0113] FIG. 2 is a flowchart of a video processing method according to an exemplary embodiment of the present disclosure. Referring to FIG. 2, the video processing method can include the following steps.

[0114] S201, obtaining a reference image and text information.

[0115] The execution subject of the embodiments of the present disclosure can be an electronic device, or a video processing apparatus provided in the electronic device. The video processing apparatus can be implemented by software, or by a combination of software and hardware. For ease of understanding, in the following, the execution subject is taken as an example to be an electronic device. The electronic device can be a mobile phone, a computer, a server or a cloud, and the specific type of the electronic device is not limited in the embodiments of the present disclosure.

[0116] In the embodiments of the present disclosure, the reference image can refer to an image used for video processing such as character video generation, which can include part or the whole of a character. It should be noted that the character described in the embodiments of the present disclosure is not limited to the general sense of “person”, but can also include various cartoon characters or virtual characters, etc. The text information can refer to text description information, which can include scene description text, environment description text and action description text, etc.

[0117] S202, determining a target action video corresponding to the text information.

[0118] In the embodiments of the present disclosure, the target action video can refer to a guide action video corresponding to the reference image required by the user, and the electronic device can subsequently extract a guide action sequence of the character based on the target action video, and realize the generation of the character video based on the guide action sequence. In this step, the electronic device can extract the action description text in the text information according to the text information, and then perform feature matching in the preset action video library, so as to determine the target action video corresponding to the action description text. In this way, the electronic device can automatically determine the corresponding target action video according to the text information, without the need for the user to manually upload the guide action video, thereby simplifying the input operation of the user and making the interaction mode of the user more flexible.

[0119] S203, if the first human body key point information of the reference image and the second human body key point information of the target image frame in the target action video fail to match, performing completion processing on the reference image to obtain a target reference image.

[0120] In the embodiments of the present disclosure, the first human body key point information can refer to the human body key point information corresponding to the character part in the reference image. The target image frame can refer to a specific image frame in the target action video, for example, can refer to any image frame in the target action video or the first image frame of the target action video, etc. The second human body key point information can refer to the human body key point information corresponding to the character part in the target image frame. The target reference image can refer to an image obtained by completing the processing of the reference image, which can match the guide action sequence in the target action video and can exactly retain the character attributes in the reference image, thereby improving the accuracy and rationality of video processing.

[0121] In this step, after the electronic device obtains the reference image and determines the target action video, the electronic device can further determine whether the reference image is complete. Specifically, the electronic device can determine whether the first human body key point information of the reference image matches the second human body key point information of the target image frame in the target action video. If the match is successful, the electronic device can determine that the reference image is complete and no completion processing is required, and the electronic device can directly use the reference image as the target reference image. If the match is unsuccessful, the electronic device can determine that the reference image is incomplete, and the reference image needs to be completed to obtain the target reference image. The specific completion processing of the electronic device can be realized based on an algorithm such as a reference network, and the embodiments of the present disclosure do not make any limitation in this regard.

[0122] S204, generating a target video according to the target reference image, the text information, and the target action video.

[0123] In the embodiments of the present disclosure, the target video can refer to a character video finally generated by the electronic device. After the target reference image is determined, the electronic device can generate the final target video according to the text information, the target action video and the target reference image. For example, the electronic device can input the text information, the target action video and the target reference image into a target video generation model to obtain the target video.

[0124] In the embodiments of the present disclosure, the electronic device obtains a reference image and text information, determines a target action video corresponding to the text information, performs completion processing on the reference image to obtain a target reference image if the first human body key point information of the reference image fails to match the second human body key point information of a target image frame in the target action video, and generates a target video according to the target reference image, the text information and the target action video. In the present disclosure, the electronic device automatically determines a target action video corresponding to the text information according to the text information, and when the human body key point information in the reference image is not complete enough, the electronic device can automatically perform completion processing on the reference image to obtain a target reference image, and then generate a final target video according to the target reference image, the text information and the target action video. In this way, the electronic device can automatically determine a target action video according to text information without the need for a user to manually upload a guide action video, and can automatically perform completion processing on a reference image, thereby avoiding the limitation on the completeness of the image, enriching the user interaction form, improving the flexibility of user interaction, and meeting the diversified needs of users.

[0125] On the basis of the above embodiments, FIG. 3 is a flowchart of another video processing method provided by an exemplary embodiment of the present disclosure. Referring to FIG. 3, the video processing method can include:

[0126] S301, obtaining a reference image and text information.

[0127] S302, determining an action description text corresponding to the text information, and extracting first text features from the action description text.

[0128] In the embodiments of the present disclosure, the action description text can refer to the description text related to the action in the text information, such as an action keyword sentence and the like. The first text feature can refer to the text feature corresponding to the action description text. Specifically, after the electronic device obtains the text information, the action description text included in the text information can be extracted through a word segmentation processing and a keyword extraction algorithm. Then, the action description text is identified and detected through a text feature extraction algorithm, and the first text feature corresponding to the action description text is determined. The text feature extraction algorithm can be a contrastive language-image pre-training (CLIP) model or the like, and can also be other algorithms, which are not limited in the embodiments of the present disclosure.

[0129] In S303, the first text feature is matched with the second text features corresponding to each preset action video in the preset action video library.

[0130] In S304, the preset action video corresponding to the second text feature matched with the first text feature is determined as the target action video.

[0131] In the embodiments of the present disclosure, the preset action video library can refer to a preset action video database, which includes a plurality of preset action videos and the second text features corresponding to each preset action video. After the electronic device determines the first text feature corresponding to the action description text of the user, the first text feature can be compared and matched with each second text feature in the preset action database, which can be similarity calculation or vector retrieval, and then the electronic device can determine the preset action video corresponding to the second text feature matched with the first text feature as the target action video. For example, the electronic device can determine the preset action video corresponding to the second text feature with a similarity greater than a preset similarity threshold to the first text feature as the target action video, and the preset similarity threshold can be 0.7, 0.8 or 0.9, and the like.

[0132] In the embodiments of the present disclosure, the electronic device extracts the action description text from the text information input by the user, and then determines the target action video corresponding to the action description text in the preset action video database through the text feature matching mode, so as to automatically and accurately determine the target action video, without the need for the user to manually upload the video, thereby simplifying the operation of the user and making the interactive mode of the user more flexible.

[0133] In a possible implementation, the preset action video library can be created through the following steps (1) to (3):

[0134] (1) Obtain at least one preset action video.

[0135] (2) For each preset action video, determine the text description information of the preset action video, and extract the second text feature corresponding to the text description information.

[0136] (3) Store the preset action video and the second text feature corresponding to the preset action video to obtain a preset action video library.

[0137] In the embodiments of the present disclosure, the electronic device can obtain at least one preset action video, and then for each preset action video, determine the text description information (caption) corresponding to the preset action video through a video conversation model (such as Video Chat 2, etc.). After that, the electronic device can extract the description of the action from the text description information, and extract the text embedding feature corresponding to the action description in the text description information through a text feature extraction model, such as a CLIP model, etc., to obtain the second text feature corresponding to the preset action video. After that, the electronic device can store the preset action video and the second text feature corresponding to the preset action video in the database to obtain a preset action video library. Of course, in order to facilitate searching and retrieval, the electronic device can number each preset action video in the preset action video library, and determine and store the video identity (Identity document, ID) of each preset action video.

[0138] On this basis, when the user inputs text information, the electronic device can use the same text feature extraction model to extract the first text feature of the action description text in the text information, and then determine the video identity corresponding to the second text feature matching the first text feature from the preset action video library through vector retrieval matching or similarity calculation algorithm, etc. Then, the electronic device can determine the preset action video corresponding to the video identity as the target action video matching the user action description text, and subsequently the electronic device can extract the guide action sequence according to the target action video to realize accurate generation of the character video.

[0139] It should be noted that the user can also interact by directly uploading the target action video, and the electronic device can call a deep pose estimation (DWpose) algorithm to extract the human skeleton pose of each frame as the guide action sequence for video processing, which can make the flexibility of video processing interaction higher.

[0140] In a possible implementation, whether the first human key point information of the reference image matches the second human key point information of the target image frame in the target action video can be determined through the following steps (4) to (6):

[0141] (4), extracting first human key point information from the reference image; determining a target image frame corresponding to the target action video, and extracting second human key point information from the target image frame.

[0142] (5), calculating the similarity between the first human key point information and the second human key point information.

[0143] (6), if the similarity is less than a preset similarity threshold, it is determined that the first human key point information and the second human key point information fail to match.

[0144] In the embodiments of the present disclosure, the similarity can refer to the similarity between the first human key point information and the second human key point information, which can be determined using Euclidean distance, cosine similarity or Manhattan distance. The preset similarity threshold can refer to the critical threshold of the preset similarity, which can be 0.8 or 0.9, etc., and the embodiments of the present disclosure are not limited thereto.

[0145] Specifically, the electronic device can extract the first human key point information in the reference image, determine the target image frame in the target action video and extract the second human key point information corresponding to the target image frame, then the electronic device can calculate the similarity of the first human key point information and the second human key point information, and determine whether they match according to the size relationship between the similarity and the preset similarity threshold. If the similarity is greater than or equal to the preset similarity threshold, the electronic device can determine that the first human key point information and the second human key point information match, and the reference image is complete; if the similarity is less than the preset similarity threshold, the electronic device can determine that the first human key point information and the second human key point information do not match, and the reference image is not complete. In this way, the electronic device determines whether the reference image is complete by calculating the similarity of the human key point information in the reference image and the target image frame in the target action video, which can realize accurate judgment of the completeness of the reference image.

[0146] S305, if the first human key point information of the reference image and the second human key point information of the target image frame in the target action video fail to match, a target action image corresponding to the target action video is obtained; the target action image includes the action posture corresponding to the target image frame.

[0147] S306, aligning the reference image and the target action image to obtain an intermediate image corresponding to the reference image.

[0148] In the embodiments of the present disclosure, the target action image can refer to the human posture image in the target image frame, which can include action postures composed of human key points and skeletons, etc. The intermediate image can refer to the image obtained by aligning and simply filling the reference image and the target action image.

[0149] Specifically, the electronic device can first obtain a target image frame corresponding to the target action video and determine a target action image corresponding to the target image frame when determining that the completeness of the reference image does not meet the preset condition. Then, the electronic device can perform alignment processing on the reference image and the target action image to obtain an intermediate image with a resolution consistent with that of the target action image, and the non-aligned region of the intermediate image can be simply filled with a black mask.

[0150] S307, extracting first image features corresponding to the intermediate image and second image features corresponding to the target action image through the target image completion model.

[0151] S308, inputting the intermediate image, the first image features and the second image features into the target image completion model to generate a target reference image.

[0152] In the embodiments of the present disclosure, the target image completion model can refer to an image feature extraction model, which can include three parts. The first part is a referenceNet, which is used to extract first image features of the intermediate image. The second part is a pose-guider, which is used to extract second image features of the target action image. The third part is a deep learning model U-net, which is used for image feature learning and image completion processing. The first image features can refer to fine-grained features of the effective region (i.e., the aligned region) of the intermediate image. The second image features can refer to latent features of the target action image.

[0153] Specifically, after obtaining the intermediate image corresponding to the reference image, the electronic device can extract the first image features of the intermediate image and the second image features of the target action image through the target image completion model, and then generate a target reference image based on the first image features and the second image features. In this way, the electronic device can automatically complete the reference image in the case of incomplete reference image, can provide more rich portrait semantic information, can ensure that the video processing process fits the character attributes in the reference image, improves the accuracy of video generation, and the user interaction mode is more flexible.

[0154] In one possible implementation, the target image completion model can be trained through the following steps (7) to (10):

[0155] (7) obtaining sample video data.

[0156] (8) For each sample video in the sample video data, extract a first image frame and a second image frame in the sample video.

[0157] In the embodiments of the present disclosure, the sample video data can refer to training data used for model training, which can include a large number of sample videos, and the sample video can specifically refer to a high-quality video clip with single shot, same character, same clothes, no special effects, and no complex handheld objects. The first image frame and the second image frame can refer to two image frames at different times in the sample video. The subsequent electronic device can perform different processing on the first image frame and the second image frame, facilitating subsequent iterative training.

[0158] (9) Randomly crop the first image frame to obtain a sample reference image corresponding to the first image frame.

[0159] (10) Extract an action pose from the second image frame, and generate a sample action image corresponding to the second image frame according to the action pose.

[0160] (11) According to the sample reference image and the sample action image, iteratively train a preset image completion model to obtain a target image completion model.

[0161] In the embodiments of the present disclosure, the sample reference image can refer to an image including a mask filling area after the cropping processing. The sample action image can refer to an image including an action pose. The preset image completion model can refer to an image completion model in an initial state.

[0162] Specifically, after determining the first image frame and the second image frame, the electronic device can perform cropping processing on the first image frame, and fill the uncropped area with a black mask to obtain a sample reference image corresponding to the first image frame. Meanwhile, the electronic device can extract an action pose in the second image frame to generate a sample action image corresponding to the second image frame. The electronic device can adjust the resolution of the sample reference image (for example, scale to 768 pixels*768 pixels, etc.), so that the resolution of the sample reference image is consistent with that of the sample action image, improving the accuracy of model training. Then, the electronic device can iteratively train the preset image completion model according to the sample reference image and the sample action image, and finally obtain a target image completion model.

[0163] Exemplarily, taking the preset image completion model including three parts of U-net, referenceNet and pose-guider as an example, the electronic device needs to train the three parts at the same time, wherein the weights of the sd1.5 model are used as the initialization of U-net and referenceNet, specifically, four cards A100, 100K iterations, batch size of 6 can be used, at the same time, the AdamW optimizer can be used, the learning rate is set to 1e-5, and the weight decay is set to 1e-2. Of course, the training method of the preset image completion model can also adopt other forms, which can be flexibly set based on actual needs, and the embodiments of the present disclosure do not limit this.

[0164] Exemplarily, FIG. 4 is a schematic diagram of generating a target reference image provided by an exemplary embodiment of the present disclosure. As shown in FIG. 4, the target image completion model includes three parts of U-type network U-net, feature extraction network referenceNet and action guide pose-guider. The input of the target image completion model is a reference image and a target action image. Specifically, the target image completion model first aligns the reference image and the target action image to generate an intermediate image with a resolution consistent with the target action image. The non-aligned region in the intermediate image is filled with a mask, and then the intermediate image is input into the referenceNet to obtain the first image feature of the effective region and inject it into the U-net; the pose-guider is used to obtain the second image feature of the target action image and input it into the U-net, and finally the U-net is used to combine these features to generate a target reference image based on the intermediate image.

[0165] In this way, when the first human body key point information of the reference image and the second human body key point information of the target image frame in the target action video fail to match, the target image completion model can automatically complete and generate the reference image guided by the target action video, and the final result generates a reasonable expansion region on the basis of the original input part, for example, generates the missing lower body part image based on the reference image of the upper body of the person, to obtain a complete target reference image.

[0166] S309, generating a target video according to the target reference image, the text information and the target action video.

[0167] In the embodiments of the present disclosure, after the target reference image is determined, the electronic device can input the target reference image, the text information and the target action video into a target video generation model to generate a final target video. The target video generation model can be a pre-trained video generation model, and the input of the target video generation model is the target reference image, the text information and the target action video, and the output is the target video.

[0168] Specifically, the target video generation model can include a noise video control network (Video ControlNet), a denoising U-Net (Denoising U-Net), and a content guider (Content Guider). The content guider can be used to encode text information, face images, and character costume images into latent embedding expressions, respectively. Meanwhile, the target video generation model encodes the character action sequence of the target action video into a latent feature expression through the video control network, and finally generates a target video based on the text features, image features, and action features through the denoising U-Net.

[0169] In this way, the target video generation model can not only make the generated video conform to the input face and costume, but also conform to the background described in the text, achieving a multi-modal controllable video generation effect and improving the diversity of video generation. On the other hand, the generated video can also make the character actions conform to the input video actions, improving the video fluency and action diversity, while supporting full-body and half-body action input, and the consistency of the character expressions with the input action video is high. In addition, the target video generation model uses the method of inserting a time sequence module in the denoising U-Net to further improve the continuity and motion realism of the generated video actions.

[0170] Exemplarily, FIG. 5 is an architecture schematic diagram of a target video generation model provided by an embodiment of the present disclosure. As shown in FIG. 5, the target video generation model includes a video control network, a denoising U-Net, and a content guider. Specifically, similar to the control network (ControlNet) in the image diffusion model, the video control network extracts the features of the input action sequence and adds them to the intermediate features of the denoising U-Net to pass control information. Additionally, the video control network Video ControlNet adds a time domain module (motion module) compared to ControlNet to alleviate the time domain jitter of the control condition. The time domain module adopts the structure of AnimateDiff, which extends the original 24-frame model to a 64-frame model, and is initialized from “mm_sd_v15.ckpt” for training. Similarly, the main network Denoising U-Net also embeds a time domain module to ensure inter-frame consistency.

[0171] The content guide can make the generated character video consistent with the input reference image. Similar to the IP-adapter, the information of the reference image is introduced through Cross Attention. The difference is that the input reference image is preprocessed in the embodiment of the disclosure to extract the face, and a face-related model (such as Arcface, etc.) is used to extract features. The whole image still uses the OpenCLIP ViT-H14 model for encoding. Specifically, the content guide can obtain text features through a text encoder for text information, and can obtain image features through an image encoder for reference images, and then obtain text-image features through vector connection (Concat) and vector embedding processing (Content Embaddings). In this way, the content guide can decouple the face and the character costume of the reference image, and the character costume can be used more flexibly as an optional form to use the information of the reference image. The denoising U-Net obtains random noise, and according to the text-image features sent by the content guide and the action sequence features input by the video control network, iteratively denoises to obtain video encoding, and then decodes the video encoding through the VAE decoder to obtain the target video.

[0172] The target video generation model based on the diffusion model in the embodiment of the disclosure can be used to generate high-quality custom character videos. In particular, users can control the content in the generated video through text information. For example, through different text information, users can randomly switch scenes and costumes for the girl dancing action in FIG. 5, with higher freedom. In this way, the user can input a reference image and text information, and the electronic device can automatically determine a target action video corresponding to the text information based on the reference image and the text information, and generate a target video according to the reference image, the text information and the target action video.

[0173] In addition, in the data collection and processing process, the electronic device can obtain a large number of (for example, 1000) high-quality single-person videos for preprocessing to obtain training videos. The preprocessing process can include manual annotation and extraction of single-take, same character, same clothes, no special effects, and no complex handheld video clips. Finally, about 6000 video clips are obtained, each with a duration of 8 to 10 seconds. The electronic device can use Minigpt-v2 to describe the key frames in the middle of the video. Specifically, the "grounding" mode of this method can be used. The image description of the key frame can basically describe the video content except for specific actions, and the action aspect of the target video generation model relies on the action sequence rather than the text description.

[0174] During the training process of the target video generation model, the content guide serves as a module independent of the base diffusion model and can be generalized to other user-defined diffusion models. Specifically, portrait data can be filtered from the LAION-2B dataset according to face detection results, and images without faces are filtered out. During the training process, the images are randomly cropped and scaled to 512x512. The module requires 8 V100 computing cards, 100K iterations, and a batch size of 128. The AdamW optimizer is used, the learning rate is set to 1e-4, and the weight decay is set to 1e-2.

[0175] During the long sequence pre-training process, pre-warming training can be performed, the time domain module of Animatediff is extended from 16 frames to 64 frames using 5000 videos of the WebVid-10M validation set. During this stage of training, the parameters of the time domain module of the denoising U-Net are variable, and the other parts of the denoising U-Net are fixed. This stage does not involve ControlNet and the content guide, and the training is decoupled. The learning rate is set to 1e-4, and the training resolution is set to 256x256 (scaled and center cropped). The iteration is 10K, and the batch size is 1.

[0176] During the video control network training process, after the long sequence pre-training, the video control network and the denoising U-Net are jointly trained, the denoising U-Net is fixed, and the video control network is trained. The time domain module weight in the video control network is initialized by the long sequence pre-training. In this stage, the collected 6K character dance video data is used to train the network. The content guide does not participate in the training in this stage. The character pose (extracted using DWPose) or depth map (extracted using ZoeDepth) is extracted from the input as the input of the video control network. The learning rate is set to 1e-4, and the training resolution is set to 352x352. The iteration is 25K, and the batch size is 1.

[0177] In addition, in order to better generate character expressions, the time domain module in the denoising U-Net and the video control network are further combined during the training process, and are fine-tuned on the collected 6K character dance video data. In this stage, the video control network and the U-Net module in the denoising U-Net are fixed, and only the time domain module in the denoising U-Net network is fine-tuned. The learning rate is set to 1e-5, and the training resolution is set to 512x512. The iteration is 20K, and the batch size is 1. Of course, it should be noted that the training process of the target video generation model can also use other ways, which can be flexibly set based on actual needs, and the embodiments of the present disclosure do not limit this.

[0178] On the basis of the above-mentioned embodiments, FIG. 6 is a flowchart of another video processing method provided by an exemplary embodiment of the present disclosure. As shown in FIG. 6, the video processing method can include:

[0179] S601, in response to the interactive operation of the user, obtaining a reference image and text information corresponding to the interactive operation.

[0180] In the embodiments of the present disclosure, the interactive operation can refer to the interactive operation of the user to the electronic device, and specifically can refer to the input operation. Specifically, in the video generation scene such as intelligent question answering, the user can perform an input operation in the electronic device when generating a character video, input a reference image and text information. The electronic device can obtain the reference image and text information corresponding to the input operation in response to the input operation of the user, and can subsequently automatically generate a target video based on the reference image and text information.

[0181] S602, determining a target action video corresponding to the text information.

[0182] S603, if the first human body key point information of the reference image fails to match the second human body key point information of the target image frame in the target action video, performing completion processing on the reference image to obtain a target reference image.

[0183] S604, generating and displaying a target video according to the target reference image, the text information and the target action video.

[0184] In the embodiments of the present disclosure, the electronic device obtains the text information and the reference image input by the user, automatically determines the target action video corresponding to the user, and when the first human body key point information of the reference image fails to match the second human body key point information of the target image frame in the target action video, the electronic device can automatically perform completion processing on the reference image to obtain a target reference image, and then can generate a final target video according to the target reference image, the text information and the target action video, and can display the target video. In this way, the user does not need to manually upload the action video and the complete character image, which enriches the interactive form of the user, improves the flexibility of the user interaction, and can meet the diversified needs of the user.

[0185] It should be noted that the video processing method in the embodiments of the present disclosure can be applied to various video generation applications and is suitable for multiple scenarios, such as an automatic question and answer scenario, an intelligent robot scenario, and a digital virtual character scenario, and the like. For example, in the automatic question and answer scenario, the electronic device can display a dialogue interface in the display screen. The user can input the reference image and the text information in the dialogue interface through touch operation or voice input and the like. The electronic device can obtain the reference image and the text information in response to the interaction operation, and further determine the target action video. In addition, the electronic device can automatically complete the reference image when the reference image is incomplete to obtain the target reference image. Then, the electronic device can input the target reference image, the text information, and the target action video into the target video generation model to obtain the target video and display the target video in the dialogue interface, thereby realizing automatic generation of the video and meeting the actual needs of the user.

[0186] In a possible implementation, the determination of the target action video in step S602 can be implemented in the following manner:

[0187] determining an action description text corresponding to the text information, and extracting a first text feature from the action description text; matching the first text feature with a second text feature corresponding to each preset action video in the preset action video library; and determining, as the target action video, a preset action video corresponding to the second text feature matched with the first text feature.

[0188] The specific implementation of the above steps can refer to the foregoing embodiments, and will not be described here again in the embodiments of the present disclosure.

[0189] It should be understood that in various embodiments of the present disclosure, the size of the serial number of each process does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure.

[0190] FIG. 7 is a structural schematic diagram of a video processing apparatus provided by an exemplary embodiment of the present disclosure. Referring to FIG. 7, the video processing apparatus 70 comprises:

[0191] The acquisition module 71 is configured to acquire the reference image and the text information.

[0192] The determination module 72 is configured to determine a target action video corresponding to the text information.

[0193] The completion module 73 is configured to, if the first human body key point information of the reference image fails to match the second human body key point information of the target image frame in the target action video, perform completion processing on the reference image to obtain a target reference image.

[0194] The generating module 74 is configured to generate the target video according to the target reference image, the text information, and the target action video.

[0195] In a possible implementation, the determining module 72 is specifically configured to:

[0196] determine an action description text corresponding to the text information, and extract a first text feature from the action description text;

[0197] match the first text feature with a second text feature corresponding to each preset action video in the preset action video library;

[0198] determine, as the target action video, a preset action video corresponding to the second text feature matched with the first text feature.

[0199] In a possible implementation, the apparatus 70 is further configured to:

[0200] obtain at least one preset action video;

[0201] for each preset action video, determine text description information of the preset action video, and extract a second text feature corresponding to the text description information;

[0202] store the preset action video and the second text feature corresponding to the preset action video, to obtain a preset action video library.

[0203] In a possible implementation, the apparatus 70 is further configured to:

[0204] extract first human body key point information from the reference image;

[0205] determine a target image frame corresponding to the target action video, and extract second human body key point information from the target image frame;

[0206] calculate a similarity between the first human body key point information and the second human body key point information;

[0207] if the similarity is less than a preset similarity threshold, determine that the first human body key point information and the second human body key point information fail to match.

[0208] In a possible implementation, the supplementing module 73 is specifically configured to:

[0209] obtain a target action image corresponding to the target action video; the target action image includes an action posture corresponding to the target image frame;

[0210] align the reference image and the target action image to obtain an intermediate image corresponding to the reference image;

[0211] The target image completion model is used to extract the first image feature corresponding to the intermediate image and extract the second image feature corresponding to the target action image.

[0212] The intermediate image, the first image feature, and the second image feature are input into the target image completion model to generate a target reference image.

[0213] In a possible implementation, the apparatus 70 is further configured to:

[0214] Obtain sample video data.

[0215] For each sample video in the sample video data, extract a first image frame and a second image frame in the sample video.

[0216] Randomly crop the first image frame to obtain a sample reference image corresponding to the first image frame.

[0217] Extract an action pose from the second image frame, and generate a sample action image corresponding to the second image frame according to the action pose.

[0218] Iteratively train a preset image completion model according to the sample reference image and the sample action image to obtain a target image completion model.

[0219] The video processing apparatus 70 provided in the embodiments of the present disclosure can perform the technical solutions shown in the method embodiments, and the implementation principles and beneficial effects are similar, which will not be described here in detail.

[0220] FIG. 8 is a structural schematic diagram of a video processing apparatus according to an example embodiment of the present disclosure. As shown in FIG. 8, the video processing apparatus 80 includes:

[0221] The obtaining module 81 is configured to obtain a reference image and text information corresponding to an interactive operation of a user in response to the interactive operation of the user.

[0222] The determining module 82 is configured to determine a target action video corresponding to the text information.

[0223] The completion module 83 is configured to, if the first human body key point information of the reference image fails to match the second human body key point information of a target image frame in the target action video, perform completion processing on the reference image to obtain a target reference image.

[0224] The generating module 84 is configured to generate and display a target video according to the target reference image, the text information, and the target action video.

[0225] In a possible implementation, the determining module 82 is specifically configured to:

[0226] determine an action description text corresponding to the text information, and extract a first text feature from the action description text;

[0227] match the first text feature with a second text feature corresponding to each preset action video in a preset action video library;

[0228] determine, as a target action video, a preset action video corresponding to the second text feature matched with the first text feature.

[0229] The video processing apparatus 80 provided by the embodiments of the present disclosure can execute the technical solutions shown in the above method embodiments, and the implementation principles and beneficial effects are similar, which will not be repeated here.

[0230] FIG. 9 is a structural schematic diagram of an electronic device provided by an exemplary embodiment of the present disclosure. As shown in FIG. 9, the electronic device 90 can include a processor 91 and a memory 92. Exemplarily, the processor 91, the memory 92, and each part are connected with each other through a bus 93.

[0231] The memory 92 stores computer execution instructions;

[0232] The processor 91 executes the computer execution instructions stored in the memory 92, so that the processor 91 executes the video processing method shown in the above method embodiments.

[0233] Correspondingly, the embodiments of the present disclosure provide a computer readable storage medium, and the computer readable storage medium stores computer execution instructions. When the computer execution instructions are executed by a processor, the computer execution instructions are used to implement the video processing method of the above method embodiments.

[0234] Correspondingly, the embodiments of the present disclosure can also provide a computer program product, which includes a computer program. When the computer program is executed by a processor, the computer program can implement the video processing method shown in the above method embodiments.

[0235] Those skilled in the art should understand that the embodiments of the present disclosure can be provided as a method, a system, or a computer program product. Therefore, the present disclosure can adopt a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present disclosure can adopt a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes.

[0236] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks.

[0237] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks.

[0238] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks.

[0239] In one typical configuration, the computing device includes one or more processors, input / output interfaces, network interfaces, and memory.

[0240] The memory can include non-persistent memory and / or persistent memory, such as flash memory, read-only memory (ROM), and / or volatile or non-volatile random access memory (RAM), among others. The memory is an example of computer readable media.

[0241] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0242] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not only include those elements, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0243] The above only describes the embodiments of the present disclosure and is not intended to limit the present disclosure. The present disclosure can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present disclosure shall be included in the scope of claims of the present disclosure.

Claims

1. A method of video processing, the method comprising: The method comprises: obtaining a reference image and text information; determining a target action video corresponding to the text information; if first human body key point information of the reference image fails to match second human body key point information of a target image frame in the target action video, performing completion processing on the reference image to obtain a target reference image; generating a target video according to the target reference image, the text information and the target action video.

2. The method of claim 1, wherein, The determination of the target action video corresponding to the text information comprises: determining an action description text corresponding to the text information, and extracting first text features from the action description text; matching the first text features with second text features corresponding to each preset action video in a preset action video library; determining, as the target action video, the preset action video corresponding to the second text features matched with the first text features.

3. The method of claim 2, wherein, The method further comprises: obtaining at least one preset action video; for each preset action video, determining text description information of the preset action video, and extracting second text features corresponding to the text description information; storing the preset action video and the second text features corresponding to the preset action video to obtain the preset action video library.

4. The method according to any one of claims 1 to 3, characterized in that, The method further comprises: extracting the first human body key point information from the reference image; determining a target image frame corresponding to the target action video, and extracting the second human body key point information from the target image frame; calculating a similarity between the first human body key point information and the second human body key point information; if the similarity is less than a preset similarity threshold, determining that the first human body key point information fails to match the second human body key point information.

5. The method according to any one of claims 1 to 4, characterized in that, The completion processing on the reference image to obtain a target reference image comprises: obtaining a target action image corresponding to the target action video; the target action image comprises an action posture corresponding to the target image frame; aligning the reference image and the target action image to obtain an intermediate image corresponding to the reference image; extracting first image features corresponding to the intermediate image and second image features corresponding to the target action image through a target image completion model; inputting the intermediate image, the first image features and the second image features into the target image completion model to generate the target reference image.

6. The method of claim 5, wherein, The method further comprises: obtaining sample video data; for each sample video in the sample video data, extracting a first image frame and a second image frame in the sample video; randomly cropping the first image frame to obtain a sample reference image corresponding to the first image frame; extracting an action posture from the second image frame, and generating a sample action image corresponding to the second image frame according to the action posture; iteratively training a preset image completion model according to the sample reference image and the sample action image to obtain the target image completion model.

7. A method for video processing, comprising: The method comprises: in response to an interactive operation of a user, obtaining a reference image and text information corresponding to the interactive operation; determine a target action video corresponding to the text information; if the first human body key point information of the reference image fails to match the second human body key point information of the target image frame in the target action video, perform completion processing on the reference image to obtain a target reference image; generate a target video according to the target reference image, the text information, and the target action video, and display the target video.

8. The method of claim 7, wherein, The determining of the target action video corresponding to the text information comprises: determining an action description text corresponding to the text information, and extracting first text features from the action description text; matching the first text features with second text features corresponding to each preset action video in a preset action video library; determining, as the target action video, the preset action video corresponding to the second text features matched with the first text features.

9. A video processing apparatus, comprising: comprise: an acquisition module configured to acquire a reference image and text information; a determination module configured to determine a target action video corresponding to the text information; a completion module configured to, if the first human body key point information of the reference image fails to match the second human body key point information of a target image frame in the target action video, perform completion processing on the reference image to obtain a target reference image; a generation module configured to generate a target video according to the target reference image, the text information, and the target action video.

10. An electronic device, comprising: comprise: a memory and a processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, so that the processor executes the video processing method according to any one of claims 1 to 8.

11. A computer readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and when the computer execution instructions are executed by the processor, the video processing method according to any one of claims 1 to 8 is implemented.

12. A computer program product, characterised in that, The computer program is executed by the computer to implement the video processing method according to any one of claims 1 to 8.

Images