Animation video generation method and device based on key frame, equipment and storage medium

Abstract

The invention belongs to the technical field of video generation, and discloses an animation video generation method and device based on key frames, equipment and a storage medium, and the method comprises the steps: carrying out the key frame detection of a video frame sequence, obtaining key frame data, and generating intermediate frame data according to the key frame data; obtaining a first position feature of the target and a second position feature of the target; performing training learning on the autoregression task model by using the first text feature, the first image feature and the first position feature of the key frame data and the second text feature, the second image feature and the second position feature of the middle frame data to obtain a target autoregression task model; and inputting the key frame into the target autoregressive task model to generate an animation video. The key frames of the video data are detected, the training data set is generated, and the autoregression task model is trained and learned, so that the animation video can be generated according to the prefix feature description, and the required animation video is generated through the key frames.

Description

Technical Field

[0001] The present invention relates to the technical field of animation video generation, and in particular to a key-frame-based animation video generation method, device, equipment and storage medium. Background Art

[0002] Animated videos can be used to create virtual demonstrations and simulations to showcase a company’s products, services, or solutions. For example, a simulated insurance claim process can be generated so that customers can learn how to file a claim and understand the relevant processes and requirements.

[0003] In animation video generation, keyframe data plays an important role. Animation generation refers to the use of computer algorithms and techniques to generate new animation content, rather than shooting real scenes. In generation, keyframe data is usually used to specify the starting state, ending state, and important intermediate states of the video. By setting keyframe data, the generation algorithm can generate intermediate frame data based on these keyframe data to achieve a smooth transition effect. In short, keyframe data plays a very important role in animation generation. They define important change points and key actions in the video, and generate intermediate frame data through algorithms to achieve a smooth transition effect.

[0004] However, there is currently no mature solution for generating keyframe-based animation videos. Summary of the invention

[0005] The purpose of the embodiments of the present application is to propose a key-frame-based animation video generation method, device, equipment and storage medium to solve the problem of how to generate the required animation video through key-frame data.

[0006] In order to solve the above technical problems, the embodiment of the present application provides a keyframe-based animation video generation method, which adopts the following technical solution, including:

[0007] Acquire training video data, preprocess the acquired training video data to obtain a video frame sequence, perform key frame detection on the video frame sequence to acquire key frame data, and generate intermediate frame data according to the key frame data;

[0008] The key frame data is segmented using the first word segmenter and the second word segmenter to obtain a first text feature and a first image feature, and the image in the key frame data is detected to obtain a first position feature of the target. The intermediate frame data is segmented using the first word segmenter and the second word segmenter to obtain a second text feature and a second image feature, and the image in the intermediate frame data is detected to obtain a second position feature of the target.

[0009] Using the first text feature, the first image feature, the first position feature of the key frame data and the second text feature, the second image feature, the second position feature of the intermediate frame data to train and learn the autoregressive task model, to obtain a converged target autoregressive task model;

[0010] The preset Prefix feature description is used, and the key frames are input into the target autoregressive task model to generate the animation video.

[0011] Further, before obtaining the training video data, preprocessing the obtained training video data to obtain a video frame sequence, performing key frame detection on the video frame sequence to obtain key frame data, and generating intermediate frame data according to the key frame data, the method further includes:

[0012] Adjusting the resolution of the acquired video data and normalizing the colors to obtain standard video data;

[0013] The standard video data is edited to obtain the training video data.

[0014] Further, the acquiring of training video data and preprocessing of the acquired training video data to obtain a video frame sequence include:

[0015] Decode the training video data to obtain a frame-by-frame image sequence;

[0016] Each frame of image is subjected to standardization processing to obtain the video frame sequence.

[0017] Furthermore, the performing key frame detection on the video frame sequence to obtain key frame data includes:

[0018] Calculate the image feature difference or motion feature difference between two adjacent frames of images respectively, and determine the candidate key frame according to the calculation results;

[0019] Perform K-Means clustering analysis on the candidate key frames to obtain key frame data of the video.

[0020] Further, the step of performing word segmentation processing on the key frame data using the first word segmenter and the second word segmenter to obtain the first text feature and the first image feature, performing target detection on the image in the key frame data, and obtaining the first position feature of the target includes:

[0021] Using the first word segmenter to segment the image of the key frame data into a plurality of image blocks, and calculating the average grayscale value of the image blocks to obtain a first image feature;

[0022] The second word segmenter maps the average grayscale value to a corresponding token according to the grayscale interval in which the average grayscale value is located, to obtain a first text feature;

[0023] Input the image sequence of key frame data into the yolov8 model for target detection, and obtain the confidence and predicted bounding box of the target in the key frame data;

[0024] The predicted bounding boxes are screened using a non-maximum suppression algorithm, bounding boxes with high confidence and low overlap with other bounding boxes are retained, and the category, bounding box coordinates and confidence of the predicted target are output to obtain a first position feature.

[0025] Further, the first word segmenter and the second word segmenter are used to perform word segmentation processing on the intermediate frame data to obtain a second text feature and a second image feature, and target detection is performed on the image in the intermediate frame data to obtain a second position feature of the target, including:

[0026] Using the first word segmenter to segment the image of the intermediate frame data into a plurality of image blocks, and calculating the average grayscale value of the image blocks to obtain a second image feature;

[0027] Using the second word segmenter, mapping the average grayscale value to a corresponding token according to the grayscale interval in which the average grayscale value is located, to obtain a second text feature;

[0028] Input the image sequence of the intermediate frame data into the yolov8 model for target detection, and obtain the confidence and predicted bounding box of the target in the intermediate frame data;

[0029] The predicted bounding boxes are screened using a non-maximum suppression algorithm, and bounding boxes with high confidence and low overlap with other bounding boxes are retained. The category, bounding box coordinates and confidence of the predicted target are output to obtain the second position feature.

[0030] Furthermore, the method of using the preset Prefix feature description and inputting the key frame into the target autoregressive task model to generate the animation video includes:

[0031] Inputting the key frame into the autoregressive task model to obtain feature information of the key frame data image and the intermediate frame data image;

[0032] According to the feature information of the key frame data image and the intermediate frame data image, the next frame image is predicted using a preset Prefix feature description;

[0033] The predicted image frames are added to a frame sequence consisting of key frames and intermediate frames to obtain the animation video.

[0034] In order to solve the above problem, a key-frame-based animation video generation device adopts the key-frame-based animation video generation method, comprising:

[0035] An acquisition module, used to acquire training video data, preprocess the acquired training video data to obtain a video frame sequence, perform key frame detection on the video frame sequence to acquire key frame data, and generate intermediate frame data according to the key frame data;

[0036] The feature extraction module is used to perform word segmentation processing on the key frame data using the first word segmenter and the second word segmenter respectively to obtain the first text feature and the first image feature, perform target detection on the image in the key frame data, and obtain the first position feature of the target, perform word segmentation processing on the intermediate frame data using the first word segmenter and the second word segmenter respectively to obtain the second text feature and the second image feature, perform target detection on the image in the intermediate frame data, and obtain the second position feature of the target;

[0037] A training module, used to train and learn the autoregressive task model using the first text feature, the first image feature, the first position feature of the key frame data and the second text feature, the second image feature, and the second position feature of the intermediate frame data to obtain a converged target autoregressive task model;

[0038] The generation module is used to use the preset Prefix feature description and input the key frame into the target autoregressive task model to generate the animation video.

[0039] In order to solve the above problems, an embodiment of the present application further proposes a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, it implements the steps of a key-frame-based animation video generation method.

[0040] In order to solve the above problems, an embodiment of the present application further proposes a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the steps of a key-frame-based animation video generation method are implemented.

[0041] Compared with the existing technology, a training data set is generated by detecting the key frame data of the video data and generating intermediate frame data based on the key frame data. The text features and image features of the training data set are segmented and the image features are detected to obtain their position features. The autoregressive task model for animation video generation is trained, so that animation video can be generated according to the preset Prefix feature description, thereby realizing the generation of the required animation video through key frame data. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 A flowchart of an embodiment of a key-frame-based animation video generation method according to the present application;

[0044] Figure 2 yes Figure 1 A flowchart of a specific implementation method before S100;

[0045] Figure 3 yes Figure 1 A flowchart of a specific implementation method of obtaining training video data in S100, preprocessing the obtained training video data, and obtaining a video frame sequence;

[0046] Figure 4 yes Figure 1 A flowchart of a specific implementation method of performing key frame detection on a video frame sequence and obtaining key frame data in S100;

[0047] Figure 5 yes Figure 1 A flowchart of a specific implementation method of performing word segmentation processing on key frame data using the first word segmenter and the second word segmenter respectively in S200 to obtain a first text feature and a first image feature, performing target detection on the image in the key frame data, and obtaining a first position feature of the target;

[0048] Figure 6 yes Figure 1 A flowchart of a specific implementation method of performing word segmentation processing on the intermediate frame data using the first word segmenter and the second word segmenter to obtain a second text feature and a second image feature, performing target detection on the image in the intermediate frame data, and obtaining a second position feature of the target in S200;

[0049] Figure 7 yes Figure 1 A flowchart of a specific implementation of S400;

[0050] Figure 8 A schematic diagram of the module structure of a keyframe-based animation video generation device according to the present application;

[0051] Fig. 9 A schematic diagram of the module structure of a computer device according to the present application. DETAILED DESCRIPTION

[0052] The following will be combined with the accompanying drawings in the invention to clearly and completely describe the technical solutions in the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, other embodiments obtained by ordinary technicians in this field without creative work are all within the scope of protection of the present invention.

[0053] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used in the specification of the application are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.

[0055] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0056] The purpose of the embodiments of the present application is to propose a key-frame-based animation video generation method, device, equipment and storage medium to solve the problem of how to generate the required animation video through key-frame data.

[0057] In order to solve the above technical problems, the embodiment of the present application provides a method for generating an animation video based on key frames, which adopts the following technical solutions: Figure 1 , Figure 1 1 is a flowchart of an embodiment of a keyframe-based animation video generation method according to the present application, comprising:

[0058] SS100, obtaining training video data, preprocessing the obtained training video data to obtain a video frame sequence, performing key frame detection on the video frame sequence, obtaining key frame data, and generating intermediate frame data according to the key frame data;

[0059] In this embodiment, a large amount of video data can be collected from the Internet as training samples, and key frames of the video data can be detected by a key frame detection algorithm.

[0060] Specifically, in order to collect higher quality training data, animation and movie videos can be collected because these videos themselves come with key frames.

[0061] After obtaining the key frames of the video, each video can be cut into several segments, and each segment can output a corresponding description through relevant AI tools.

[0062] In this embodiment, the acquired video training data can be represented as {[prompt1, keyframe1, vidoe1], [prompt2, keyframe2, video2]}, where prompt is the description text of each video, keyframe is the key frame of each video, and vidoe is the video to the next key frame.

[0063] The keyframe-based animation video generation method in the embodiment of the present application runs on a computer device, and the communication module of the computer device can adopt a wired connection method or a wireless connection method.

[0064] It should be noted that the above-mentioned wireless connection methods may include but are not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or developed in the future.

[0065] The keyframe-based animation video generation method provided in the embodiment of the present application is mainly used to solve the following technical problems:

[0066] The keyframe-based animation video generation method belongs to conditional video generation. Conditional video generation refers to inputting text, pictures or other conditions. In the past, the training data for video generation was randomly sampled, such as fixed frame intervals or fixed durations, which ignored the internal structure of the video. If the video is disconnected according to the keyframe data, the video can be split into several segments, and each segment is semantically independent. This is very important for both video generation training and reasoning, because the video itself is very semantically rich. If the model is allowed to learn across different keyframe data, the difficulty will increase.

[0067] Key frame data plays a very important role in animation generation. They define important change points and key actions in the video, and generate intermediate frame data through algorithms to achieve a smooth transition effect. Therefore, the purpose of the embodiment of the present application is to achieve the generation of animation video based on key frame data.

[0068] In this embodiment, a deep learning algorithm can be used to use a deep neural network to learn the motion patterns between key frames and predict and generate intermediate frames. For example, a convolutional neural network (CNN) or a recurrent neural network (RNN) can be used to process a video frame sequence and predict intermediate frames.

[0069] In some optional embodiments, such as Figure 2 , Figure 2 yes Figure 1 A flowchart of a specific implementation method before S100; before S100, the key frame-based animation video generation method also includes: S101, adjusting the resolution of the acquired video data, and normalizing the color to obtain standard video data; S102, editing the standard video data to obtain training video data.

[0070] In this embodiment, the resolution is adjusted to reduce the data dimension and improve the training speed. For example, if the resolution of the original video data is 4K (3840*2160), adjusting it to a lower resolution of 720P (1280*720) can significantly reduce the amount of calculation. Adjusting the resolution of the video can also improve the generalization ability of the model. Color normalization is mainly to eliminate the differences between different video data.

[0071] When editing, the quality of training data can be improved. For example, if you are training to recognize human actions, you can edit out scenes that are not related to human actions and only keep clips of specific actions to focus on key information.

[0072] This application collects a large amount of video data sets, and clips the videos, splits them into frames, adjusts the resolution, and normalizes the colors, so that the subsequent autoregressive task model (Prefix LLM model) can better learn the video data.

[0073] In some optional embodiments, such as Figure 3 , Figure 3 yes Figure 1 A flowchart of a specific implementation method of obtaining training video data in S100, preprocessing the obtained training video data, and obtaining a video frame sequence; comprising:

[0074] S110, decode the training video data to obtain a frame-by-frame image sequence; S120, standardize each frame of the image to obtain a video frame sequence. In an embodiment of the present application, each frame of the image is standardized, for example, adjusting the image size and color space conversion. By pre-standardizing and feature extracting the video data, it is beneficial to improve the processing efficiency of the subsequent autoregressive task model (Prefix LLM model) and to improve its accuracy.

[0075] In some optional embodiments, such as Figure 4 , Figure 4 yes Figure 1 The flowchart of a specific implementation method of performing key frame detection on a video frame sequence and obtaining key frame data in S100; S130, respectively calculating the image feature difference or motion feature difference between two adjacent frames of images, and determining candidate key frames based on the calculation results; S140, performing K-Means clustering analysis on the candidate key frames to obtain the key frame data of the video.

[0076] In this embodiment, by comparing the feature differences between adjacent frames, if the feature differences exceed a certain threshold, the two adjacent frames are determined as candidate key frame data. For example, the difference in color histograms of adjacent frames can be calculated, or a distance metric based on deep learning features can be calculated. If the motion changes between adjacent frames are large, they can also be used as candidates for key frame data. For example, when the amplitude or direction of the optical flow vector changes significantly, the two adjacent frames are determined as candidate key frame data.

[0077] In this embodiment, K-Means clustering analysis of video key frames can be performed by time interval sampling: frames are extracted from the video as key frames at fixed time intervals. For example, one frame is taken every 5 seconds or 10 seconds. The appropriate K value can be determined by trying different K values ​​and observing the quality of the clustering results. K key frames can be randomly selected as initial cluster centers. For each key frame, its distance from each cluster center is calculated. Metrics such as Euclidean distance and cosine distance can be used. The key frame is assigned to the cluster where the nearest cluster center is located. Then update the cluster center: recalculate the center of each cluster, that is, the feature mean of all key frames in the cluster. Repeat the above steps until the cluster center no longer changes or the preset number of iterations is reached.

[0078] In this embodiment, the final key frame data is obtained by clustering the candidate key frame data, and the frames with similar features are clustered into one category. Then a representative frame is selected from each cluster as the key frame data. In addition, in order to avoid the key frame data being too dense or sparse, a time interval threshold can be set to ensure that the key frame data is reasonably distributed in time.

[0079] By adopting cluster analysis method to obtain the final key frame data, the accuracy of key frame data detection is improved.

[0080] S200, respectively use the first word segmenter and the second word segmenter to perform word segmentation processing on the key frame data to obtain the first text feature and the first image feature, perform target detection on the image in the key frame data to obtain the first position feature of the target, respectively use the first word segmenter and the second word segmenter to perform word segmentation processing on the intermediate frame data to obtain the second text feature and the second image feature, perform target detection on the image in the intermediate frame data to obtain the second position feature of the target.

[0081] In some optional embodiments, such as Figure 5 , respectively using the first word segmenter and the second word segmenter to perform word segmentation processing on the key frame data to obtain the first text feature and the first image feature, performing target detection on the image in the key frame data to obtain the first position feature of the target, including:

[0082] S210. Use the first word segmenter to divide the image of the key frame data into multiple image blocks, and calculate the average grayscale value of the image blocks to obtain the first image feature; S220. Use the second word segmenter to map the average grayscale value to the corresponding token according to the grayscale interval in which it is located to obtain the first text feature; S230. Input the image sequence of the key frame data into the yolov8 model for target detection, and obtain the confidence of the target in the key frame data and the predicted bounding box; S240. Use the non-maximum suppression algorithm to screen the predicted bounding boxes, retain the bounding boxes with higher confidence and lower overlap with other bounding boxes, and output the category of the predicted target, the bounding box coordinates and the confidence to obtain the first position feature.

[0083] In this embodiment, the first word segmenter uses the T5 word segmenter to perform text segmentation processing and extract text features, and the second word segmenter uses the Magvit-V2 word segmenter as a video word segmenter to extract image features of video data.

[0084] For example, suppose the key frame data is a grayscale image with a resolution of 64x64 (in practice, the Magvit-V2 tokenizer can also process color videos and color images), and each pixel in the image is represented by an integer between 0 and 255 to represent its grayscale value. Divide the image into several small image blocks, which can be 4x4 image blocks. Calculate the average grayscale value of each 4x4 image block. Divide the grayscale value range from 0 to 255 into 32 intervals, and map each average grayscale value to a corresponding "token" (which can be understood as a discrete symbol) according to the interval in which it is located. For example, if the average grayscale value of a 4x4 image block is 128 and it falls in the 16th interval, then the 16th token is used to represent this image block. Tokens are the discrete representations generated by the Magvit-V2 image-video joint tokenizer for the image. Subsequent language models can be learned and generated based on these tokens, such as generating text describing the content of the image, or generating similar images based on the text.

[0085] By adopting the Magvit-V2 image-video joint segmenter, it is possible to segment images and videos using a shared vocabulary, which is conducive to the effective use of the model to process and generate visual information.

[0086] In some optional embodiments, such as Figure 6 ,yes Figure 1 In S200, the first word segmenter and the second word segmenter are used to perform word segmentation processing on the intermediate frame data to obtain the second text feature and the second image feature, and the target is detected on the image in the intermediate frame data to obtain the second position feature of the target. A flowchart of a specific implementation method; S200 includes:

[0087] S250. Use the first word segmenter to divide the image of the intermediate frame data into multiple image blocks, and calculate the average grayscale value of the image blocks to obtain the second image feature; S260. Use the second word segmenter to map the average grayscale value to the corresponding token according to the grayscale interval in which it is located to obtain the second text feature; S270. Input the image sequence of the intermediate frame data into the yolov8 model for target detection, and obtain the confidence of the target in the intermediate frame data and the predicted bounding box; S280. Use the non-maximum suppression algorithm to screen the predicted bounding boxes, retain the bounding boxes with higher confidence and lower overlap with other bounding boxes, and output the category of the predicted target, the bounding box coordinates and the confidence to obtain the second position feature.

[0088] The yolov8 model uses the key frame data and the image frame of the intermediate frame data for target detection, and divides the input image into grid cells. For example, the image is divided into grid cells of different sizes such as 7×7 or 19×19. Multiple bounding boxes are predicted for each grid cell, and each bounding box contains 5 parameters, namely the center coordinates, width, height, and confidence. The confidence reflects the probability that the bounding box contains the target and the accuracy of the bounding box, and its value is between 0 and 1.

[0089] Non-maximum suppression (NMS): Filters multiple predicted bounding boxes to remove redundant bounding boxes. The NMS algorithm retains bounding boxes with high confidence and low overlap with other bounding boxes based on the confidence and overlap of the bounding boxes to avoid repeated detection of the same object by multiple bounding boxes.

[0090] By inputting the image sequence of key frame data and intermediate frame data into the yolov8 model for target detection, the target category, bounding box coordinates, confidence level and other information are obtained, thereby completing the detection and positioning of the target in the image, and providing more detailed training data for the autoregressive task model (Prefix LLM model) in the embodiment of the present application, which is beneficial to improve the accuracy of video generation of the autoregressive task model (Prefix LLM model) and generate more details of the required video.

[0091] S300, using the first text feature, the first image feature, the first position feature of the key frame data and the second text feature, the second image feature, the second position feature of the intermediate frame data to train and learn the autoregressive task model to obtain a converged target autoregressive task model.

[0092] S400, using the preset Prefix feature description, and inputting the key frame into the target autoregressive task model to generate an animation video.

[0093] In the embodiment of the present application, the autoregressive task model (Prefix LLM model) is trained and learned by detecting the key frame data of a large amount of video data, and the concept of prefix is ​​introduced on the basis of the traditional language model structure. The prefix part is usually composed of a series of special tokens, which can contain information about the task, context or specific instructions.

[0094] When processing text, the autoregressive task model (Prefix LLM model) in the embodiment of the present application will consider both the input text content and the information in the Prefix to better understand the task requirements and generate a scene video that meets the requirements.

[0095] For example, based on the Prefix feature description and the input key frame data image data, create virtual demonstrations and simulation scenes to showcase the company's products, services or solutions. Specifically, a simulated insurance claims process can be generated so that customers can understand how to apply for claims and understand the relevant processes and requirements.

[0096] In some optional embodiments, such as Figure 7 , Figure 7 yes Figure 1 Flowchart of the first specific implementation method of S400; S400 includes: S410, inputting the key frame into the autoregressive task model to obtain the feature information of the key frame data image and the intermediate frame data image; S420, predicting the next frame image according to the feature information of the key frame data image and the intermediate frame data image using the preset Prefix feature description; S430, adding the predicted image frame to the frame sequence composed of the key frame and the intermediate frame to obtain the animation video.

[0097] Prefix feature description is a prefix, which can also be understood as the condition for generating an animated video. It is usually composed of a series of special tokens, which can contain information about tasks, contexts, or specific instructions. Prefix feature description is used to guide the video generation process. It can contain information about the theme, style, scene, action, etc. of the video, or specific instructions, such as "generate a video of animals running in the forest." The setting of Prefix feature description needs to be determined according to specific tasks and requirements.

[0098] In this embodiment, the autoregressive task refers to the model predicting the next image frame or video frame based on given condition information and key frame data and intermediate value data information.

[0099] By using the autoregressive task based on the Prefix LLM model, we can learn the patterns and regularities in the video data and generate new video content based on the given Prefix feature description. In this way, we can gradually generate coherent, high-quality, and realistic videos by using the representation and learning capabilities of the language model and the autoregressive method.

[0100] In order to solve the above problems, the embodiment of the present application further proposes a key frame-based animation video generation device 500, which adopts a key frame-based animation video generation method, such as Figure 8 , Figure 8 A schematic diagram of the module structure of a keyframe-based animation video generation device according to the present application; comprising:

[0101] The acquisition module 501 is used to acquire training video data, pre-process the acquired training video data to obtain a video frame sequence, perform key frame detection on the video frame sequence, acquire key frame data, and generate intermediate frame data according to the key frame data;

[0102] The feature extraction module 502 is used to perform word segmentation processing on the key frame data using the first word segmenter and the second word segmenter respectively to obtain the first text feature and the first image feature, perform target detection on the image in the key frame data to obtain the first position feature of the target, perform word segmentation processing on the intermediate frame data using the first word segmenter and the second word segmenter respectively to obtain the second text feature and the second image feature, perform target detection on the image in the intermediate frame data to obtain the second position feature of the target;

[0103] A training module 503 is used to train and learn the autoregressive task model using the first text feature, the first image feature, the first position feature of the key frame data and the second text feature, the second image feature, and the second position feature of the intermediate frame data to obtain a converged target autoregressive task model;

[0104] The generation module 504 is used to use the preset Prefix feature description and input the key frame into the target autoregressive task model to generate the animation video.

[0105] In order to solve the above problems, Fig. 9 , Fig. 9 A schematic diagram of a module structure of a computer device according to the present application. The present application also proposes a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, it implements the steps of a key-frame-based animation video generation method.

[0106] In order to solve the above problems, an embodiment of the present application further proposes a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, steps of a key-frame-based animation video generation method are implemented.

[0107] In order to solve the above problems, an embodiment of the present application also proposes a computer device, which adopts the following technical solution: it includes a memory, a communication module and a processor, the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, it implements the steps of a key-frame-based animation video generation method.

[0108] The computer device may be a computer, server, workstation or other device, or a mobile phone, tablet, vehicle-mounted mobile terminal or other device with program execution capability. The internal structure diagram of the computer device may be as follows: Fig. 9 As shown, Fig. 9It is a structural diagram of an embodiment of a computer device according to the present application. The computer device includes a processor, a memory and a communication module. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage medium stores an operating system, instructions or codes. The internal memory provides an environment for the operation of the operating system and instructions or codes in the non-volatile storage medium. When the instructions or codes are executed by the processor, the functions or steps of a method for fine-tuning the above-mentioned pre-trained model are implemented. The communication module of the computer device may include a network interface and / or a wireless communication module, and the computer device may communicate with other devices or service platforms through the communication module. In addition, the computer device may also include a display screen and an input device, etc.

[0109] The memory is used to store a computer program, the computer program includes program instructions, the processor is configured to call the program instructions, and the processor implements the following steps when executing the instructions or codes: S100, obtaining training video data, preprocessing the obtained training video data, obtaining a video frame sequence, performing key frame detection on the video frame sequence, obtaining key frame data, and generating intermediate frame data based on the key frame data; S200, performing word segmentation processing on the key frame data using the first word segmenter and the second word segmenter respectively, obtaining a first text feature and a first image feature, performing target detection on the image in the key frame data, and obtaining a first position feature of the target. S300, respectively use the first word segmenter and the second word segmenter to perform word segmentation processing on the intermediate frame data to obtain the second text feature and the second image feature, perform target detection on the image in the intermediate frame data, and obtain the second position feature of the target; S300, use the first text feature, the first image feature, the first position feature of the key frame data obtained and the second text feature, the second image feature, and the second position feature of the intermediate frame data to train and learn the autoregressive task model to obtain a converged target autoregressive task model; S400, use the preset Prefix feature description, and input the key frame into the target autoregressive task model to generate an animation video.

[0110] In order to solve the above problems, the embodiment of the present application further proposes a computer-readable storage medium, which adopts the following technical solution:

[0111] The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor executes an adaptive parameter efficient fine-tuning method.

[0112] The present application also provides a computer-readable storage medium storing a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, Figures 1 to 7The key frame-based animation video generation method provided in each step can be specifically referred to the implementation methods provided in the above steps, which will not be repeated here.

[0113] The computer-readable storage medium may be a keyframe-based animation video generation device provided in any of the aforementioned embodiments, or an internal storage unit of the terminal device, such as a hard disk or memory of a computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., provided on the computer device.

[0114] Furthermore, the computer-readable storage medium may also include both an internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0115] However, it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), digital signal processors (DSP), embedded devices, etc.

[0116] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0117] The non-Company software tools or components appearing in the embodiments of this application are merely examples and do not represent actual use.

[0118] Compared with the existing technology, a training data set is generated by detecting the key frames of video data and generating intermediate frame data based on the key frame data. The text features and image features of the training data set are segmented and the image features are detected to obtain their position features. The autoregressive task model for animation video generation is trained, so that animation video can be generated according to the preset Prefix feature description, thereby realizing the generation of required animation video through key frames.

[0119] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating an animation video based on key frames, comprising: Acquire training video data, preprocess the acquired training video data to obtain a video frame sequence, perform key frame detection on the video frame sequence to acquire key frame data, and generate intermediate frame data according to the key frame data; The key frame data is segmented using the first word segmenter and the second word segmenter to obtain a first text feature and a first image feature, and the image in the key frame data is detected to obtain a first position feature of the target. The intermediate frame data is segmented using the first word segmenter and the second word segmenter to obtain a second text feature and a second image feature, and the image in the intermediate frame data is detected to obtain a second position feature of the target. Using the first text feature, the first image feature, the first position feature of the key frame data and the second text feature, the second image feature, the second position feature of the intermediate frame data to train and learn the autoregressive task model, to obtain a converged target autoregressive task model; The preset Prefix feature description is used, and the key frames are input into the target autoregressive task model to generate the animation video.

2. The keyframe-based animation video generation method according to claim 1, characterized in that: Before acquiring the training video data, preprocessing the acquired training video data to obtain a video frame sequence, performing key frame detection on the video frame sequence to acquire key frame data, and generating intermediate frame data according to the key frame data, the method further includes: Adjusting the resolution of the acquired video data and normalizing the colors to obtain standard video data; The standard video data is edited to obtain the training video data.

3. The keyframe-based animation video generation method according to claim 1 is characterized in that the step of obtaining training video data and preprocessing the obtained training video data to obtain a video frame sequence comprises: Decode the training video data to obtain a frame-by-frame image sequence; Each frame of image is subjected to standardization processing to obtain the video frame sequence.

4. The method for generating an animation video based on key frames according to claim 3, characterized in that the step of performing key frame detection on the video frame sequence to obtain key frame data comprises: Calculate the image feature difference or motion feature difference between two adjacent frames of images respectively, and determine the candidate key frame according to the calculation results; Perform K-Means clustering analysis on the candidate key frames to obtain key frame data of the video.

5. The method for generating an animation video based on key frames according to claim 1, characterized in that the first word segmenter and the second word segmenter are used to perform word segmentation processing on the key frame data to obtain the first text feature and the first image feature, and the image in the key frame data is subjected to target detection to obtain the first position feature of the target, including: Using the first word segmenter to segment the image of the key frame data into a plurality of image blocks, and calculating the average grayscale value of the image blocks to obtain a first image feature; The second word segmenter maps the average grayscale value to a corresponding token according to the grayscale interval in which the average grayscale value is located, to obtain a first text feature; Input the image sequence of key frame data into the yo l ov8 model for target detection, and obtain the confidence and predicted bounding box of the target in the key frame data; The predicted bounding boxes are screened using a non-maximum suppression algorithm, bounding boxes with high confidence and low overlap with other bounding boxes are retained, and the category, bounding box coordinates and confidence of the predicted target are output to obtain a first position feature.

6. The keyframe-based animation video generation method according to claim 1 is characterized in that the first word segmenter and the second word segmenter are used to perform word segmentation processing on the intermediate frame data to obtain the second text feature and the second image feature, and the target is detected on the image in the intermediate frame data to obtain the second position feature of the target, including: Using the first word segmenter to segment the image of the intermediate frame data into a plurality of image blocks, and calculating the average grayscale value of the image blocks to obtain a second image feature; Using the second word segmenter, mapping the average grayscale value to a corresponding token according to the grayscale interval in which the average grayscale value is located, to obtain a second text feature; Input the image sequence of the intermediate frame data into the yo l ov8 model for target detection, and obtain the confidence and predicted bounding box of the target in the intermediate frame data; The predicted bounding boxes are screened using a non-maximum suppression algorithm, and bounding boxes with high confidence and low overlap with other bounding boxes are retained. The category, bounding box coordinates and confidence of the predicted target are output to obtain the second position feature.

7. The method for generating an animated video based on key frames according to claim 1, characterized in that the step of using a preset Prefix feature description and inputting the key frames into the target autoregressive task model to generate an animated video comprises: Inputting the key frame into the autoregressive task model to obtain feature information of the key frame data image and the intermediate frame data image; According to the feature information of the key frame data image and the intermediate frame data image, the next frame image is predicted using a preset Prefix feature description; The predicted image frames are added to a frame sequence consisting of key frames and intermediate frames to obtain the animation video.

8. A key-frame based animation video generation device, using the key-frame based animation video generation method according to any one of claims 1 to 7, characterized in that: include: An acquisition module, used to acquire training video data, preprocess the acquired training video data to obtain a video frame sequence, perform key frame detection on the video frame sequence to acquire key frame data, and generate intermediate frame data according to the key frame data; The feature extraction module is used to perform word segmentation processing on the key frame data using the first word segmenter and the second word segmenter respectively to obtain the first text feature and the first image feature, perform target detection on the image in the key frame data, and obtain the first position feature of the target, perform word segmentation processing on the intermediate frame data using the first word segmenter and the second word segmenter respectively to obtain the second text feature and the second image feature, perform target detection on the image in the intermediate frame data, and obtain the second position feature of the target; A training module, used to train and learn the autoregressive task model using the first text feature, the first image feature, the first position feature of the key frame data and the second text feature, the second image feature, and the second position feature of the intermediate frame data to obtain a converged target autoregressive task model; The generation module is used to use the preset Prefix feature description and input the key frame into the target autoregressive task model to generate the animation video.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the steps of the key-frame-based animation video generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer-readable instructions stored thereon, characterized in that: When the computer-readable instructions are executed by a processor, the steps of the key-frame-based animation video generation method according to any one of claims 1 to 7 are implemented.

Images