Method, apparatus, device, and storage medium for generating keyframe-based animated videos.

Abstract

This application belongs to the field of video generation technology and discloses a method, apparatus, device, and storage medium for generating animated videos based on keyframes. The method includes the following steps: performing keyframe detection on a video frame sequence to obtain keyframe data, and generating intermediate frame data based on the keyframe data; obtaining a first position feature and a second position feature of the target; training an autoregressive task model using the first text feature, first image feature, and first position feature of the keyframe data, and the second text feature, second image feature, and second position feature of the intermediate frame data to obtain a target autoregressive task model; and inputting the keyframes into the target autoregressive task model to generate the animated video. By detecting keyframes in the video data to generate a training dataset, and by training the autoregressive task model, animated videos can be generated based on prefix feature descriptions, thus realizing the generation of desired animated videos using keyframes.

Description

Technical Field

[0001] This invention relates to the field of animation video generation technology, and in particular to a method, apparatus, device, and storage medium for generating animation videos based on keyframes. Background Technology

[0002] Animated videos can be used to create virtual demonstrations and simulations to showcase a company's products, services, or solutions. For example, a simulated insurance claims process can be generated so that customers understand how to file a claim and learn about the relevant procedures and requirements.

[0003] Keyframe data plays a crucial role in animation video generation. Animation generation refers to using computer algorithms and techniques to create new animated content, rather than through filming real-world scenes. In generation, keyframe data is typically used to specify the start, end, and important intermediate states of the video. By setting keyframe data, the generation algorithm can generate intermediate frames based on these keyframes to achieve smooth transitions. In short, keyframe data plays a vital role in animation generation; they define important points of change and key actions in the video, and algorithms use them to generate intermediate frames for smooth transitions.

[0004] However, there is currently no mature solution for generating keyframe-based animated videos. Summary of the Invention

[0005] The purpose of this application is to provide a method, apparatus, device, and storage medium for generating animated videos based on keyframes, so as to solve the problem of how to generate the required animated videos from keyframe data.

[0006] To address the aforementioned technical problems, this application provides a keyframe-based animation video generation method, employing the following technical solution:

[0007] Acquire training video data, preprocess the acquired training video data to obtain a video frame sequence, perform keyframe detection on the video frame sequence to obtain keyframe data, and generate intermediate frame data based on the keyframe data.

[0008] The keyframe data is segmented using a first word segmenter and a second word segmenter to obtain first text features and first image features. Target detection is performed on the image in the keyframe data to obtain the first position feature of the target. The intermediate frame data is segmented using a first word segmenter and a second word segmenter to obtain second text features and second image features. Target detection is performed on the image in the intermediate frame data to obtain the second position feature of the target.

[0009] The autoregressive task model is trained and learned using the first text features, first image features, and first position features of the acquired keyframe data and the second text features, second image features, and second position features of the intermediate frame data, to obtain a converged target autoregressive task model.

[0010] Animation videos are generated by using preset prefix features and inputting keyframes into the target autoregressive task model.

[0011] Furthermore, prior to acquiring training video data, preprocessing the acquired training video data to obtain a video frame sequence, performing keyframe detection on the video frame sequence to obtain keyframe data, and generating intermediate frame data based on the keyframe data, the method further includes:

[0012] The resolution of the acquired video data is adjusted, and the colors are normalized to obtain standard video data.

[0013] The standard video data is edited to obtain the training video data.

[0014] Furthermore, the step of acquiring training video data and preprocessing the acquired training video data to obtain a video frame sequence includes:

[0015] Decode the training video data to obtain frame-by-frame image sequences;

[0016] Each frame of the image is standardized to obtain the video frame sequence.

[0017] Furthermore, the step of performing keyframe detection on the video frame sequence to obtain keyframe data includes:

[0018] Calculate the differences in image features or motion features between two adjacent frames, and determine candidate keyframes based on the calculation results;

[0019] K-Means clustering analysis is performed on candidate keyframes to obtain keyframe data of the video.

[0020] Furthermore, the step of segmenting the keyframe data using the first and second word segmenters respectively to obtain first text features and first image features, and then performing target detection on the image in the keyframe data to obtain the first position features of the target, includes:

[0021] The first word segmenter is used to segment the keyframe data image into multiple image blocks, and the average gray value of the image blocks is calculated to obtain the first image feature;

[0022] The second word segmenter maps the average gray value to the corresponding token based on the gray range in which the gray value is located, and obtains the first text feature;

[0023] The image sequence of keyframe data is input into the YOLOv8 model for target detection, and the confidence of the target in the keyframe data and the predicted bounding box are obtained.

[0024] The predicted bounding boxes are filtered using a non-maximum suppression algorithm, retaining those with high confidence and low overlap with other bounding boxes, and the predicted target category, bounding box coordinates, and confidence are output to obtain the first location feature.

[0025] Furthermore, the step of segmenting the intermediate frame data using the first and second word segmenters respectively to obtain the second text features and the second image features, and then performing target detection on the image in the intermediate frame data to obtain the second location features of the target, includes:

[0026] The first word segmenter is used to segment the image of the intermediate frame data into multiple image blocks, and the average gray value of the image blocks is calculated to obtain the second image feature;

[0027] The second word segmenter maps the average gray value to the corresponding token based on the gray range in which the gray value is located, thus obtaining the second text feature;

[0028] The image sequence of the intermediate frame data is input into the YOLOv8 model for target detection, and the confidence score and predicted bounding box of the target in the intermediate frame data are obtained.

[0029] The non-maximum suppression algorithm is used to filter the predicted bounding boxes, retaining those with high confidence and low overlap with other bounding boxes, and outputting the predicted target category, bounding box coordinates and confidence, thus obtaining the second location feature.

[0030] Furthermore, the step of using preset prefix features to describe and inputting keyframes into the target autoregressive task model to generate animation videos includes:

[0031] The keyframes are input into the autoregressive task model to obtain the feature information of the keyframe data image and the intermediate frame data image.

[0032] Based on the feature information of the keyframe data image and the intermediate frame data image, the next frame image is predicted using the preset Prefix feature description;

[0033] The predicted image frames are added to a frame sequence consisting of keyframes and intermediate frames to obtain the animated video.

[0034] To address the aforementioned problems, a keyframe-based animation video generation device employs the aforementioned keyframe-based animation video generation method, comprising:

[0035] The acquisition module is used to acquire training video data, preprocess the acquired training video data to obtain a video frame sequence, perform keyframe detection on the video frame sequence to acquire keyframe data, and generate intermediate frame data based on the keyframe data.

[0036] The feature extraction module is used to perform word segmentation on the keyframe data using the first word segmenter and the second word segmenter respectively to obtain the first text feature and the first image feature; to perform target detection on the image in the keyframe data to obtain the first position feature of the target; to perform word segmentation on the intermediate frame data using the first word segmenter and the second word segmenter respectively to obtain the second text feature and the second image feature; and to perform target detection on the image in the intermediate frame data to obtain the second position feature of the target.

[0037] The training module is used to train and learn the autoregressive task model using the first text features, first image features, and first position features of the acquired keyframe data and the second text features, second image features, and second position features of the intermediate frame data, so as to obtain a converged target autoregressive task model.

[0038] The generation module is used to describe the target autoregressive task model using preset prefix features and input keyframes to generate animation videos.

[0039] To address the aforementioned issues, this application also proposes a computer device, including a memory and a processor. The memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of a keyframe-based animation video generation method.

[0040] To address the aforementioned issues, this application also proposes a computer-readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the steps of a keyframe-based animation video generation method.

[0041] Compared with existing technologies, this method generates an animation video by detecting keyframe data in video data, generating intermediate frame data based on the keyframe data, and creating a training dataset. By performing word segmentation on the text and image features of the training dataset and object detection on the image features to obtain their positional features, the method trains an autoregressive task model for animation video generation. This allows animation videos to be generated based on preset prefix feature descriptions, thus realizing the generation of required animation videos using keyframe data. Attached Figure Description

[0042] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 A flowchart of an embodiment of a keyframe-based animation video generation method according to this application;

[0044] Figure 2 yes Figure 1 A flowchart of a specific implementation method prior to S100;

[0045] Figure 3 yes Figure 1 The flowchart shows a specific implementation method of acquiring training video data in S100, preprocessing the acquired training video data, and obtaining a video frame sequence.

[0046] Figure 4 yes Figure 1 A flowchart of a specific implementation method for S100 to perform keyframe detection on a video frame sequence and obtain keyframe data;

[0047] Figure 5 yes Figure 1 The flowchart shows a specific implementation method of using a first word segmenter and a second word segmenter to segment keyframe data to obtain first text features and first image features, and performing target detection on the image in the keyframe data to obtain the first position features of the target.

[0048] Figure 6 yes Figure 1 The flowchart shows a specific implementation of the S200 method that uses a first word segmenter and a second word segmenter to segment the intermediate frame data to obtain second text features and second image features, performs target detection on the image in the intermediate frame data, and obtains the second position features of the target.

[0049] Figure 7 yes Figure 1 A flowchart of a specific implementation of the S400;

[0050] Figure 8 A schematic diagram of the module structure of a keyframe-based animation video generation device according to this application;

[0051] Figure 9 A schematic diagram of the modular structure of a computer device according to this application. Detailed Implementation

[0052] The technical solutions of this invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are all within the scope of protection of this invention.

[0053] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings of this application are intended to cover non-exclusive inclusion.

[0055] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0056] The purpose of this application is to provide a method, apparatus, device, and storage medium for generating animated videos based on keyframes, so as to solve the problem of how to generate the required animated videos from keyframe data.

[0057] To address the aforementioned technical problems, this application provides a keyframe-based animation video generation method, employing the following technical solution: Figure 1 , Figure 1 This is a flowchart of an embodiment of a keyframe-based animation video generation method according to this application, including:

[0058] SS100: Acquire training video data, preprocess the acquired training video data to obtain a video frame sequence, perform keyframe detection on the video frame sequence, acquire keyframe data, and generate intermediate frame data based on the keyframe data.

[0059] In this embodiment, a large amount of video data can be collected from the Internet as training samples, and keyframe detection algorithms can be used to detect keyframes in the video data.

[0060] Specifically, in order to collect higher quality training data, animation and movie videos can be collected, because these videos themselves contain keyframes.

[0061] After obtaining the keyframes of the video, each video segment can be cut into several segments, and each segment can be output with a corresponding description through relevant AI tools.

[0062] In this embodiment, the acquired video training data can be represented as {[prompt1,keyframe1,video1],[prompt2,keyframe2,video2]}, where prompt is the description text of each video segment, keyframe is the keyframe of each video segment, and video is the video up to the next keyframe.

[0063] The keyframe-based animation video generation method in this application embodiment runs on a computer device, and the communication module of the computer device can be connected via wired or wireless means.

[0064] It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G / 5G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultra wideband) connections, and other currently known or future wireless connection methods.

[0065] The keyframe-based animation video generation method provided in this application mainly addresses the following technical problems:

[0066] Keyframe-based animation video generation methods belong to conditional video generation. Conditional video generation refers to inputting text, images, or other conditions. Previous video generation training data was randomly sampled, such as at fixed intervals or for fixed durations, thus ignoring the video's internal structure. If the video is broken down according to keyframe data, it can be divided into several segments, each semantically independent. This is crucial for both training and inference in video generation, as the inherent semantics of videos are rich, and the difficulty increases when the model learns across different keyframe datasets.

[0067] Keyframe data plays a crucial role in animation generation. It defines important changes and key actions in the video and uses algorithms to generate intermediate frame data to achieve smooth transitions. Therefore, the purpose of this application is to realize the generation of animated videos based on keyframe data.

[0068] In this embodiment, deep learning algorithms can be used to learn motion patterns between keyframes using deep neural networks and predict intermediate frames. For example, convolutional neural networks (CNNs) or recurrent neural networks (RNNs) can be used to process video frame sequences and predict intermediate frames.

[0069] In some alternative implementations, such as Figure 2 , Figure 2 yes Figure 1 A flowchart of a specific implementation method prior to S100; prior to S100, the keyframe-based animation video generation method further includes: S101, adjusting the resolution of the acquired video data and normalizing the colors to obtain standard video data; S102, editing the standard video data to obtain training video data.

[0070] In this embodiment, adjusting the resolution aims to reduce data dimensionality and improve training speed. For example, if the original video data has a resolution of 4K (3840*2160), adjusting it to a lower resolution of 720P (1280*720) can significantly reduce computational load. Adjusting the video resolution can also improve the model's generalization ability. Color normalization is mainly to eliminate differences between different video data.

[0071] Editing can improve the quality of training data. For example, when training for recognizing human actions, scenes unrelated to human actions can be edited out, keeping only segments of specific actions and focusing on key information.

[0072] This application collects a large video dataset and performs operations such as video editing, frame segmentation, resolution adjustment, and color normalization on the videos so that the subsequent autoregressive task model (Prefix LLM model) can better learn from the video data.

[0073] In some alternative implementations, such as Figure 3 , Figure 3 yes Figure 1 The flowchart below shows a specific implementation method for obtaining training video data and preprocessing the obtained training video data to obtain a video frame sequence in S100; including:

[0074] S110. Decode the training video data to obtain a frame-by-frame image sequence; S120. Standardize each frame image to obtain a video frame sequence. In this embodiment, standardization is performed on each frame image, for example, by adjusting the image size and converting the color space. By pre-standardizing and extracting features from the video data, the processing efficiency of the subsequent autoregressive task model (Prefix LLM model) is improved, and its accuracy is also enhanced.

[0075] In some alternative implementations, such as Figure 4 , Figure 4 yes Figure 1 The flowchart shows a specific implementation of keyframe detection in video frame sequence in step S100 to obtain keyframe data; step S130: calculate the image feature difference or motion feature difference between two adjacent frames respectively, and determine candidate keyframes based on the calculation results; step S140: perform K-Means clustering analysis on the candidate keyframes to obtain keyframe data of the video.

[0076] In this embodiment, feature differences between adjacent frames can be compared. If the feature difference exceeds a certain threshold, the two adjacent frames are identified as candidate keyframe data. For example, the difference in color histograms between adjacent frames can be calculated, or a distance metric based on deep learning features can be calculated. If there are significant motion changes between adjacent frames, they can also be considered as candidate keyframe data. For example, when the amplitude or direction of the optical flow vector changes significantly, the two adjacent frames are identified as candidate keyframe data.

[0077] In this embodiment, K-Means clustering analysis of video keyframes can be performed by time-interval sampling: frames are extracted from the video as keyframes at fixed time intervals. For example, one frame is taken every 5 or 10 seconds. A suitable K value can be determined by trying different K values ​​and observing the quality of the clustering results. K keyframes can be randomly selected as initial cluster centers. For each keyframe, its distance to each cluster center is calculated, using metrics such as Euclidean distance or cosine distance. The keyframe is assigned to the cluster containing the nearest cluster center. Then, the cluster centers are updated: the center of each cluster is recalculated, which is the mean feature value of all keyframes in that cluster. These steps are repeated until the cluster centers no longer change or a preset number of iterations is reached.

[0078] In this embodiment, the final keyframe data is obtained through cluster analysis of the candidate keyframe data, grouping frames with similar characteristics into one category. Then, a representative frame is selected from each cluster as the keyframe data. Furthermore, to avoid the keyframe data being too dense or sparse, a time interval threshold can be set to ensure a reasonable temporal distribution of the keyframe data.

[0079] By employing cluster analysis to obtain the final keyframe data, the accuracy of keyframe data detection was improved.

[0080] S200: The keyframe data is segmented using the first word segmenter and the second word segmenter respectively to obtain the first text feature and the first image feature. Target detection is performed on the image in the keyframe data to obtain the first position feature of the target. The intermediate frame data is segmented using the first word segmenter and the second word segmenter respectively to obtain the second text feature and the second image feature. Target detection is performed on the image in the intermediate frame data to obtain the second position feature of the target.

[0081] In some alternative implementations, such as Figure 5 The keyframe data is segmented using a first word segmenter and a second word segmenter to obtain first text features and first image features. Target detection is then performed on the image within the keyframe data to obtain the first location features of the target, including:

[0082] S210. The first word segmenter segments the keyframe data image into multiple image blocks and calculates the average gray value of the image blocks to obtain the first image feature; S220. The second word segmenter maps the average gray value to the corresponding token according to the gray range of the gray value to obtain the first text feature; S230. The image sequence of the keyframe data is input into the YOLOv8 model for target detection to obtain the confidence of the target in the keyframe data and the predicted bounding box; S240. The non-maximum suppression algorithm is used to filter the predicted bounding boxes, retaining the bounding boxes with high confidence and low overlap with other bounding boxes, and outputting the predicted target category, bounding box coordinates and confidence to obtain the first location feature.

[0083] In this embodiment, the first word segmenter uses the T5 word segmenter to perform text word segmentation and extract text features, while the second word segmenter uses the Magvit-V2 word segmenter as a video word segmenter to extract image features from the video data.

[0084] For example, suppose the keyframe data is a 64x64 grayscale image (in practice, the Magvit-V2 word segmenter can also handle color videos and color images). Each pixel in the image is represented by an integer between 0 and 255, indicating its grayscale value. The image is divided into several small image patches, such as 4x4 patches. The average grayscale value of each 4x4 patch is calculated. The grayscale value range of 0 to 255 is divided into 32 intervals. Based on the interval in which each average grayscale value falls, it is mapped to the corresponding "token" (which can be understood as a discrete symbol). For example, if the average grayscale value of a 4x4 patch is 128, and it falls in the 16th interval, then the 16th token is used to represent this patch. Tokens are the discrete representations generated for the image by the Magvit-V2 image-video joint word segmenter. Subsequent language models can then learn and generate based on these tokens, such as generating text describing the content of the image, or generating similar images based on the text.

[0085] By employing the Magvit-V2 image-video joint segmenter, it becomes possible to segment images and videos using a shared vocabulary, which facilitates the effective use of the model to process and generate visual information.

[0086] In some alternative implementations, such as Figure 6 ,yes Figure 1 A flowchart of a specific implementation of step S200 is provided, which describes how S200 uses a first word segmenter and a second word segmenter to segment the intermediate frame data to obtain second text features and second image features, and performs target detection on the image in the intermediate frame data to obtain the second location features of the target. Step S200 includes:

[0087] S250. The image of the intermediate frame data is segmented into multiple image blocks using the first word segmenter, and the average gray value of the image blocks is calculated to obtain the second image feature; S260. The image of the intermediate frame data is mapped to the corresponding token according to the gray range of the average gray value using the second word segmenter to obtain the second text feature; S270. The image sequence of the intermediate frame data is input into the YOLOv8 model for target detection to obtain the confidence of the target in the intermediate frame data and the predicted bounding box; S280. The predicted bounding box is filtered using the non-maximum suppression algorithm, retaining the bounding boxes with high confidence and low overlap with other bounding boxes, and outputting the predicted target category, bounding box coordinates and confidence to obtain the second location feature.

[0088] The YOLOv8 model takes keyframe data and intermediate frame data as input for object detection, dividing the input image into grid cells. For example, the image can be divided into grid cells of different sizes, such as 7×7 or 19×19. Multiple bounding boxes are predicted for each grid cell. Each bounding box contains five parameters: center coordinates, width, height, and confidence. The confidence reflects the probability that the bounding box contains the target, as well as the accuracy of the bounding box; its value is between 0 and 1.

[0089] Non-maximum suppression (NMS): This algorithm filters out redundant bounding boxes from multiple predicted bounding boxes. Based on the confidence level and overlap of the bounding boxes, NMS retains those with higher confidence and lower overlap with other bounding boxes to prevent the same target from being detected repeatedly by multiple bounding boxes.

[0090] By inputting image sequences containing keyframe data and intermediate frame data into the YOLOv8 model for target detection, information such as target category, bounding box coordinates, and confidence level is obtained, thereby completing the detection and localization of targets in the image. This provides more detailed training data for the autoregressive task model (Prefix LLM model) in this application embodiment, which is beneficial to improving the accuracy of video generation by the autoregressive task model (Prefix LLM model) and generating more details in the required video.

[0091] S300. The autoregressive task model is trained and learned using the first text feature, first image feature, and first position feature of the acquired keyframe data and the second text feature, second image feature, and second position feature of the intermediate frame data, to obtain a converged target autoregressive task model.

[0092] S400: Utilize preset prefix features for description and input keyframes into the target autoregressive task model to generate animation videos.

[0093] In this embodiment, an autoregressive task model (Prefix LLM model) is trained by detecting keyframe data from a large amount of video data. Based on the traditional language model structure, the concept of a prefix is ​​introduced. The prefix typically consists of a series of special tokens, which may contain information about the task, context, or specific instructions.

[0094] When processing text, the autoregressive task model (Prefix LLM model) in this application embodiment considers both the input text content and the information in the prefix to better understand the task requirements and generate scene videos that meet the requirements.

[0095] For example, based on prefix feature descriptions and input keyframe image data, virtual demonstrations and simulation scenarios can be created to showcase a company's products, services, or solutions. Specifically, a simulated insurance claims process can be generated so that customers understand how to apply for a claim and are familiar with the relevant procedures and requirements.

[0096] In some alternative implementations, such as Figure 7 , Figure 7 yes Figure 1 A flowchart of the first specific implementation of S400; S400 includes: S410, inputting keyframes into an autoregressive task model to obtain feature information of keyframe data images and intermediate frame data images; S420, predicting the next frame image using a preset Prefix feature description based on the feature information of keyframe data images and intermediate frame data images; S430, adding the predicted image frame to the frame sequence composed of keyframes and intermediate frames to obtain an animation video.

[0097] Prefix descriptions are prefixes, which can also be understood as conditions for generating animated videos. They typically consist of a series of special tokens, which can contain information about the task, context, or specific instructions. Prefix descriptions guide the video generation process and can include information about the video's theme, style, scene, actions, etc., or specific instructions, such as "generate a video about animals running in a forest." The settings for prefix descriptions need to be determined based on the specific task and requirements.

[0098] In this embodiment, the autoregressive task refers to the model predicting the next image frame or video frame based on given conditional information, keyframe data, and intermediate value data.

[0099] By utilizing an autoregressive task based on a prefix LLM model, patterns and regularities in video data can be learned, and new video content can be generated based on given prefix feature descriptions. This allows the representational and learning capabilities of language models, along with an autoregressive approach, to progressively generate coherent, high-quality, and realistic videos.

[0100] To address the aforementioned problems, this application also proposes a keyframe-based animation video generation device 500, employing a keyframe-based animation video generation method, such as... Figure 8 , Figure 8 A schematic diagram of the module structure of a keyframe-based animation video generation device according to this application; including:

[0101] The acquisition module 501 is used to acquire training video data, preprocess the acquired training video data to obtain a video frame sequence, perform keyframe detection on the video frame sequence, acquire keyframe data, and generate intermediate frame data based on the keyframe data.

[0102] The feature extraction module 502 is used to perform word segmentation on the keyframe data using the first word segmenter and the second word segmenter respectively to obtain the first text feature and the first image feature, to perform target detection on the image in the keyframe data to obtain the first position feature of the target, to perform word segmentation on the intermediate frame data using the first word segmenter and the second word segmenter respectively to obtain the second text feature and the second image feature, to perform target detection on the image in the intermediate frame data to obtain the second position feature of the target;

[0103] Training module 503 is used to train the autoregressive task model using the first text features, first image features, and first position features of the acquired keyframe data and the second text features, second image features, and second position features of the intermediate frame data, so as to obtain a converged target autoregressive task model.

[0104] The generation module 504 is used to generate animation videos by using preset prefix feature descriptions and inputting keyframes into the target autoregressive task model.

[0105] To solve the above problems, such as Figure 9 , Figure 9 A schematic diagram of the module structure of a computer device according to this application. Embodiments of this application also propose a computer device including a memory and a processor. The memory stores computer-readable instructions, and the processor, when executing the computer-readable instructions, implements the steps of a keyframe-based animation video generation method.

[0106] To address the aforementioned issues, this application also proposes a computer-readable storage medium storing computer-readable instructions. When these instructions are executed by a processor, they implement the steps of a keyframe-based animation video generation method.

[0107] To address the aforementioned issues, this application also proposes a computer device that employs the following technical solution: it includes a memory, a communication module, and a processor. The memory stores computer-readable instructions, and the processor executes these computer-readable instructions to implement a keyframe-based animation video generation method.

[0108] This computer device can be a computer, server, workstation, or other similar device; it can also be a mobile device such as a mobile phone, tablet, or in-vehicle mobile terminal; or other devices with program execution capabilities. Its internal structure diagram can be as follows: Figure 9 As shown, Figure 9This is a schematic diagram of a computer device according to an embodiment of the present application. The computer device includes a processor, a memory, and a communication module. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores an operating system, instructions, or code. The internal memory provides an environment for the operation of the operating system and instructions or code in the non-volatile storage media. When the instructions or code are executed by the processor, they implement the functions or steps of a fine-tuning method for a pre-trained model as described above. The communication module of the computer device may include a network interface and / or a wireless communication module, allowing the computer device to communicate with other devices or service platforms. Furthermore, the computer device may also include a display screen and input devices, etc.

[0109] The memory stores computer programs, which include program instructions. The processor is configured to call these program instructions. When the processor executes the instructions or code, it performs the following steps: S100: Acquire training video data; preprocess the acquired training video data to obtain a video frame sequence; perform keyframe detection on the video frame sequence to obtain keyframe data; and generate intermediate frame data based on the keyframe data; S200: Perform word segmentation on the keyframe data using a first word segmenter and a second word segmenter respectively to obtain first text features and first image features; perform target detection on the image in the keyframe data to obtain the first location feature of the target. The process involves: S300, using a first word segmenter and a second word segmenter to segment the intermediate frame data to obtain second text features and second image features; S400, performing target detection on the image in the intermediate frame data to obtain the second position features of the target; S301, using the first text features, first image features, and first position features of the obtained keyframe data, and the second text features, second image features, and second position features of the intermediate frame data to train the autoregressive task model, resulting in a converged target autoregressive task model; S402, using preset prefix features and inputting the keyframes into the target autoregressive task model to generate an animation video.

[0110] To address the aforementioned problems, this application also proposes a computer-readable storage medium, employing the following technical solution:

[0111] A computer-readable storage medium stores a computer program, which includes program instructions that, when executed by a processor, cause the processor to perform an adaptive parameter-efficient fine-tuning method.

[0112] This application also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which are implemented when executed by a processor. Figures 1 to 7The following steps provide a keyframe-based animation video generation method, the implementation of which can be found in the steps described above, and will not be repeated here.

[0113] The aforementioned computer-readable storage medium may be a keyframe-based animation video generation apparatus provided in any of the foregoing embodiments, or an internal storage unit of the aforementioned terminal device, such as a hard disk or memory of a computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the computer device.

[0114] Furthermore, the computer-readable storage medium may include both internal storage units and external storage devices of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or will be output.

[0115] However, it should be understood that it is not required to implement all the components shown; more or fewer components may be implemented instead. Those skilled in the art will understand that the computer device described herein is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0116] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0117] The software tools or components not belonging to our company that appear in the embodiments of this application are merely examples and do not represent actual use.

[0118] Compared with existing technologies, this method generates an animation video by detecting keyframes in video data, generating intermediate frame data based on the keyframe data, and creating a training dataset. The text and image features of the training dataset are then segmented into words, and the image features are used for object detection to obtain their positional features. This data is then used to train an autoregressive task model for animation video generation. As a result, animation videos can be generated based on preset prefix feature descriptions, thus realizing the generation of required animation videos using keyframes.

[0119] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for generating animated videos based on keyframes, characterized in that it includes: Acquire training video data, preprocess the acquired training video data to obtain a video frame sequence, perform keyframe detection on the video frame sequence to obtain keyframe data, and generate intermediate frame data based on the keyframe data. The keyframe data is segmented using a first word segmenter and a second word segmenter to obtain first text features and first image features. Target detection is performed on the image in the keyframe data to obtain the first position feature of the target. The intermediate frame data is segmented using a first word segmenter and a second word segmenter to obtain second text features and second image features. Target detection is performed on the image in the intermediate frame data to obtain the second position feature of the target. The autoregressive task model is trained and learned using the first text features, first image features, and first position features of the acquired keyframe data and the second text features, second image features, and second position features of the intermediate frame data, to obtain a converged target autoregressive task model. Animation videos are generated by using preset prefix features and inputting keyframes into the target autoregressive task model.

2. The keyframe-based animation video generation method according to claim 1, characterized in that, Before acquiring training video data, preprocessing the acquired training video data to obtain a video frame sequence, performing keyframe detection on the video frame sequence to obtain keyframe data, and generating intermediate frame data based on the keyframe data, the method further includes: The resolution of the acquired video data is adjusted, and the colors are normalized to obtain standard video data. The standard video data is edited to obtain the training video data.

3. The keyframe-based animation video generation method according to claim 1, characterized in that, the step of acquiring training video data and preprocessing the acquired training video data to obtain a video frame sequence includes: Decode the training video data to obtain frame-by-frame image sequences; Each frame of the image is standardized to obtain the video frame sequence.

4. The keyframe-based animation video generation method according to claim 3, characterized in that, the step of performing keyframe detection on the video frame sequence to obtain keyframe data includes: Calculate the differences in image features or motion features between two adjacent frames, and determine candidate keyframes based on the calculation results; K-Means clustering analysis is performed on the candidate keyframes to obtain the keyframe data of the video frame sequence.

5. The keyframe-based animation video generation method according to claim 1, characterized in that, the step of segmenting the keyframe data using a first word segmenter and a second word segmenter respectively to obtain first text features and first image features, and performing target detection on the image in the keyframe data to obtain the first position features of the target, includes: The first word segmenter is used to segment the keyframe data image into multiple image blocks, and the average gray value of the image blocks is calculated to obtain the first image feature; The second word segmenter maps the average gray value to the corresponding token based on the gray range in which the gray value is located, and obtains the first text feature; The image sequence of keyframe data is input into the YOLOv8 model for target detection, and the confidence of the target in the keyframe data and the predicted bounding box are obtained. The predicted bounding boxes are filtered using a non-maximum suppression algorithm, retaining those with high confidence and low overlap with other bounding boxes, and the predicted target category, bounding box coordinates, and confidence are output to obtain the first location feature.

6. The keyframe-based animation video generation method according to claim 1, characterized in that, the step of segmenting the intermediate frame data using a first word segmenter and a second word segmenter respectively to obtain second text features and second image features, and performing target detection on the image in the intermediate frame data to obtain the second position features of the target, includes: The first word segmenter is used to segment the image of the intermediate frame data into multiple image blocks, and the average gray value of the image blocks is calculated to obtain the second image feature; The second word segmenter maps the average gray value to the corresponding token based on the gray range in which the gray value is located, thus obtaining the second text feature; The image sequence of the intermediate frame data is input into the YOLOv8 model for target detection, and the confidence score and predicted bounding box of the target in the intermediate frame data are obtained. The non-maximum suppression algorithm is used to filter the predicted bounding boxes, retaining those with high confidence and low overlap with other bounding boxes, and outputting the predicted target category, bounding box coordinates and confidence, thus obtaining the second location feature.

7. The keyframe-based animation video generation method according to claim 1, characterized in that, the step of using a preset prefix feature description and inputting the keyframes into the target autoregressive task model for animation video generation includes: The keyframes are input into the autoregressive task model to obtain the feature information of the keyframe data image and the intermediate frame data image. Based on the feature information of the keyframe data image and the intermediate frame data image, the next frame image is predicted using the preset Prefix feature description; The predicted image frames are added to a frame sequence consisting of keyframes and intermediate frames to obtain the animated video.

8. A keyframe-based animation video generation apparatus, employing the keyframe-based animation video generation method according to any one of claims 1-7, characterized in that, include: The acquisition module is used to acquire training video data, preprocess the acquired training video data to obtain a video frame sequence, perform keyframe detection on the video frame sequence to acquire keyframe data, and generate intermediate frame data based on the keyframe data. The feature extraction module is used to perform word segmentation on the keyframe data using the first word segmenter and the second word segmenter respectively to obtain the first text feature and the first image feature; to perform target detection on the image in the keyframe data to obtain the first position feature of the target; to perform word segmentation on the intermediate frame data using the first word segmenter and the second word segmenter respectively to obtain the second text feature and the second image feature; and to perform target detection on the image in the intermediate frame data to obtain the second position feature of the target. The training module is used to train and learn the autoregressive task model using the first text features, first image features, and first position features of the acquired keyframe data and the second text features, second image features, and second position features of the intermediate frame data, so as to obtain a converged target autoregressive task model. The generation module is used to describe the target autoregressive task model using preset prefix features and input keyframes to generate animation videos.

9. A computer device, characterized in that, The method includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the steps of the keyframe-based animation video generation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing computer-readable instructions thereon, characterized in that, When the computer-readable instructions are executed by a processor, they implement the steps of the keyframe-based animation video generation method as described in any one of claims 1 to 7.

Images