An optimization method for long video generation based on tree coding

Abstract

The application discloses an optimization method for long video generation based on tree coding, comprising: constructing a video frame sequence and obtaining frame features; clustering frame features and obtaining correlation scores between key frames and video queries; calculating the number of high correlation clusters and outputting corresponding frame feature sets and correlation scores; taking the key frames of all high correlation clusters as the first layer nodes of a video feature tree, clustering the high correlation clusters into w sub-clusters respectively, constructing a second layer based on the key frames of the sub-clusters, and taking the correlation score of the second layer nodes as the correlation score of the first layer nodes minus one; constructing each layer according to the above method until the correlation score of the last layer node is one; a large language model optimizes inference results based on the output of the speculation sampling principle and the draft model; and inputting the optimized inference results and the video frame sequence into a preset video generation model to obtain an answer video. The application improves inference accuracy and efficiency through adaptive query and hierarchical long video understanding.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of video processing, and particularly relates to an optimization method for long video generation based on tree coding. BACKGROUND

[0002] Video-language understanding tasks have always focused on short video clips, and it is often difficult to handle the complexity of long video understanding. Although the field of video-language understanding has made significant progress in video question answering, past research has mainly focused on short video tasks of 5 to 15 seconds. With the popularity of minute-level video content and the importance of long human behavior analysis and film analysis applications, it becomes increasingly important to develop models that can reason on long videos and answer questions. However, most modern models developed for short videos perform poorly on long video understanding tasks.

[0003] Recently, many long video-language understanding methods utilize the long sequence reasoning capability of large language models to address the long-range modeling challenge in long video-language understanding. They convert videos into densely sampled frames, use a vision-language model to generate captions for the densely sampled frames, and represent the video in text format. Then, this text representation is input into a large language model, which reasons on the video and responds to the provided query. Although this strategy has shown great potential in video understanding tasks, it still faces three major limitations when applied to long video understanding tasks:

[0004] 1. Information overload: The frames used to generate captions are often redundant and contain irrelevant information, making dense sampling inefficient. Existing methods lack effective ways to deal with information redundancy, making the large language model overwhelmed and leading to reasoning errors. Experiments show that as the number of sampled frames increases, the performance of video tasks actually decreases;

[0005] 2. Lack of query adaptability: Existing methods do not consider the relevance of the query when converting video input into text description, resulting in irrelevant information being passed to the large language model. This suboptimal approach makes it difficult for the large language model to accurately answer specific query questions, affecting the accuracy and efficiency of reasoning;

[0006] 3. Unable to capture coarse and fine-grained video structure: Existing methods often simplify video content into a series of unstructured captions, failing to reflect the hierarchical nature of video information. In particular, in long videos, some information-intensive areas require fine-grained temporal understanding, while others have sparse information or are irrelevant to the query. Therefore, dense sampling methods not only face the problem of information overload, but also ignore the fact that video question answering requires different granularities, easily missing critical information and affecting the completeness and accuracy of video reasoning. SUMMARY

[0007] In view of the problems of low processing efficiency of redundant information and missing key information in existing methods for solving long video understanding, the present application provides an optimization method for long video generation based on tree coding, which provides a query adaptive and hierarchical long video understanding framework to improve inference accuracy and efficiency. To solve the above technical problems, the technical solution adopted by the present application is as follows:

[0008] An optimization method for long video generation based on tree coding, comprising the following steps:

[0009] S1, collecting a video query, constructing a video frame sequence based on the queried video, using a pre-trained visual encoder to obtain the frame features of each video frame in the video frame sequence, and setting the maximum depth θ and breadth of the video feature tree Initializing the number of clusters K;

[0010] S2, using the K-Means clustering method to cluster the frame features obtained in step S1, using a large language model to obtain the relevance score between each cluster and the video query in step S1;

[0011] S3, calculating the number of high-relevance clusters according to the relevance score obtained in step S2 and a preset relevance threshold, and determining If yes, execute K=K+1, and return to step S2, otherwise, output the frame feature set corresponding to the high-relevance clusters and the relevance score of each cluster key frame, and execute step S4, wherein ε represents the number of high-relevance clusters;

[0012] S4, taking the key frames corresponding to all high-relevance clusters output in step S3 as the first layer nodes of the video feature tree, using the K-Means clustering method to cluster all high-relevance clusters into w sub-clusters, and constructing the key frames of the w sub-clusters as the second layer nodes of the corresponding nodes, the relevance score of the second layer nodes is set to the relevance score of the first layer nodes minus one, using the K-Means clustering method to cluster each sub-cluster into w grandchild clusters, and constructing the key frames of the w grandchild clusters as the third layer nodes of the corresponding nodes, the relevance score of the third layer nodes is set to the relevance score of the second layer nodes minus one, and the above method is used to construct each layer node until the relevance score of the last layer node is one, and then the video feature tree is obtained;

[0013] S5, inputting the video feature tree obtained in step S4 and the video query into a plurality of preset draft models, and receiving the outputs of all draft models by a large language model, and generating an optimal inference result based on the principle of speculative sampling.

[0014] The step S2 comprises the following steps:

[0015] S2.1, clustering the frame features obtained in step S1 using a K-Means clustering method to obtain key frames of each cluster;

[0016] S2.2, obtaining the text subtitles of each key frame obtained in step S2.1 using a subtitle generator;

[0017] S2.3, obtaining the relevance score between each text subtitle and the video query using a large language model.

[0018] In step S3, when r k is a high-relevance cluster, and when r k is a low-relevance cluster, where ξ represents a preset relevance threshold, r k is the relevance score between the key frame of the kth cluster and the video query.

[0019] The step S5 includes the following steps:

[0020] S5.1, inputting the video feature tree obtained in step S4 and the video query into a plurality of preset draft models respectively to obtain corresponding first inference results;

[0021] The first inference result of each draft model includes a first inference text composed of a plurality of tokens and a first sampling probability corresponding to each token;

[0022] S5.2, constructing a tree encoding model according to the first inference text obtained in step S5.1, and generating a draft inference matrix based on the tree encoding model;

[0023] S5.3, updating the draft inference matrix using a mask masking method, the large language model outputs a second sampling probability of each token according to the updated draft inference matrix, and scores each token in turn based on the first sampling probability and the second sampling probability using the speculative sampling principle, and generates an optimal inference result based on the token score.

[0024] In step S5.3, before scoring each token in turn based on the first sampling probability and the second sampling probability using the speculative sampling principle, comparing the first sampling probability and the second sampling probability, updating the second sampling probability when Q(x) > p(x) with p'(x) = norm(max(0, p(x)-Q(x))), otherwise, do not update the second sampling probability, Q(x) represents the maximum value of the first sampling probability of token x output by all draft models, and p(x) represents the second sampling probability of token x output by the large language model.

[0025] Further comprising step S6, inputting the optimal reasoning result obtained in step S5 and the video frame sequence obtained in step S1 into a preset video generation model to obtain an answer video related to the video query.

[0026] Advantages of the present application:

[0027] 1. Multi-spectral channel attention fusion unit: by adaptively selecting video frames for caption generation, clustering frames based on frame features, and scoring clusters according to relevance to queries (Query), adaptively selecting and extracting key frames for caption generation. Repeat this process until enough key frames related to the query are extracted, thereby reducing redundant information and improving processing efficiency;

[0028] 2. Hierarchical tree structure: organize visual clusters into a query-adaptive and hierarchical tree structure, the structure of the tree encodes different levels of granularity, and processes relevant segments with higher (deeper) resolution. The hierarchical representation of the video is achieved through the tree structure, allowing the system to flexibly extract and process information according to different query requirements;

[0029] 3. Tree structure-based question and answer: by traversing the key frames of the tree, their captions are passed to a question and answer model service based on a large language model to generate answers to each question. This method allows large language models to more efficiently handle long video questions and answers, improving reasoning accuracy and efficiency;

[0030] 4. Quality improvement mode based on speculative sampling: in the field of generative models, especially in large-scale image generation and video generation tasks, generating high-quality long videos has always been a technical challenge. To solve this problem, the present application proposes a quality improvement mode based on speculative sampling. The core idea of this mode is to first generate a draft model of the large model through speculative sampling. The draft model provides a preliminary, rough generated version as a basis for further refinement and optimization. After the draft model is generated, the system uses multi-GPU deployment to improve computational efficiency through parallel computing to improve overall productivity and better manage and allocate resources, avoiding single GPU overload, thereby greatly improving the efficiency and generation effect of long video generation. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0032] Figure 1 It is a schematic diagram of a video feature tree.

[0033] Figure 2 A flowchart illustrating the process of generating inference results for large language models in existing technologies.

[0034] Figure 3 This is a schematic diagram illustrating the process of generating inference results using a large language model in this invention. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] An optimization method for long video generation based on tree coding includes the following steps:

[0037] S1. Collect video queries, construct a video frame sequence based on the queried video, use a pre-trained visual encoder to obtain the frame features of each video frame in the video frame sequence, and set the maximum depth θ and breadth of the video feature tree. Initialize the number of clusters K;

[0038] To extract key information, existing methods select a fixed number of keyframes as video representations. However, this uniform keyframe selection is suboptimal for a general long video-language understanding framework because it may miss information, include redundant frames, and fail to intelligently allocate frame budgets based on the video and query.

[0039] The video frame sequence is represented by V, where V = (F1, F2, ..., F...). i F I ), F i Let represent the video frame at time step i, where i represents the video length. A pre-trained visual encoder is used to extract the frame features of each video frame, expressed as f. i =E(F i ), where f i ∈R d It is video frame F i The extracted frame features, E(·) represents the visual encoder, and d represents f i This method of frame feature extraction is less costly than using a visual-language model to extract text features such as captions, and is approximately twice as fast because it does not involve autoregressive decoding. The extracted frame features serve as a compact representation of the visual content of each video frame, capturing the diverse semantics of each frame, such as scenes and objects.

[0040] S2, using the K-Means clustering method to cluster the frame features obtained in step S1, and using a large language model to obtain the relevance score between the keyframes of each cluster and the video query in step S1, including the following steps:

[0041] S2.1, use the K-Means clustering method to cluster the frame features obtained in step S1 to obtain the key frames of each cluster;

[0042] To reduce video frame redundancy, visual clustering is first performed to group similar video frames before information extraction. This method uses K-Means clustering to divide the frame features corresponding to the video input into K distinct clusters, with the corresponding expression being:

[0043] (C1, C2, ..., C k C K ), (c1, c2, ..., c k c K )=K-Means((f1, f2,..., f i , ..., f I ), K);

[0044] In the formula, C k Let C represent the k-th cluster. K This represents the Kth cluster, where each cluster contains multiple similar frames; c k Let c represent the centroid vector of the k-th cluster. K f represents the centroid vector of the Kth cluster, where K represents the number of clusters (which can be initialized to a positive integer much smaller than the number of video frames, such as 1 or 2), and can be considered as the first level of granularity in the subsequent tree construction. I Indicates video frame F I The extracted frame features.

[0045] S2.2, Use the subtitle generator to obtain the text subtitles for each keyframe obtained in step S2.1;

[0046] To convert the frame features of a cluster into text information that can be processed by a large language model, this application generates captions for a single keyframe of each cluster. Specifically, for cluster C... k The closest centroid vector c is found based on the distance between the vectors corresponding to data points within the cluster. k Keyframes, and treat them as cluster C k The keyframes of each cluster are then input into a vision-language model-based caption generator to obtain a text caption for each cluster, with the corresponding expression being t. k =Cap(α) k), where Cap(·) denotes a caption generator, t k denotes the text caption corresponding to the key frame of cluster C k k denotes the key frame of cluster C k , and a k ∈ V.

[0047] S2.3, obtaining a relevance score between each text caption and the video query by using a large language model;

[0048] This step inputs all the text captions and the video query Q in the large language model, and outputs a relevance score of each cluster where r k is the relevance score of the kth cluster.

[0049] S3, calculating the number of high-relevance clusters according to the relevance score obtained in step S2 and a preset relevance threshold, and determining If yes, execute K = K + 1, and return to step S2, otherwise, output the frame feature set corresponding to the high-relevance clusters and the relevance score of the key frame of each cluster, and execute step S4, where ε represents the number of high-relevance clusters.

[0050] Specifically, after obtaining the text caption t k of the key frame, a prompt is input to the large language model to make the large language model determine whether the key information extracted by each cluster is sufficient to answer the given video query, and the large language model is required to assign a relevance score from 1 to θ to each text caption, 1 representing the least relevance and θ representing a preset maximum relevance score. When r k > ξ, it is called high relevance, and when r k ≤ ξ, it is called low relevance, where ξ represents a preset relevance threshold, and θ and ξ are both positive integers and θ ≥ ξ. That is, when the relevance score of the key frame of a cluster is greater than ξ, the cluster is called a high-relevance cluster, otherwise, it is called a low-relevance cluster. In this application, ξ = 1.

[0051] This application first sets a maximum value for the number of clusters The value is set according to the system computing resources and model inference ability to balance the computing cost and inference accuracy. At the same time, the number of high-relevance clusters is set (the optimal threshold ε is selected through the validation set) to adaptively extract the key frame information of each cluster. If the number of high-relevance clusters is lower than the requirement, it indicates that the information extracted from the current key frame set is insufficient for the large language model to answer the video query. In this case, the number of clusters K is increased, and the clustering, caption generation and relevance score operations are repeated. If the number of high-relevance clusters meets the requirement or the number of clusters reaches ​The extracted clusters and their keyframes are then appended to the first level of the tree, and the next step is continued.

[0052] S4, as shown in the figure, the keyframes corresponding to all high correlation clusters output by step S3 are taken as the first level nodes of the video feature tree, and the K-Means clustering method is used to cluster all high correlation clusters into w sub-clusters respectively, the keyframes of the w sub-clusters are constructed as the second level nodes of the corresponding nodes, and the correlation score of the second level nodes is set to the correlation score of the first level nodes minus one, and the K-Means clustering method is used to cluster each sub-cluster into w grandchild clusters, and the keyframes of the w grandchild clusters are constructed as the third level nodes of the corresponding nodes, and the correlation score of the third level nodes is set to the correlation score of the second level nodes minus one, and the above method is used to construct the nodes of each level until the correlation score of the last level node is one. Figure 1

[0053] The existing keyframe selection methods usually take the selected frames as an unstructured list, ignoring the internal structure that may exist in the video information. Specifically, these methods ignore that some video area information is rich and needs to be sampled in detail, while other areas irrelevant to the query may only need to be sampled roughly or minimally. The uniform or random sampling method will result in a long, repetitive and chaotic caption list, which may confuse the large language model and ultimately limit its ability to effectively reason, and fail to clearly describe the structure of the video.

[0054] The present application constructs a query-adaptive hierarchical video representation on the basis of the breadth-expanded clusters in the first step. Specifically, the depth of the tree is expanded according to the correlation scores of the K clusters obtained in step S3. Intuitively, for high correlation clusters, the large language model needs more detailed and fine-grained information, while for low correlation clusters, more information may actually lead to irrelevant details, which may overwhelm the large language model and thus lead to incorrect reasoning. The correlation score of the top-level node determines how many layers of more fine-grained information can be extracted from it, i.e., how many child nodes or grandchild nodes the top-level node will have.

[0055] Since the correlation score r is one of [1, 2,..., θ], the video feature tree is expanded according to the remaining correlation of the parent node above. Therefore, the value of θ is equal to the maximum depth of the video feature tree. Specifically, nodes with low correlation (r k = 1) do not need to be expanded further. For nodes with other correlations (r k > 1), the constituent frames are re-clustered into w sub-clusters, where w represents the branch width of the video feature tree, and the value of w is determined based on the correlation score of the cluster, system computing resources and validation set experimental results. For high correlation clusters (r k ​For high correlation clusters, the value of w is increased to capture more details; for low correlation clusters, w is appropriately reduced to reduce the number of generated sub-clusters. These sub-clusters are then added as child nodes of the tree, and are given a correlation that is one less than that of the parent node. By recursively repeating this process, the nodes of the last layer are all made into leaf nodes with a correlation of 1. After the breadth and depth expansion of the tree, an adaptive hierarchical video representation for long video large language model inference is obtained.

[0056] S5, input the video feature tree and the video query obtained in step S4 into a plurality of preset draft models respectively, receive the outputs of all the draft models by the large language model, and generate an optimal inference result based on the speculative sampling principle;

[0057] Ordinary large language models are applied to the process of video generation and inference. Because the large language model is based on autoregressive string inference, it will cause the parallelism to be reduced and the inference speed to be slow. Through the method of speculative sampling, that is, generating a preliminary result by the draft model generated by fine-tuning, and then performing rapid verification based on parallelism, the generation efficiency can be improved. As shown in Figure 2 In the autoregressive model based on string, for example, to generate a sentence of 'This is a big apple', it often needs to perform 5 times of model calculation, and the performance is relatively low. As shown in Figure 3 Through different draft models to calculate and output, the outputs of the draft models are assembled into a matrix in the form of tree encoding, and finally the calculation is performed in the form of model verification, so that the same result can be achieved by using the model only once. The model verification part which consumes the most resources is optimized in the present application, so that the result generation speed can be greatly improved.

[0058] The step S5 includes the following steps:

[0059] S5.1, input the video feature tree and the video query obtained in step S4 into a plurality of preset draft models respectively to obtain corresponding first inference results;

[0060] The first inference result of each draft model includes a first inference text composed of a plurality of tokens and a first sampling probability corresponding to each token.

[0061] S5.2, construct a tree encoding model according to the first inference result obtained in step S5.1, and generate a draft inference matrix based on the tree encoding model;

[0062] Specifically, first, a tree encoding model is constructed based on the first inference text output of all draft models, a path in the tree encoding model corresponds to the output of a draft model, then the columns of the draft inference matrix are confirmed based on all nodes of the tree encoding model, and finally the draft inference matrix is constructed based on all paths of the tree encoding model. For example, the output of the draft model 1 is "This is a big apple", the output of the draft model 2 is "This is a error", and the output of the draft model 3 is "This is not". Through the matrix encoding method, the following matrix can be generated:

[0063] This is a big apple error not Draft Model 1 1 1 1 1 1 0 0 Draft Model 2 1 1 1 0 0 1 0 Draft Model 3 1 1 0 0 0 0 1

[0064] S5.3, the draft inference matrix is updated by using the mask masking method, the large language model outputs the second sampling probability of each token according to the updated draft inference matrix, and each token is scored in turn based on the first sampling probability and the second sampling probability by using the speculation sampling principle, and the optimal inference result is generated based on the token selected based on the score.

[0065] The large language model uses the speculation sampling principle to optimize the inference of the video query based on the updated draft inference matrix and the received video feature tree;

[0066] As shown in the above table, there are a large number of elements with a value of 0 in the draft inference matrix. These elements will bring additional computational overhead in the calculation process and will not have any value, so it is best to avoid them in the calculation. It is worth noting that the output of any draft model corresponds to the starting position and the ending position of its content, for example, the draft model 1 does not need to calculate the two characters "error" and "not".

[0067] The present application cuts the draft inference matrix, uses the mask unit as the draft marker, marks the starting position and the ending position of the output of each draft model, and determines whether each small block needs to be calculated through the marker, so as to avoid the calculation of the mask unit. By cutting first and then calculating, the calculation of the 0 block marker is avoided, which can greatly reduce the GPU calculation load and thus improve the inference speed by 2-3 times.

[0068] In the speculation sampling, the output path is realized by adopting the adoption probability, that is, each token on the tree encoding model is scored. The calculation formula of the score is p(x) is the sampling probability of the token x in the large language model, that is, the second sampling probability, is the sampling probability of the token x output by the i-th draft model, that is, the first sampling probability, ​The maximum value of the sampling probability of token x output by all draft models. When The confidence of the large language model in this token x is lower than that of all draft models, and the reasoning of the draft model may be wrong, so the large language model resamples this token with a probability of p'(x) = norm(max(0, p(x) - Q(x))), updates the second sampling probability p(x) = p'(x), and scores with the updated second sampling probability. Q(x) represents the maximum value of the sampling probability of token x output by all draft models. When The confidence of the large language model in this token x is higher than that of the draft model, and at this time, the large language model will not resample this token, but directly score this token based on p(x). Through the above method, the score of each token on each path is calculated. The large language model selects the token path with the highest score as the final output result of the tree encoding composed of the content checked by the draft model output, that is, the optimal reasoning result.

[0069] S6, input the optimal reasoning result output in step S5 and the video frame sequence obtained in step S1 into the preset video generation model to obtain an answer video related to the video query;

[0070] The video generation model is responsible for converting the text or other intermediate forms generated by the large language model into video content. This step includes complex tasks such as scene understanding of text, picture generation, and animation production. We use Text2Video-Zero as the video generation model, which generates corresponding video sequences from Gaussian noise through a diffusion process based on text prompts. The model learns the temporal dependency between text and video during the generation process, ensuring that the generated video is consistent with the semantics of the text prompt.

[0071] The process of outputting the generated video is essentially processing the original video content to meet the user's specific semantic interpretation needs of the original input video content, so the video frame sequence obtained in step S1 needs to be re-input into the video generation model as the video content base. The model optimizes the video on this basis and merges the corresponding text of the reasoning result into the new video content in the form of subtitles.

[0072] The embodiment of the application also provides an optimization device for long video generation based on tree encoding, comprising:

[0073] The sample construction module is configured to collect a video query, construct a video frame sequence based on a video to be queried, obtain frame features of each video frame in the video frame sequence by using a pre-trained visual encoder,

[0074] The score calculation module is configured to cluster the frame features obtained by the sample construction module by using a K-Means clustering method, and obtain a relevance score between each cluster and the video query obtained by the sample construction module by using a large language model in the large language model storage module.

[0075] The high-relevance cluster acquisition module is configured to calculate the number of high-relevance clusters based on the relevance score obtained by the score calculation module and a preset relevance threshold ξ.

[0076] The key frame output module is configured to determine whether K is less than the maximum breadth of the preset video feature tree. If yes, execute K = K + 1, and sequentially restart the score calculation module and the high-relevance cluster acquisition module, and determine whether K is less than the maximum breadth of the preset video feature tree based on the updated number of high-relevance clusters. Otherwise, output the frame feature set corresponding to the high-relevance clusters and the relevance score of each cluster key frame, and start the feature tree construction module, wherein, K represents the number of clusters of the score calculation module.

[0077] The feature tree construction module is configured to take the key frames corresponding to all the high-relevance clusters output by the key frame output module as first-layer nodes of the video feature tree, cluster all the high-relevance clusters into w sub-clusters by using a K-Means clustering method, construct the key frames of the w sub-clusters as second-layer nodes of the corresponding nodes, set the relevance score of the second-layer nodes to the relevance score of the first-layer nodes minus one, cluster each sub-cluster into w grandchild clusters by using a K-Means clustering method, construct the key frames of the w grandchild clusters as third-layer nodes of the corresponding nodes, set the relevance score of the third-layer nodes to the relevance score of the second-layer nodes minus one, and construct the nodes of the following layers in the above manner until the relevance score of the last-layer nodes is one, thereby generating the video feature tree.

[0078] The draft model construction module is configured to store a plurality of draft models, and output a first inference result related to the video query by using the plurality of draft models based on the video feature tree output by the feature tree construction module and the video query obtained by the sample construction module. The first inference result of each draft model includes a first inference text composed of a plurality of tokens and a first sampling probability corresponding to each token.

[0079] The draft model is a simplified version of the large language model and is used to generate a preliminary content output. Its main function is to quickly provide a preliminary content structure or framework before generating a complete output. These draft models can quickly process input data and generate a rough version of the expected output, providing guidance for subsequent detailed generation steps. Since the computational demand of the draft model is relatively low, allocating 1 to 2 GPU cards to deploy 4 draft models is sufficient to meet its task demand.

[0080] Large language model storage module: for storing a large language model, outputting a second sampling probability for each token based on the first inference text and the first sampling probability output by the draft model construction module, and selecting tokens to generate an optimal inference result according to the first sampling probability and the second sampling using the gambling sampling principle;

[0081] This layer is the core computing part of the system, responsible for generating detailed content and refining from the preliminary results generated by the draft model. The complete large language model has higher computational complexity and can consider more context and details in the inference process, thereby generating high-quality output. This layer requires higher computing resources, therefore, 4 GPUs are allocated for deploying the complete large language model to ensure sufficient computing power and processing speed.

[0082] Video generation model storage module: for storing a video generation model, and outputting a video query related answer video using the video generation model based on the optimal inference text output by the large language model storage module and the video frame sequence constructed by the sample construction module;

[0083] Since the video generation model layer requires special GPU resources to process image and video data generation and rendering work, the present application uses 2 GPUs for deploying the video generation model.

[0084] In the process of efficient video generation and inference, multi-card layered deployment is an effective acceleration strategy. By reasonably configuring and allocating GPU resources, parallel processing of different task modules can be achieved, thereby improving the processing efficiency and response speed of the overall system. The present application embodiment uses 8 GPUs for layered deployment, which can maximize the use of GPU resources and optimize the allocation of computing tasks between layers. The fast response of the draft model layer provides a basis for subsequent detailed processing, while the deep analysis of the large language model layer is responsible for ensuring the accuracy and richness of the output content. The addition of the video generation model layer enables the system to directly generate visual content, meeting the demand for multimedia output. The advantages of this multi-card layered deployment strategy are:

[0085] 1. Task parallelization: different task modules run simultaneously on different GPUs, avoiding single-card bottleneck problems and fully utilizing the parallel processing capabilities of GPUs; 2. Resource optimization: GPU resources are allocated reasonably according to the computing needs of each module, avoiding resource waste while ensuring that each module has sufficient computing power to complete its tasks; 3. Strong scalability: this solution supports flexible expansion and adjustment. For example, the number and configuration of GPUs in each layer can be adjusted according to specific task requirements and GPU performance characteristics, adapting to different application scenarios and loads.

[0086] In summary, by using the multi-card layered deployment method, the inference process is accelerated, and the overall processing capacity and stability of the system are improved. This method is particularly suitable for large model inference tasks that require real-time processing and multimedia generation, providing an effective solution to improve user experience and meet high performance requirements.

[0087] The embodiment of the present application also provides an electronic device, which comprises a processor and a memory, and the memory stores a computer program. When the computer program is executed by the processor, the steps of the optimization method for long video generation based on tree coding are executed.

[0088] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program. When the computer program is executed by the processor, the steps of the optimization method for long video generation based on tree coding are executed. Specifically, the storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the storage medium is executed, the embodiment of the optimization method for long video generation based on tree coding can be executed.

[0089] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An optimized method for long video generation based on tree coding, characterized in that, Includes the following steps: S1. Collect video queries, construct a video frame sequence based on the queried video, use a pre-trained visual encoder to obtain the frame features of each video frame in the video frame sequence, and set the maximum depth θ and breadth of the video feature tree. Initialize the number of clusters K; S2, use the K-Means clustering method to cluster the frame features obtained in step S1, and use the large language model to obtain the relevance score between the key frames of each cluster and the video query in step S1. S3, Calculate the number of highly correlated clusters based on the correlation score obtained in step S2 and the preset correlation threshold, and determine... If yes, execute K = K + 1 and return to step S2; otherwise, output the frame feature set corresponding to the high-relevance clusters and the relevance score of each cluster keyframe, and execute step S4, where ε represents the number of high-relevance clusters. S4. Take the keyframes corresponding to all the high-correlation clusters output in step S3 as the first-level nodes of the video feature tree. Use the K-Means clustering method to cluster all the high-correlation clusters into w sub-clusters. Use the keyframes of the w sub-clusters to construct the second-level nodes of the corresponding nodes. Set the correlation score of the second-level nodes to the correlation score of the first-level nodes minus one. Use the K-Means clustering method to cluster each sub-cluster into w grandchild clusters. Use the keyframes of the w grandchild clusters to construct the third-level nodes of the corresponding nodes. Set the correlation score of the third-level nodes to the correlation score of the second-level nodes minus one. Construct each level of nodes in the above way until the correlation score of the last level of nodes is one, thus obtaining the video feature tree. S5. Input the video feature tree and video query obtained in step S4 into several preset draft models respectively. The large language model receives the output of all draft models and generates the optimal inference result based on the speculative sampling principle.

2. The optimized method for long video generation based on tree coding according to claim 1, characterized in that, Step S2 includes the following steps: S2.1, use the K-Means clustering method to cluster the frame features obtained in step S1 to obtain the key frames of each cluster; S2.2, Use the subtitle generator to obtain the text subtitles for each keyframe obtained in step S2.1; S2.3 utilizes a large language model to obtain a relevance score between each text caption and video query.

3. The optimized method for long video generation based on tree coding according to claim 1, characterized in that, In step S3, when r k When r > ξ, it indicates that the k-th cluster is a highly correlated cluster. k When r ≤ ξ, it indicates that the k-th cluster is a low-correlation cluster, where ξ represents the preset correlation threshold, and r k It is the relevance score between the keyframe of the k-th cluster and the video query.

4. The optimized method for long video generation based on tree coding according to claim 1, characterized in that, Step S5 includes the following steps: S5.1 Input the video feature tree and video query obtained in step S4 into several preset draft models to obtain the corresponding first inference results; The first inference result of each draft model includes a first inference text consisting of several tokens and a first sampling probability corresponding to each token; S5.2, Construct a tree coding model based on the first reasoning text obtained in step S5.1, and generate a draft reasoning matrix based on the tree coding model; In S5.3, the draft inference matrix is ​​updated using the masking method. The large language model outputs the second sampling probability of each token based on the updated draft inference matrix. Based on the first and second sampling probabilities, the speculative sampling principle is used to score each token in turn. The token is selected based on the score to generate the optimal inference result.

5. The optimized method for long video generation based on tree coding according to claim 4, characterized in that, In step S5.3, before scoring each token sequentially using the speculative sampling principle based on the first and second sampling probabilities, the first and second sampling probabilities are compared. When Q(x) > p(x), p is used as the basis for scoring. ′ The function (x) = norm(max(0, p(x) - Q(x))) updates the second sampling probability; otherwise, it does not update the second sampling probability. Q(x) represents the maximum value of the first sampling probability of tokenx output by all draft models, and p(x) represents the second sampling probability of tokenx output by the large language model.

6. The optimized method for generating long videos based on tree coding according to claim 4 or 5, characterized in that, It also includes step S6, which inputs the optimal reasoning result obtained in step S5 and the video frame sequence obtained in step S1 into a preset video generation model to obtain a response video related to the video query.

Images