Optimization method for long video generation based on tree coding

Abstract

The invention discloses a tree coding-based long video generation optimization method, which comprises the following steps of: constructing a video frame sequence, and obtaining frame features; clustering the frame features to obtain a correlation score between the key frame and the video query; calculating the number of high-correlation clusters, and outputting a corresponding frame feature set and a correlation score; key frames of all the high-correlation clusters serve as first-layer nodes of a video feature tree, the high-correlation clusters are clustered into w sub-clusters respectively, a second layer is constructed based on the sub-cluster key frames, the correlation score of the second-layer nodes is obtained by subtracting one from the correlation score of the first-layer nodes, each layer is constructed according to the method, and the first-layer nodes and the second-layer nodes are obtained; the relevance score of the last layer of nodes is 1; the large language model outputs an optimization reasoning result based on the speculation sampling principle and the draft model; and inputting the optimized reasoning result and the video frame sequence into a preset video generation model to obtain an answer video. According to the method, the reasoning accuracy and efficiency are improved by inquiring self-adaptive and hierarchical long video understanding.

Description

Technical Field

[0001] The invention belongs to the technical field of video processing, and in particular relates to an optimization method for long video generation based on tree coding. Background Art

[0002] Video-language understanding tasks have traditionally focused on short video clips, often struggling to handle the complexity of understanding longer videos. While the field of video-language understanding has made significant progress in video question answering, past research has focused on short video tasks of 5 to 15 seconds. With the prevalence of minute-long video content and the importance of applications such as long-duration human behavior analysis and movie analysis, it becomes increasingly important to develop models that can reason about and answer questions on long videos. However, most modern models developed for short videos perform poorly on long video understanding tasks.

[0003] Recently, many long video-language understanding methods have used the long sequence reasoning ability of large language models to solve the long-range modeling challenges in long video-language understanding. The video is converted into densely sampled frames, and the visual-language model is used to generate captions for the densely sampled frames to represent the video in text format. This text representation is then input into the large language model, which infers the video and responds to the provided query. Although this strategy has shown great potential in video understanding tasks, it still faces three major limitations when applied to long video understanding tasks:

[0004] 1. Information overload: The frames used to generate subtitles are often redundant and contain irrelevant information, making dense sampling inefficient. Existing methods lack effective methods to deal with information redundancy, which overloads large language models and leads to reasoning errors. Experiments show that the performance of video tasks actually decreases as the number of sampled frames increases;

[0005] 2. Lack of query adaptability: Existing methods do not consider the relevance of the query when converting video input into text descriptions, resulting in irrelevant information being passed to the large language model. This suboptimal approach makes it difficult for the large language model to accurately answer specific query questions, affecting the accuracy and efficiency of reasoning;

[0006] 3. Unable to capture coarse-grained and fine-grained video structures: Existing methods usually simplify video content into a series of unstructured subtitles, failing to reflect the hierarchy of video information. Especially in long videos, some information-dense areas require fine-grained temporal understanding, while other areas are information-sparse or irrelevant to the query. Therefore, dense sampling methods not only face the problem of information overload, but also ignore the fact that video question answering requires different granularities, which easily misses key information and affects the completeness and accuracy of video reasoning. Summary of the invention

[0007] In order to solve the problems of low efficiency in redundant information processing and missing key information in the existing methods for solving the long video understanding, the present invention proposes an optimization method for long video generation based on tree coding, which improves the reasoning accuracy and efficiency by providing a query adaptive and hierarchical long video understanding framework. In order to solve the above technical problems, the technical solutions adopted by the present invention are as follows:

[0008] An optimization method for long video generation based on tree coding comprises the following steps:

[0009] S1, collect video queries, build a video frame sequence based on the queried video, use the pre-trained visual encoder to obtain the frame features of each video frame in the video frame sequence, and set the maximum depth θ and width of the video feature tree Initialize the number of clusters K;

[0010] S2, clustering the frame features obtained in step S1 using the K-Means clustering method, and obtaining the correlation score between the key frame of each cluster and the video query in step S1 using the large language model;

[0011] S3, calculating the number of high correlation clusters according to the correlation score obtained in step S2 and the preset correlation threshold, and judging If yes, execute K=K+1 and return to step S2; otherwise, output the frame feature set corresponding to the high correlation cluster and the correlation score of each cluster key frame, and execute step S4, where ε represents the number of high correlation clusters;

[0012] S4, using the key frames corresponding to all the high correlation clusters outputted in step S3 as the first-layer nodes of the video feature tree, clustering all the high correlation clusters into w sub-clusters using the K-Means clustering method, constructing the key frames of the w sub-clusters as the second-layer nodes of the corresponding nodes, setting the correlation score of the second-layer nodes to the correlation score of the first-layer nodes minus one, clustering each sub-cluster into w grandchild clusters using the K-Means clustering method, constructing the key frames of the w grandchild clusters as the third-layer nodes of the corresponding nodes, setting the correlation score of the third-layer nodes to the correlation score of the second-layer nodes minus one, constructing each layer of nodes according to the above method until the correlation score of the last layer of nodes is one, thereby obtaining the video feature tree;

[0013] S5, input the video feature tree and video query obtained in step S4 into several preset draft models respectively, the large language model receives the outputs of all the draft models, and generates the optimal inference result based on the principle of speculative sampling.

[0014] The step S2 comprises the following steps:

[0015] S2.1, clustering the frame features obtained in step S1 using the K-Means clustering method to obtain the key frame of each cluster;

[0016] S2.2, using a subtitle generator to obtain text subtitles for each key frame obtained in step S2.1;

[0017] S2.3, use a large language model to obtain the relevance score between each text subtitle and the video query.

[0018] In step S3, when r k >ξ, it means that the kth cluster is a high correlation cluster. k ≤ξ, it means that the kth cluster is a low correlation cluster, where ξ represents the preset correlation threshold and r k is the relevance score between the keyframe of the kth cluster and the video query.

[0019] The step S5 comprises the following steps:

[0020] S5.1, inputting the video feature tree and the video query obtained in step S4 into a plurality of preset draft models to obtain corresponding first inference results;

[0021] The first inference result of each draft model includes a first inference text composed of a number of tokens and a first sampling probability corresponding to each token;

[0022] S5.2, constructing a tree coding model according to the first inference text obtained in step S5.1, and generating a draft inference matrix based on the tree coding model;

[0023] S5.3, use the mask method to update the draft inference matrix, the large language model outputs the second sampling probability of each token according to the updated draft inference matrix, and scores each token in turn based on the first sampling probability and the second sampling probability using the speculative sampling principle, and selects the token based on the score to generate the optimal inference result.

[0024] In step S5.3, in step S5.3, before scoring each token in turn based on the first sampling probability and the second sampling probability using the speculative sampling principle, the first sampling probability and the second sampling probability are compared. When Q(x)>p(x), the second sampling probability is updated with p′(x)=norm(max(0,p(x)-Q(x))). Otherwise, the second sampling probability is not updated. Q(x) represents the maximum value of the first sampling probability of token x output by all draft models, and p(x) represents the second sampling probability of token x output by the large language model.

[0025] The method further includes step S6, wherein the optimal inference result obtained in step S5 and the video frame sequence obtained in step S1 are input into a preset video generation model to obtain an answer video related to the video query.

[0026] Beneficial effects of the present invention:

[0027] 1. Multi-spectral channel attention fusion unit: adaptively select video frames for subtitle generation, cluster frames based on frame features, score clusters based on relevance to the query, and adaptively select and extract key frames for subtitle generation. Repeat this process until enough key frames relevant to the query are extracted, thereby reducing redundant information and improving processing efficiency;

[0028] 2. Hierarchical tree structure: Organize visual clusters into a query-adaptive and hierarchical tree structure. The tree structure encodes different levels of granularity and processes related segments at higher (deeper) resolution. The hierarchical representation of the video is achieved through the tree structure, allowing the system to flexibly extract and process information according to different query requirements;

[0029] 3. Tree-structured question answering: By traversing the key frames of the tree, their captions are passed to the question answering model service built on the large language model to generate the answer to each question. This method enables the large language model to process long video question answering more efficiently, improving inference accuracy and efficiency;

[0030] 4. Quality improvement model based on speculative sampling: In the field of generative models, especially in large-scale image generation and video generation tasks, generating high-quality long videos has always been a technical challenge. In order to solve this problem, the present invention proposes a quality improvement model based on speculative sampling. The core idea of ​​this model is to first generate a draft model of a large model through a speculative sampling method. The draft model provides a preliminary, rough generation version as the basis for further refinement and optimization. After the draft model is generated, the system will use multi-graphics card deployment to improve computing efficiency through parallel computing to improve overall productivity, and better manage and allocate resources to avoid overloading a single graphics card, thereby greatly improving the efficiency and generation effect of long video generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 Schematic diagram of video feature tree.

[0033] Figure 2 Schematic diagram of the process of generating inference results for a large language model in the prior art.

[0034] Figure 3 A schematic diagram of the process of generating inference results for a large language model in the present invention. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] An optimization method for long video generation based on tree coding comprises the following steps:

[0037] S1, collect video queries, build a video frame sequence based on the queried video, use the pre-trained visual encoder to obtain the frame features of each video frame in the video frame sequence, and set the maximum depth θ and width of the video feature tree Initialize the number of clusters K;

[0038] To extract key information, existing methods select a fixed number of keyframes as video representation. However, this uniform keyframe selection is suboptimal for a general long video-language understanding framework because it may miss information, contain redundant frames, and cannot intelligently allocate frame budgets based on the video and query.

[0039] The video frame sequence is represented by V, V = (F1, F2, ..., F i , ..., F I ), F i represents the video frame corresponding to the i time step, and I represents the video length. The pre-trained visual encoder is used to extract the frame features of each video frame, and the corresponding expression is f i =E(F i ), where f i ∈R d is the video frame F i The extracted frame features, E(·) represents the visual encoder, and d represents f i This method of frame feature extraction is cheaper than using a visual-language model to extract text features such as subtitles, and because frame feature extraction does not involve autoregressive decoding, it is about 2 times faster. The extracted frame features are a compact representation of the visual content of each video frame, which can capture the diverse semantics of each video frame, such as scenes and objects.

[0040] S2, clustering the frame features obtained in step S1 using the K-Means clustering method, and obtaining the correlation score between the key frame of each cluster and the video query in step S1 using the large language model, including the following steps:

[0041] S2.1, clustering the frame features obtained in step S1 using the K-Means clustering method to obtain the key frame of each cluster;

[0042] In order to reduce the redundancy of video frames, a visual clustering operation is first performed to group similar video frames before extracting information. This method uses the K-Means clustering method to divide the frame features corresponding to the video input into K different clusters. The corresponding expression is:

[0043] (C1, C2, ..., C k , ..., C K ), (c1, c2, ..., c k , ..., c K )=K-Means((f1, f2,..., f i , ..., f I ), K);

[0044] In the formula, C k represents the kth cluster, C K represents the Kth cluster, each of which contains multiple similar frames; c k represents the centroid vector of the kth cluster, c K represents the centroid vector of the Kth cluster, K represents the number of clusters (its initialization value can be set to a positive integer such as 1 or 2 that is much smaller than the number of video frames), which can be regarded as the first-level granularity of the tree established later. I Represents the video frame F I The extracted frame features.

[0045] S2.2, using a subtitle generator to obtain text subtitles for each key frame obtained in step S2.1;

[0046] In order to convert the cluster frame features into text information that can be processed by the large language model, this application generates subtitles for a single key frame of each cluster. Specifically, for cluster C k , find the closest centroid vector c based on the distance between the vectors corresponding to the data points in the cluster k The key frames of cluster C k Then, the key frames of each cluster are input into the subtitle generator based on the visual-language model to obtain a text subtitle for each cluster. The corresponding expression is t k =Cap(α k), where Cap(·) represents the subtitle generator, t k Represents cluster C k The text subtitle corresponding to the key frame of α k Represents cluster C k The key frame of α k ∈V.

[0047] S2.3, using a large language model to obtain the relevance score between each text subtitle and the video query;

[0048] This step will subtitle all the text in the previous step The video query Q is input into the large language model and the relevance score of each cluster is output. Among them, r k is the relevance score of the kth cluster.

[0049] S3, calculating the number of high correlation clusters according to the correlation score obtained in step S2 and the preset correlation threshold, and judging If yes, execute K=K+1 and return to step S2; otherwise, output the frame feature set corresponding to the high correlation cluster and the correlation score of each cluster key frame, and execute step S4, where ε represents the number of high correlation clusters;

[0050] Specifically, after obtaining the text subtitle of the key frame t k Finally, a prompt is input to the large language model to make the large language model decide whether the key information extracted by each cluster is sufficient to answer the given video query. The large language model is required to assign a relevance score from 1 to θ to each text subtitle, where 1 represents the least relevant and θ represents the preset maximum relevance score. k >ξ, it is called high correlation. k ≤ξ, it is called low correlation, where ξ represents the preset correlation threshold, θ and ξ are both positive integers, and θ≥ξ. That is, when the correlation score of a cluster's key frames is greater than ξ, the cluster is called a high correlation cluster, otherwise, it is called a low correlation cluster. In this application, ξ=1.

[0051] This application first sets a maximum value for the number of clusters The value of is set according to the system computing resources and model reasoning capabilities to balance the computing overhead and reasoning accuracy. At the same time, the number of high-correlation clusters is set (the optimal threshold ε is selected through the validation set) to adaptively extract the key frame information of each cluster. If the number of high-correlation clusters is lower than the requirement, this indicates that the information extracted from the current key frame set is insufficient for the large language model to answer the video query. In this case, this method increases the number of clusters K and repeats the clustering, subtitle generation, and relevance scoring operations. If the number of high-correlation clusters meets the requirement or the number of clusters reaches Then append the extracted clusters and their keyframes to the first level of the tree and proceed to the next step.

[0052] S4, such as Figure 1 As shown, the key frames corresponding to all high-correlation clusters output in step S3 are used as the first-layer nodes of the video feature tree, and the K-Means clustering method is used to cluster all high-correlation clusters into w sub-clusters, and the key frames of the w sub-clusters are constructed as the second-layer nodes of the corresponding nodes. The relevance score of the second-layer nodes is set to the relevance score of the first-layer nodes minus one. The K-Means clustering method is used to cluster each sub-cluster into w grandchild clusters, and the key frames of the w grandchild clusters are constructed as the third-layer nodes of the corresponding nodes. The relevance score of the third-layer nodes is set to the relevance score of the second-layer nodes minus one. The following layers of nodes are constructed according to the above method until the relevance score of the last layer of nodes is one;

[0053] Existing keyframe selection methods usually treat the selected frames as an unstructured list, ignoring the possible internal structure in the video information. Specifically, these methods ignore that some video regions are rich in information and require detailed sampling, while other regions that are irrelevant to the query may only require coarse or minimal sampling. Uniform or random sampling methods result in long, repetitive, and messy caption lists, which may confuse large language models, ultimately limiting their ability to reason effectively and failing to clearly describe the structure of the video.

[0054] The present application builds a query-adaptive hierarchical video representation based on the breadth-expanded clusters in the first step. Specifically, the depth of the tree is expanded according to the relevance scores of the K clusters obtained in step S3. Intuitively, for high-relevance clusters, a large language model requires more detailed and fine-grained information, while for low-relevance clusters, more information may actually lead to irrelevant details, which may overwhelm the large language model and lead to incorrect reasoning. The relevance score of the top-level node determines how many layers of finer-grained information can be extracted from it, that is, how many child nodes or grandchild nodes the top-level node will have.

[0055] Since the relevance score r is one of [1, 2, ..., θ], the above expands the video feature tree according to the residual relevance of the parent node. Therefore, the value of θ is equal to the maximum depth of the video feature tree. Specifically, there is no need to further expand the low relevance (r k =1). For other correlations (r k >1), its constituent frames are re-clustered into w sub-clusters, where w represents the branch width of the video feature tree, and the value of w is determined based on the cluster correlation score, system computing resources, and validation set experimental results. k>1), the value of w is increased to capture more details; for low-correlation clusters, w is appropriately reduced to reduce the number of sub-clusters generated. These sub-clusters are then added as child nodes of the tree, giving them a correlation one less than the parent node. By recursively repeating this process, all nodes in the last layer become leaf nodes with a correlation of 1. After expanding the breadth and depth of the tree, an adaptive hierarchical video representation for long video large language model inference can be obtained.

[0056] S5, inputting the video feature tree and the video query obtained in step S4 into several preset draft models respectively, the large language model receives the outputs of all the draft models, and generates the optimal inference result based on the principle of speculative sampling;

[0057] When ordinary large language models are applied to video generation and reasoning, the large language model is based on autoregressive serial reasoning, which will lead to reduced parallelism and slower reasoning speed. However, the speculative sampling method, that is, the preliminary results are generated by fine-tuning the draft model, and then rapid verification based on parallelism can improve the generation efficiency. Figure 2 In the string-based autoregressive model shown in FIG, for example, to generate a sentence like "This is a big apple", it often requires five model calculations, and its performance is relatively low. Figure 3 As shown, different draft models are used to calculate the output, and the output of each draft model is assembled into a matrix through tree encoding, and finally calculated by model verification, so that the same result can be achieved with only one model calculation. This application optimizes the model verification part that consumes the most resources, thereby greatly improving the speed of result generation.

[0058] The step S5 comprises the following steps:

[0059] S5.1, inputting the video feature tree and the video query obtained in step S4 into a plurality of preset draft models to obtain corresponding first inference results;

[0060] The first inference result of each draft model includes a first inference text composed of a plurality of tokens and a first sampling probability corresponding to each token.

[0061] S5.2, constructing a tree coding model according to the first inference result obtained in step S5.1, and generating a draft inference matrix based on the tree coding model;

[0062] Specifically, a tree coding model is first constructed based on the first reasoning text output by all draft models. A path in the tree coding model is equivalent to the output of a draft model. Then, the columns of the draft reasoning matrix are confirmed based on all nodes of the tree coding model. Finally, the draft reasoning matrix is ​​constructed based on all paths of the tree coding model. For example, the output of draft model 1 is This is a big apple, the output of draft model 2 is This is a error, and the output of draft model 3 is This is not. Through matrix coding, the following matrix can be generated:

[0063] This is a big apple error not Draft Model 1 1 1 1 1 1 0 0 Draft Model 2 1 1 1 0 0 1 0 Draft Model 3 1 1 0 0 0 0 1

[0064] S5.3, use the mask method to update the draft inference matrix, the large language model outputs the second sampling probability of each token according to the updated draft inference matrix, and scores each token in turn based on the first sampling probability and the second sampling probability using the speculative sampling principle, and selects the token based on the score to generate the optimal inference result.

[0065] The large language model uses the speculative sampling principle to perform inference optimization on the video query based on the updated draft inference matrix and the received video feature tree;

[0066] As shown in the table above, there are a large number of elements with data equal to 0 in the draft inference matrix. These elements will bring extra computational overhead during the calculation process and will not have any value, so it is best to avoid them during calculation. It is worth noting that the output of any draft model corresponds to the starting position and ending position of its content. For example, draft model 1 does not need to calculate the two characters error and not.

[0067] This application cuts the draft inference matrix into blocks, uses the mask unit as a draft mark, and marks the starting and ending positions of each draft model output to determine whether each small block needs to be calculated through the mark, thereby avoiding the calculation of the mask unit. Cutting blocks first and then calculating avoids the calculation of blocks marked as 0, which can greatly reduce the GPU computing load and increase the inference speed by 2-3 times.

[0068] In speculative sampling, the output path is scored by adopting probability, that is, each token on the tree encoding model is scored. The score calculation formula is p(x) is the sampling probability of tokenx in the large language model, that is, the second sampling probability. For the The sampling probability of tokenx output by the draft model is also the first sampling probability. Represents the maximum sampling probability of tokenx output by all draft models. This indicates that the confidence of the large language model for this tokenx is lower than that of all the draft models. The reasoning of the draft model may be wrong. Therefore, the large language model resamples this token with a probability of p′(x)=norm(max(0,p(x)-Q(x))) and updates the second sampling probability, p(x)=p′(x). The score is scored with the updated second sampling probability. Q(x) represents the maximum sampling probability of tokenx output by all the draft models. When , it means that the confidence of the large language model for this tokenx is higher than that of the draft model. At this time, the large language model will not resample this token, but directly score this token based on p(x). The above method is used to calculate the score of each token on each path. The large language model verifies the tree encoding composed of the content output by the draft model and selects the token path with the highest score as the final output result, which is the optimal inference result.

[0069] S6, inputting the optimal inference result outputted from step S5 and the video frame sequence obtained from step S1 into a preset video generation model to obtain an answer video related to the video query;

[0070] The video generation model is responsible for converting the text or other intermediate forms generated by the large language model into video content. This step includes complex tasks such as contextual understanding of the text, image generation, and animation production. We used Text2Video-Zero as the video generation model, which generates corresponding video sequences from Gaussian noise through a diffusion process based on text prompts. During the generation process, the model learns the temporal dependency between text and video, ensuring that the generated video is semantically consistent with the text prompt.

[0071] The process of outputting the generated video is essentially a process based on the original video content to meet the user's needs for specific semantic interpretation of the original input video content. Therefore, the video frame sequence obtained in step S1 needs to be re-input into the video generation model as the video content basis. On this basis, the model optimizes the coherence of the video and merges the text corresponding to the inference result into the new video content in the form of subtitles.

[0072] The embodiment of the present application also provides an optimization device for long video generation based on tree coding, including:

[0073] Sample construction module: used to collect video queries, construct video frame sequences based on the queried video, and use the pre-trained visual encoder to obtain the frame features of each video frame in the video frame sequence.

[0074] Scoring calculation module: used to cluster the frame features obtained by the sample construction module using the K-Means clustering method, and use the large language model of the large language model storage module to obtain the correlation score between the key frame of each cluster and the video query obtained by the sample construction module;

[0075] High correlation cluster acquisition module: used to calculate the number ε of high correlation clusters according to the correlation score obtained by the score calculation module and the preset correlation threshold ξ;

[0076] Key frame output module: used to judge If yes, execute K=K+1, restart the score calculation module and the high correlation cluster acquisition module in sequence, and re-judge based on the updated number of high correlation clusters ε Otherwise, the frame feature set corresponding to the high correlation cluster and the correlation score of each cluster key frame are output, and the feature tree construction module is started, where: represents the maximum width of the preset video feature tree, and K represents the number of clusters in the score calculation module;

[0077] Feature tree construction module: used to use the key frames corresponding to all high-correlation clusters output by the key frame output module as the first-layer nodes of the video feature tree, cluster all high-correlation clusters into w sub-clusters using the K-Means clustering method, construct the key frames of the w sub-clusters as the second-layer nodes of the corresponding nodes, and set the relevance score of the second-layer nodes to be the relevance score of the first-layer nodes minus one, cluster each sub-cluster into w grandchild clusters using the K-Means clustering method, construct the key frames of the w grandchild clusters as the third-layer nodes of the corresponding nodes, and set the relevance score of the third-layer nodes to be the relevance score of the second-layer nodes minus one, and construct the following layers of nodes according to the above method until the relevance score of the last layer of nodes is one, thereby generating a video feature tree;

[0078] A draft model construction module: used to store a plurality of draft models, and based on the visual feature tree output by the feature tree construction module and the video query obtained by the sample construction module, use a plurality of draft models to respectively output first reasoning results related to the video query, wherein the first reasoning result of each draft model includes a first reasoning text composed of a plurality of tokens and a first sampling probability corresponding to each token;

[0079] A draft model is a simplified version of a large language model used to generate preliminary content output. Its main function is to quickly provide a preliminary content structure or framework before generating the full output. These draft models can quickly process input data, generate a rough version of the expected output, and provide guidance for subsequent detailed generation steps. Since the computational requirements of the draft model are relatively low, allocating 1 to 2 graphics cards to deploy 4 draft models is sufficient to meet its task requirements.

[0080] Large language model storage module: used to store the large language model, output the second sampling probability of each token based on the first inference text and the first sampling probability output by the draft model construction module, and select the token based on the first sampling probability and the second sampling using the speculative sampling principle to generate the optimal inference result;

[0081] This layer is the core computing part of the system, responsible for detailed content generation and improvement starting from the preliminary results generated by the draft model. The complete large language model has higher computational complexity and is able to consider more context and details during the reasoning process to generate high-quality output. This layer requires higher computing resources, so 4 graphics cards are allocated for deploying the complete large language model to ensure sufficient computing power and processing speed.

[0082] Video generation model storage module: used to store the video generation model, and use the video generation model to output the answer video related to the video query based on the optimal inference text output by the large language model storage module and the video frame sequence constructed by the sample construction module;

[0083] Since the video generation model layer requires dedicated graphics card resources to handle the generation and rendering of image and video data, this application uses 2 graphics cards to deploy the video generation model.

[0084] In the process of efficient video generation and reasoning, the use of multi-card layered deployment is an effective acceleration strategy. By reasonably configuring and allocating graphics card resources, parallel processing of different task modules can be achieved, thereby improving the processing efficiency and response speed of the overall system. The embodiment of the present application uses 8 graphics cards for layered deployment. Through this multi-card layered deployment method, the graphics card resources can be maximized and the computing task allocation between layers can be optimized. The rapid response of the draft model layer can provide a basis for subsequent detailed processing, while the in-depth analysis of the large language model layer is responsible for ensuring the accuracy and richness of the output content. The addition of the video generation model layer enables the system to directly generate visual content to meet the needs of multimedia output. The advantages of this multi-card layered deployment strategy are:

[0085] 1. Task parallelization: Different task modules run simultaneously on different graphics cards, avoiding the bottleneck problem of a single card and making full use of the parallel processing capabilities of the graphics card; 2. Resource optimization: Reasonably allocate graphics card resources according to the computing requirements of each module to avoid resource waste, while ensuring that each module has sufficient computing power to complete its tasks; 3. Strong scalability: The solution supports flexible expansion and adjustment. For example, the number and configuration of graphics cards at each layer can be adjusted according to specific task requirements and the performance characteristics of the graphics card to adapt to different application scenarios and loads.

[0086] In short, the multi-card layered deployment not only accelerates the inference process, but also improves the overall processing power and stability of the system. This approach is particularly suitable for large-scale model inference tasks that require real-time processing and multimedia generation, and provides an effective solution for improving user experience and meeting high performance requirements.

[0087] An embodiment of the present application also provides an electronic device, which includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of the above-mentioned optimization method for long video generation based on tree coding are performed.

[0088] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned optimization method for long video generation based on tree coding are executed. Specifically, the storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the storage medium is run, the embodiment of the above-mentioned optimization method for long video generation based on tree coding can be executed.

[0089] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An optimization method for long video generation based on tree coding, characterized in that: The steps include: S1, collect video queries, build a video frame sequence based on the queried video, use the pre-trained visual encoder to obtain the frame features of each video frame in the video frame sequence, and set the maximum depth θ and width of the video feature tree Initialize the number of clusters K; S2, clustering the frame features obtained in step S1 using the K-Means clustering method, and obtaining the correlation score between the key frame of each cluster and the video query in step S1 using the large language model; S3, calculating the number of high correlation clusters according to the correlation score obtained in step S2 and the preset correlation threshold, and judging If yes, execute K=K+1 and return to step S2; otherwise, output the frame feature set corresponding to the high correlation cluster and the correlation score of each cluster key frame, and execute step S4, where ε represents the number of high correlation clusters; S4, using the key frames corresponding to all the high correlation clusters outputted in step S3 as the first-layer nodes of the video feature tree, clustering all the high correlation clusters into w sub-clusters using the K-Means clustering method, constructing the key frames of the w sub-clusters as the second-layer nodes of the corresponding nodes, setting the correlation score of the second-layer nodes to the correlation score of the first-layer nodes minus one, clustering each sub-cluster into w grandchild clusters using the K-Means clustering method, constructing the key frames of the w grandchild clusters as the third-layer nodes of the corresponding nodes, setting the correlation score of the third-layer nodes to the correlation score of the second-layer nodes minus one, constructing each layer of nodes according to the above method until the correlation score of the last layer of nodes is one, thereby obtaining the video feature tree; S5, input the video feature tree and video query obtained in step S4 into several preset draft models respectively, the large language model receives the outputs of all the draft models, and generates the optimal inference result based on the principle of speculative sampling.

2. The optimization method for long video generation based on tree coding according to claim 1, characterized in that: The step S2 comprises the following steps: S2.1, clustering the frame features obtained in step S1 using the K-Means clustering method to obtain the key frame of each cluster; S2.2, using a subtitle generator to obtain text subtitles for each key frame obtained in step S2.1; S2.3, use a large language model to obtain the relevance score between each text subtitle and the video query.

3. The optimization method for long video generation based on tree coding according to claim 1, characterized in that: In step S3, when r k >ξ, it means that the kth cluster is a high correlation cluster. k ≤ξ, it means that the kth cluster is a low correlation cluster, where ξ represents the preset correlation threshold and r k is the relevance score between the keyframe of the kth cluster and the video query.

4. The optimization method for long video generation based on tree coding according to claim 1, characterized in that: The step S5 comprises the following steps: S5.1, inputting the video feature tree and the video query obtained in step S4 into a plurality of preset draft models to obtain corresponding first inference results; The first inference result of each draft model includes a first inference text composed of a number of tokens and a first sampling probability corresponding to each token; S5.2, constructing a tree coding model according to the first inference text obtained in step S5.1, and generating a draft inference matrix based on the tree coding model; S5.3, use the mask method to update the draft inference matrix, the large language model outputs the second sampling probability of each token according to the updated draft inference matrix, and scores each token in turn based on the first sampling probability and the second sampling probability using the speculative sampling principle, and selects the token based on the score to generate the optimal inference result.

5. The optimization method for long video generation based on tree coding according to claim 4 is characterized in that: In step S5.3, before scoring each token in turn based on the first sampling probability and the second sampling probability using the speculative sampling principle, the first sampling probability and the second sampling probability are compared. When Q(x)>p(x), p ′ (x) = norm(max(0, p(x) - Q(x))) updates the second sampling probability, otherwise, the second sampling probability is not updated. Q(x) represents the maximum value of the first sampling probability of tokenx output by all draft models, and p(x) represents the second sampling probability of tokenx output by the large language model.

6. The optimization method for long video generation based on tree coding according to claim 4 or 5, characterized in that: The method further includes step S6, wherein the optimal inference result obtained in step S5 and the video frame sequence obtained in step S1 are input into a preset video generation model to obtain an answer video related to the video query.

Images