Video understanding method and device based on dynamic token, storage medium and computer equipment
By dynamically adjusting the token set, the problems of high computational cost and redundant computation in long video understanding are solved, enabling faster and more accurate video reasoning.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-16
- Publication Date
- 2026-04-10
AI Technical Summary
Existing video understanding methods are computationally expensive and involve a lot of redundant computation when processing long videos. Furthermore, static token reduction methods affect the model's ability to fully understand the content, especially in tasks that require sophisticated spatiotemporal reasoning.
A video understanding method based on dynamic tokens is adopted, which eliminates the initial tokens by similarity and dynamically adjusts the core token set in the middle layer to reduce the amount of computation and avoid redundant computation, thus ensuring the integrity of key information.
It reduces computational load, speeds up processing, improves reasoning accuracy for long and complex videos, and enhances the model's reasoning capabilities.
Smart Images
Figure CN121838020A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of video processing, in particular to a video understanding method and device based on dynamic tokens, a storage medium and a computer device. BACKGROUND
[0002] In the field of video understanding, with the explosive growth of video data and the increasing complexity of video understanding tasks, how to efficiently and accurately analyze and infer video content becomes a key problem. For example, in the medical and health field, when processing long medical videos, traditional video large language models usually use dense sampling to record rehabilitation pictures frame by frame, generate a large number of visual tokens, and then uniformly send them into the Transformer layer for fixed processing, treating all tokens equally. To reduce computational cost, some pre-trained or static token reduction methods are introduced into the medical field. They remove or aggregate some tokens at the input end according to pre-set rules. For example, when analyzing rehabilitation videos, similar action frames reflecting different rehabilitation stages may be incorrectly merged, affecting the judgment of rehabilitation progress.
[0003] However, in the long video understanding task, thousands of visual tokens need to be processed, which causes the computational cost, especially the inference cost, to grow exponentially; at the same time, not all visual tokens are equally important for the final understanding and inference, and a large number of redundant or irrelevant tokens are sent to the deep layer of the model for unnecessary calculation, resulting in low efficiency of visual tokens; and the existing static token reduction or alternative representation method, although it improves efficiency, often sacrifices the overall understanding ability of the model, and performs poorly in long video tasks that require fine spatio-temporal reasoning. SUMMARY
[0004] Therefore, the embodiments of the present application provide a video understanding method and device based on dynamic tokens, a storage medium and a computer device, which reduce the number of initial tokens by similarity elimination, reduce the computational cost, dynamically adjust the middle layer to avoid redundant calculation, speed up the processing speed, and are especially beneficial for fast inference of long videos and complex videos. The dynamic adjustment mechanism can also ensure the integrity of key information, remove interference redundant tokens, revive important eliminated tokens, and enhance inference accuracy.
[0005] According to one aspect of the present application, a video understanding method based on dynamic tokens is provided, which comprises: obtaining a target video and an inference task; inputting the target video and the inference task into a video large model to perform the following steps through the video large model to complete the task inference of the inference task: frame sampling and encoding the target video to obtain an initial visual token set of the target video, performing similarity-based elimination on tokens in the initial visual token set according to the inference task to obtain a core visual token set; sending the core visual token set to a shallow layer of the video large model for processing, and then transmitting the processed core visual token set to a middle layer of the video large model for processing, wherein the middle layer dynamically adjusts the core visual token set based on a real-time processing state to delete tokens in the core visual token set and / or revive eliminated tokens in the process of processing the core visual token set; transmitting the core visual token set obtained after processing of the middle layer to a deep layer of the video large model for processing to determine an inference result of the inference task.
[0006] Optionally, the shallow layer, the middle layer and the deep layer of the video large model each include a plurality of Transformer layers; and the method further includes: In the process of processing the core visual token set by the deep layer of the video large model, a prediction result and a corresponding confidence of at least one preset Transformer layer in the deep layer are captured, and when it is identified that the confidence of the captured prediction result meets a preset exit confidence condition, the task inference of the video large model is ended, and the prediction result meeting the preset exit confidence condition is output as the inference result of the inference task.
[0007] Optionally, before the target video and the inference task are input into the video large model, the method further includes: inserting a shallow layer token selector in the shallow layer, a middle layer dynamic token adjuster in the middle layer, and a deep layer exit mechanism executor in the deep layer of the pre-trained video large model; The shallow layer token selector is configured to perform similarity-based elimination on tokens in the initial visual token set according to the inference task to obtain the core visual token set. The middle layer dynamic token adjuster is configured to dynamically adjust the core visual token set based on a real-time processing state to delete tokens in the core visual token set and / or revive eliminated tokens. The deep layer exit mechanism executor is configured to capture a prediction result and a corresponding confidence of at least one preset Transformer layer in the deep layer, and when it is identified that the confidence of the captured prediction result meets a preset exit confidence condition, the task inference of the video large model is ended, and the prediction result meeting the preset exit confidence condition is output as the inference result of the inference task.
[0008] Optionally, the preset Transformer layer comprises every preset number of Transformer layers in the plurality of Transformer layers in the deep layer; and the prediction result and the corresponding confidence of at least one preset Transformer layer in the deep layer are captured, comprising: The deep layer exit mechanism executor captures the prediction result and the corresponding confidence of the preset Transformer layer in the deep layer; If the inference task is a selection type task, the confidences of the prediction results are sorted from large to small, the difference between the confidences of the first and second in the sorting is calculated, and the prediction result of the confidence of the first in the sorting is taken as the inference result of the inference task when the confidence of the first in the sorting is greater than a preset confidence threshold and the difference is greater than a preset difference threshold. If the inference task is a question and answer type task, the prediction result is taken as the inference result of the inference task when the confidence of the prediction result is greater than a preset confidence threshold.
[0009] Optionally, the tokens in the initial visual token set are eliminated based on similarity according to the inference task to obtain a core visual token set, comprising: The shallow layer token selector calculates the similarity between the inference task and each token in the initial visual token set; eliminates the tokens in the initial visual token set whose similarity is lower than a preset similarity threshold, takes the eliminated initial visual token set as the core visual token set, and constructs an eliminated token set based on the eliminated tokens.
[0010] Optionally, the process of processing the core visual token set by the middle layer is dynamically adjusted based on the real-time processing state of the core visual token set to delete the tokens in the core visual token set and / or revive the eliminated tokens, comprising: During the process of processing the core visual token set by the middle layer, the middle layer dynamic token adjuster monitors the real-time inference state of the inference task by the middle layer, calculates the first correlation between the inference task and each token in the eliminated token set when the uncertainty of the task inference based on the core visual token set reflected by the real-time inference state is higher than a preset uncertainty threshold, and revives the tokens in the eliminated token set based on the first correlation. The middle layer dynamic token adjuster calculates the second correlation between the inference task and each token in the core visual token set, and deletes the tokens in the core visual token set based on the second correlation.
[0011] Optionally, after the middle layer dynamic token adjuster monitors the real-time inference state of the inference task by the middle layer, the method further comprises: When the real-time inference state reflects the lack of inference information for the inference task in the core visual token set, the third correlation between the inference task and each token in the discarded token set is calculated, and the tokens in the discarded token set are revived based on the third correlation.
[0012] According to another aspect of this application, a video understanding device based on dynamic tokens is provided, the device comprising: The task acquisition module is used to acquire the target video and the inference task; The task reasoning module is used to input the target video and the reasoning task into the video large model, so as to complete the task reasoning by performing the following steps through the video large model: The target video is sampled and encoded frame by frame to obtain an initial visual token set. The tokens in the initial visual token set are eliminated based on similarity according to the reasoning task to obtain a core visual token set. After the core visual token set is sent to the shallow layer of the video model for processing, the processed core visual token set is passed to the middle layer of the video model for processing. During the processing of the core visual token set by the middle layer, the core visual token set is dynamically adjusted based on the real-time processing status to delete tokens in the core visual token set and / or revive eliminated tokens. The core visual token set obtained after the intermediate layer processing is passed to the deep layer of the large video model for further processing to determine the reasoning result of the reasoning task.
[0013] Optionally, the shallow, middle, and deep layers of the video model each include multiple Transformer layers; the task inference module is further used for: During the deep processing of the core visual token set in the video big model, the prediction results and corresponding confidence scores of at least one preset Transformer layer in the deep layer are captured. When the confidence score of the captured prediction result meets the preset exit confidence score condition, the task inference of the video big model ends, and the prediction result that meets the preset exit confidence score condition is output as the inference result of the inference task.
[0014] Optionally, the device further includes: a model modification module, used for: In the pre-trained video large model, a shallow token selector is inserted in the shallow layer, a medium-level dynamic token regulator is inserted in the middle layer, and a deep exit mechanism executor is inserted in the deep layer. The shallow token selector is used to eliminate tokens in the initial visual token set based on similarity according to the reasoning task, so as to obtain the core visual token set. The mid-level dynamic token adjuster is used to dynamically adjust the core visual token set based on the real-time processing status, in order to delete tokens in the core visual token set and / or revive eliminated tokens. The deep exit mechanism executor is used to capture the prediction results and corresponding confidence scores of at least one preset Transformer layer in the deep layer, and when it is identified that the confidence score of the captured prediction result meets the preset exit confidence score condition, the task inference of the video large model is terminated, and the prediction result that meets the preset exit confidence score condition is output as the inference result of the inference task.
[0015] Optionally, the preset Transformer layer includes Transformer layers spaced at preset intervals from the multiple deep Transformer layers; the task inference module is specifically used for: The deep exit mechanism executor captures the prediction results and corresponding confidence scores of the preset Transformer layer in the deep layer; If the reasoning task is a selection task, the confidence scores of the prediction results are sorted from largest to smallest, the difference between the confidence scores of the first and second ranked results is calculated, and when the confidence score of the first ranked result is greater than a preset confidence threshold and the difference is greater than a preset difference threshold, the prediction result of the first ranked result is taken as the reasoning result of the reasoning task. If the reasoning task is a question-and-answer type task, then the prediction result is taken as the reasoning result of the reasoning task when the confidence level of the prediction result is greater than a preset confidence threshold.
[0016] Optionally, the task reasoning module is specifically used for: The shallow token selector calculates the similarity between the reasoning task and each token in the initial visual token set; it eliminates tokens in the initial visual token set whose similarity is lower than a preset similarity threshold, uses the eliminated initial visual token set as the core visual token set, and constructs an eliminated token set based on the eliminated tokens.
[0017] Optionally, the task reasoning module is specifically used for: During the processing of the core visual token set by the middle layer, the middle layer dynamic token adjuster monitors the real-time reasoning status of the middle layer for the reasoning task. When the real-time reasoning status reflects that the uncertainty of the task reasoning based on the core visual token set is higher than a preset uncertainty threshold, the first correlation between the reasoning task and each token in the elimination token set is calculated, and the tokens in the elimination token set are revived based on the first correlation. The mid-level dynamic token regulator calculates the second correlation between the inference task and each token in the core visual token set, and deletes tokens from the core visual token set based on the second correlation.
[0018] Optionally, the task reasoning module is specifically used for: When the real-time inference state reflects the lack of inference information for the inference task in the core visual token set, the third correlation between the inference task and each token in the discarded token set is calculated, and the tokens in the discarded token set are revived based on the third correlation.
[0019] According to another aspect of this application, a storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the above-described video understanding method based on dynamic tokens.
[0020] According to another aspect of this application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the program to implement the above-described video understanding method based on dynamic tokens.
[0021] By employing the above technical solutions, this application provides a video understanding method, apparatus, storage medium, and computer device based on dynamic tokens. First, a target video and corresponding inference task are acquired and input into a large video model. The large video model first samples and encodes the target video frames to obtain an initial visual token set. Based on the inference task, some tokens are eliminated according to similarity to obtain a core visual token set. This set is then sent to a shallow processing layer and then to a middle layer. The middle layer dynamically adjusts the core visual token set according to the real-time processing status, deleting or reviving tokens. Finally, the token set processed by the middle layer is sent to a deep processing layer to determine the inference task result. This application reduces the number of initial tokens through similarity elimination, lowering the computational load. Dynamic adjustment in the middle layer avoids redundant computation, accelerating processing speed, especially beneficial for rapid inference of long and complex videos. The dynamic adjustment mechanism also ensures the integrity of key information, removes interfering and redundant tokens, revives important eliminated tokens, and enhances inference accuracy.
[0022] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description
[0023] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart illustrating a video understanding method based on dynamic tokens provided in an embodiment of this application is shown. Figure 2 A flowchart illustrating another video understanding method based on dynamic tokens provided in an embodiment of this application is shown. Figure 3 This illustration shows a structural schematic diagram of a video understanding device based on a dynamic token, according to an embodiment of this application. Figure 4 A schematic diagram of the device structure of a computer device provided in an embodiment of this application is shown. Detailed Implementation
[0024] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.
[0025] This embodiment provides a video understanding method based on dynamic tokens. The method includes: acquiring a target video and a reasoning task; inputting the target video and the reasoning task into a large video model, and then performing the following steps through the large video model to complete the task reasoning for the reasoning task, such as... Figure 1 As shown, it includes: Step 101: Perform frame sampling and encoding on the target video to obtain an initial visual token set of the target video. Eliminate tokens in the initial visual token set based on similarity according to the reasoning task to obtain a core visual token set. Step 102: After the core visual token set is sent to the shallow layer of the video model for processing, the processed core visual token set is passed to the middle layer of the video model for processing. During the processing of the core visual token set by the middle layer, the core visual token set is dynamically adjusted based on the real-time processing status to delete tokens in the core visual token set and / or revive eliminated tokens. Step 103: Pass the core visual token set obtained after the intermediate layer processing to the deep layer of the large video model for processing, and determine the reasoning result of the reasoning task.
[0026] This application proposes a video understanding method based on dynamic tokens, aiming to efficiently and accurately complete inference tasks for a given video. Specifically, it acquires the target video and its corresponding inference task. The target video is the material to be analyzed, and the inference task specifies what information to extract or what judgment to make from the video. For example, in the medical field, the target video might be a surgical video, and the inference task might be to determine whether the surgical procedure conforms to regulations. Next, the target video and the inference task are input into a large video model, which then performs subsequent inference steps. The large video model consists of an encoding layer, a shallow layer, a mid-level layer, and a deep layer. This hierarchical structure allows the model to progressively extract features from the video, from low-level to high-level, achieving a deeper understanding of the video content. The encoding layer is used to sample and encode frames in the video to obtain a token set, which represents the features of the video frames. The shallow layer is mainly responsible for processing basic visual elements of the video, such as low-level features like color, texture, and edges. These features form the basis for building more complex representations, providing rich information for subsequent layers through preliminary processing of video frames. Building upon the shallow layer features, the middle layer further extracts local semantic information from the video, such as parts of objects and fragments of actions. This middle-layer processing captures relatively independent semantic units within the video, providing more specific clues for understanding the overall content. The deep layer integrates and abstracts the features extracted in the middle layer, forming a high-level semantic understanding of the video's overall content, such as its theme, events, and emotions. Deep processing grasps the global information of the video, enabling in-depth analysis and reasoning.
[0027] First, the video big data model samples and encodes frames of the target video, transforming it into an initial visual token set. These tokens represent different information segments within the video. Then, based on the inference task, the initial visual token set is eliminated according to similarity, removing tokens with low relevance or similarity to the inference task, thus obtaining a core visual token set. This step initially filters out key information for completing the inference task. Next, the core visual token set is processed in the shallow layer of the video big data model before being passed to the middle layer. During the middle layer processing, the model dynamically adjusts the core visual token set based on the real-time processing status. It flexibly decides to delete some tokens, reduce tokens with low relevance to the current inference task, maintain the simplicity and dynamic optimization of the token set, or revive previously eliminated tokens that may be important in subsequent inferences to provide supplementary information and compensate for potential information loss caused by shallow layer reduction, so that the token set can better adapt to the complexity of the video content and the needs of the inference task. Finally, the core visual token set obtained after middle layer processing is passed to the deep layer of the video big data model for further processing, ultimately determining the inference result of the inference task. Deep processing enables in-depth analysis and comprehensive judgment of the core information after screening and dynamic adjustment, thereby deriving accurate inference conclusions. This application's embodiment reduces the number of tokens entering subsequent processing and lowers the computational load by eliminating some tokens from the initial visual token set based on similarity. Simultaneously, the dynamic adjustment of the core visual token set by the middle layer avoids unnecessary computation, further improving computational efficiency and enabling faster inference results when processing long or complex videos. The dynamic adjustment mechanism can flexibly handle tokens according to the real-time processing status, removing redundant tokens that may interfere with inference and reviving eliminated tokens that may play an important role in inference, thus ensuring the integrity of key information during the inference process and improving the accuracy of the inference results.
[0028] Taking rehabilitation training videos as an example: For rehabilitation training videos, the initial visual token set records all motion frames of the patient throughout the entire rehabilitation training process. Based on the inference task of rehabilitation assessment, similarity-based elimination can remove some repetitive or irrelevant motion frames, resulting in a core visual token set that highlights key rehabilitation actions. During the intermediate dynamic adjustment process, the model can delete some motion tokens that have already met the rehabilitation standards and do not require special attention, or revive some previously eliminated tokens that reflect potential rehabilitation problems (such as improper movement, abnormal muscle exertion, etc.), based on the patient's real-time motion performance and rehabilitation progress. Finally, deep processing can more accurately assess the patient's rehabilitation status, providing a basis for developing personalized rehabilitation plans.
[0029] By applying the technical solution of this embodiment, the target video and corresponding inference task are first acquired and input into a large video model. The large video model first samples and encodes the target video frames to obtain an initial visual token set, and then eliminates some tokens based on similarity according to the inference task to obtain a core visual token set. This set is then sent to a shallow processing layer and then to a middle layer. The middle layer dynamically adjusts the core visual token set according to the real-time processing status, and can delete or revive tokens. Finally, the token set processed by the middle layer is sent to a deep processing layer to determine the inference task result. This embodiment reduces the number of initial tokens through similarity elimination, reducing computational load, and the dynamic adjustment in the middle layer avoids redundant computation, thus accelerating processing speed, especially beneficial for rapid inference of long and complex videos. The dynamic adjustment mechanism also ensures the integrity of key information, removes interfering and redundant tokens, revives important eliminated tokens, and enhances inference accuracy.
[0030] Optionally, in this embodiment of the application, before inputting the target video and the inference task into the video large model, the method further includes: In the pre-trained video large model, a shallow token selector is inserted in the shallow layer, a medium-level dynamic token regulator is inserted in the middle layer, and a deep exit mechanism executor is inserted in the deep layer. The shallow token selector is used to eliminate tokens in the initial visual token set based on similarity according to the reasoning task, so as to obtain the core visual token set. The mid-level dynamic token adjuster is used to dynamically adjust the core visual token set based on the real-time processing status, in order to delete tokens in the core visual token set and / or revive eliminated tokens. The deep exit mechanism executor is used to capture the prediction results and corresponding confidence scores of at least one preset Transformer layer in the deep layer, and when it is identified that the confidence score of the captured prediction result meets the preset exit confidence score condition, the task inference of the video large model is terminated, and the prediction result that meets the preset exit confidence score condition is output as the inference result of the inference task.
[0031] In the above embodiments, before inputting the target video and inference task into the video big model, the pre-trained video big model is structurally modified. Specifically, a shallow token selector is inserted into the shallow layer of the video big model. Its function is to filter tokens in the initial visual token set obtained through frame sampling and encoding based on similarity, eliminating tokens with low similarity to the inference task, thereby obtaining the core visual token set. A mid-layer dynamic token regulator is inserted into the middle layer of the video big model. This regulator dynamically adjusts the core visual token set according to the model's state during real-time processing. It can delete tokens with low relevance in the core visual token set and revive previously eliminated tokens that may be helpful for the current inference. A deep exit mechanism executor is inserted into the deep layer of the video big model. It captures the prediction results and corresponding confidence scores of at least one preset Transformer layer in the deep layer. When the confidence scores of these prediction results meet the preset exit confidence score conditions, the task inference of the video big model ends, and the prediction results that meet the conditions are output as the final result of the inference task. Through the above embodiments, dynamic token management of large video models is cleverly achieved without modifying the pre-trained weights or performing extensive additional fine-tuning. For many pre-trained large video models, there is no need to reinvest significant computing resources and time in weight adjustments and complex fine-tuning; simply inserting the corresponding module according to this method endows the model with the ability to dynamically manage tokens. This not only greatly reduces deployment costs, minimizing the high hardware investment and lengthy training time required for model retraining, but also lowers the technical threshold, enabling more enterprises and developers to easily apply this advanced technology and quickly integrate it into their own video processing systems, thereby improving the efficiency and accuracy of video understanding. It has broad application prospects in many fields such as video analysis, intelligent surveillance, and medical image diagnosis.
[0032] Furthermore, as a refinement and extension of the specific implementation methods of the above embodiments, and to fully illustrate the specific implementation process of this embodiment, another video understanding method based on dynamic tokens is provided, such as... Figure 2 As shown, the method includes: Step 201: Perform frame sampling and encoding on the target video to obtain the initial visual token set of the target video.
[0033] In this embodiment, frame sampling involves selecting a subset of video frames from the target video according to certain rules, such as selecting one frame at fixed time intervals or selecting frames based on key changes in the video content. Encoding converts the selected video frames into a digital representation that a computer can understand and process; these digital representations are then organized into an initial set of visual tokens. Each token represents a specific piece of information within the video.
[0034] Step 202: The shallow token selector calculates the similarity between the inference task and each token in the initial visual token set; it eliminates tokens in the initial visual token set whose similarity is lower than a preset similarity threshold, uses the eliminated initial visual token set as the core visual token set, and constructs an eliminated token set based on the eliminated tokens.
[0035] In this embodiment, the shallow token selector first calculates the similarity between the inference task and each token in the initial visual token set. The similarity calculation method can be determined according to the specific task and model design, such as using cosine similarity, Euclidean distance, or other metrics. Then, tokens with similarity below a preset similarity threshold are eliminated; these eliminated tokens are considered to have low relevance to the current inference task. The remaining initial visual token set after elimination serves as the core visual token set, and simultaneously, an elimination token set is constructed based on the eliminated tokens.
[0036] Step 203: After the core visual token set is processed in the shallow layer of the video model, the processed core visual token set is passed to the middle layer of the video model for further processing. During the processing of the core visual token set in the middle layer, the middle layer dynamic token regulator monitors the real-time inference status of the inference task. When the real-time inference status reflects that the uncertainty of the task inference based on the core visual token set is higher than a preset uncertainty threshold, the regulator calculates the first correlation between the inference task and each token in the discarded token set, and revives the tokens in the discarded token set based on the first correlation. The middle layer dynamic token regulator calculates the second correlation between the inference task and each token in the core visual token set, and deletes the tokens in the core visual token set based on the second correlation. When the real-time inference status reflects that the core visual token set lacks inference information for the inference task, the regulator calculates the third correlation between the inference task and each token in the discarded token set, and revives the tokens in the discarded token set based on the third correlation.
[0037] In this embodiment, the core visual token set is fed into the shallow processing layer of the large video model and then passed to the middle layer. The middle-layer dynamic token regulator monitors the real-time inference status of the inference task. When the real-time inference status reflects that the uncertainty of the task inference based on the core visual token set is higher than a preset uncertainty threshold, it indicates that the current core visual token set may not be sufficient to accurately complete the inference task, and more relevant information needs to be introduced. At this time, the first correlation between the inference task and each token in the discarded token set is calculated. The higher the correlation, the greater the potential help of the discarded token to the inference task. Based on the first correlation, the tokens in the discarded token set are revived, and the discarded tokens with high correlation are added back to the core visual token set. For example, when analyzing a video of a complex scene, the initial core visual token set may only focus on the main objects, but during the inference process, it is found that the understanding of the background information of the scene is insufficient, resulting in high uncertainty. At this time, some background-related tokens can be revived from the discarded token set. In addition, the middle-layer dynamic token regulator also calculates the second correlation between the inference task and each token in the core visual token set. If the correlation between a core visual token and the inference task is low, that is, the second correlation is lower than a certain standard, it indicates that the token contributes little to the current inference task, and may even interfere with the inference process. Tokens in the core visual token set are removed based on the second correlation to reduce unnecessary computation and improve inference accuracy. When the real-time inference state reflects that the core visual token set lacks inference information for the inference task, it indicates that the current core visual token set does not cover all the key information required to complete the inference task. In this case, the third correlation between the inference task and each token in the discarded token set is calculated, and tokens in the discarded token set are revived based on the third correlation, reintroducing discarded tokens that can supplement the missing information into the core visual token set. For example, when analyzing a process video containing multiple steps, if the core visual token set only focuses on some steps, causing the inference task to be incomplete, tokens related to the missing steps can be revived from the discarded token set.
[0038] Step 204: The core visual token set obtained after the intermediate layer processing is passed to the deep layer of the video big model for processing. During the processing of the core visual token set by the deep layer of the video big model, the prediction results and corresponding confidence scores of at least one preset Transformer layer in the deep layer are captured. When the confidence score of the captured prediction result meets the preset exit confidence score condition, the task inference of the video big model ends, and the prediction result that meets the preset exit confidence score condition is output as the inference result of the inference task.
[0039] In this embodiment, the core visual token set obtained after intermediate layer processing is passed to the deep layers of the large video model for further processing. During deep layer processing, the prediction results and corresponding confidence scores of at least one preset Transformer layer in the deep layer are captured. The confidence score reflects the reliability of the model's prediction results. When the confidence score of the captured prediction result meets the preset exit confidence score condition, it indicates that the model has made a sufficiently accurate judgment on the inference task. At this point, the inference task of the large video model ends, and the prediction result that meets the preset exit confidence score condition is output as the inference result of the inference task. For example, in an image classification task, if the prediction confidence score of a certain Transformer layer in the deep layer for a certain category reaches a preset high threshold, that category can be directly output as the classification result without further calculation.
[0040] Optionally, in this embodiment, the preset Transformer layer includes Transformer layers spaced at preset intervals within the deep layer; capturing the prediction results and corresponding confidence scores of at least one preset Transformer layer in the deep layer includes: The deep exit mechanism executor captures the prediction results and corresponding confidence scores of the preset Transformer layer in the deep layer; If the reasoning task is a selection task, the confidence scores of the prediction results are sorted from largest to smallest, the difference between the confidence scores of the first and second ranked results is calculated, and when the confidence score of the first ranked result is greater than a preset confidence threshold and the difference is greater than a preset difference threshold, the prediction result of the first ranked result is taken as the reasoning result of the reasoning task. If the reasoning task is a question-and-answer type task, then the prediction result is taken as the reasoning result of the reasoning task when the confidence level of the prediction result is greater than a preset confidence threshold.
[0041] In this embodiment, Transformer layers selected at predetermined intervals (e.g., every two layers) within the deep layers can be used as monitoring points. This interval sampling design can reduce the computational overhead of full-layer monitoring while ensuring inference depth. For example, in medical image analysis, monitoring every three layers can balance accuracy and efficiency. For selection-based tasks (e.g., classification / multiple selection), the prediction results of the monitoring layers can be sorted in descending order of confidence, and the confidence difference between Top1 and Top2 can be calculated. The Top1 result is output only when the confidence of Top1 is greater than a preset confidence threshold (e.g., 0.9) and the difference is greater than a preset difference threshold (e.g., 0.15). For question-and-answer-based tasks (e.g., description generation), the result is output immediately when the confidence of any monitoring layer prediction result is greater than a threshold (e.g., 0.85). For example, in rehabilitation training assessment, if the model's prediction confidence for "patient action completion rate" reaches 0.87, inference is terminated and the result is output, saving computational resources for subsequent layers.
[0042] Furthermore, as Figure 1 In terms of specific implementation, this application provides a video understanding device based on dynamic tokens, such as... Figure 3 As shown, the device includes: The task acquisition module is used to acquire the target video and the inference task; The task reasoning module is used to input the target video and the reasoning task into the video large model, so as to complete the task reasoning by performing the following steps through the video large model: The target video is sampled and encoded frame by frame to obtain an initial visual token set. The tokens in the initial visual token set are eliminated based on similarity according to the reasoning task to obtain a core visual token set. After the core visual token set is sent to the shallow layer of the video model for processing, the processed core visual token set is passed to the middle layer of the video model for processing. During the processing of the core visual token set by the middle layer, the core visual token set is dynamically adjusted based on the real-time processing status to delete tokens in the core visual token set and / or revive eliminated tokens. The core visual token set obtained after the intermediate layer processing is passed to the deep layer of the large video model for further processing to determine the reasoning result of the reasoning task.
[0043] Optionally, the shallow, middle, and deep layers of the video model each include multiple Transformer layers; the task inference module is further used for: During the deep processing of the core visual token set in the video big model, the prediction results and corresponding confidence scores of at least one preset Transformer layer in the deep layer are captured. When the confidence score of the captured prediction result meets the preset exit confidence score condition, the task inference of the video big model ends, and the prediction result that meets the preset exit confidence score condition is output as the inference result of the inference task.
[0044] Optionally, the device further includes: a model modification module, used for: In the pre-trained video large model, a shallow token selector is inserted in the shallow layer, a medium-level dynamic token regulator is inserted in the middle layer, and a deep exit mechanism executor is inserted in the deep layer. The shallow token selector is used to eliminate tokens in the initial visual token set based on similarity according to the reasoning task, so as to obtain the core visual token set. The mid-level dynamic token adjuster is used to dynamically adjust the core visual token set based on the real-time processing status, in order to delete tokens in the core visual token set and / or revive eliminated tokens. The deep exit mechanism executor is used to capture the prediction results and corresponding confidence scores of at least one preset Transformer layer in the deep layer, and when it is identified that the confidence score of the captured prediction result meets the preset exit confidence score condition, the task inference of the video large model is terminated, and the prediction result that meets the preset exit confidence score condition is output as the inference result of the inference task.
[0045] Optionally, the preset Transformer layer includes Transformer layers spaced at preset intervals from the multiple deep Transformer layers; the task inference module is specifically used for: The deep exit mechanism executor captures the prediction results and corresponding confidence scores of the preset Transformer layer in the deep layer; If the reasoning task is a selection task, the confidence scores of the prediction results are sorted from largest to smallest, the difference between the confidence scores of the first and second ranked results is calculated, and when the confidence score of the first ranked result is greater than a preset confidence threshold and the difference is greater than a preset difference threshold, the prediction result of the first ranked result is taken as the reasoning result of the reasoning task. If the reasoning task is a question-and-answer type task, then the prediction result is taken as the reasoning result of the reasoning task when the confidence level of the prediction result is greater than a preset confidence threshold.
[0046] Optionally, the task reasoning module is specifically used for: The shallow token selector calculates the similarity between the reasoning task and each token in the initial visual token set; it eliminates tokens in the initial visual token set whose similarity is lower than a preset similarity threshold, uses the eliminated initial visual token set as the core visual token set, and constructs an eliminated token set based on the eliminated tokens.
[0047] Optionally, the task reasoning module is specifically used for: During the processing of the core visual token set by the middle layer, the middle layer dynamic token adjuster monitors the real-time reasoning status of the middle layer for the reasoning task. When the real-time reasoning status reflects that the uncertainty of the task reasoning based on the core visual token set is higher than a preset uncertainty threshold, the first correlation between the reasoning task and each token in the elimination token set is calculated, and the tokens in the elimination token set are revived based on the first correlation. The mid-level dynamic token regulator calculates the second correlation between the inference task and each token in the core visual token set, and deletes tokens from the core visual token set based on the second correlation.
[0048] Optionally, the task reasoning module is specifically used for: When the real-time inference state reflects the lack of inference information for the inference task in the core visual token set, the third correlation between the inference task and each token in the discarded token set is calculated, and the tokens in the discarded token set are revived based on the third correlation.
[0049] It should be noted that other corresponding descriptions of the functional units involved in the video understanding device based on dynamic tokens provided in this application embodiment can be found in the following references. Figures 1 to 2 The corresponding descriptions in the method will not be repeated here.
[0050] This application also provides a computer device, which may specifically be a personal computer, a server, a network device, etc. Figure 4 As shown, the computer device includes a bus, a processor, memory, and a communication interface, and may also include an input / output interface and a display device. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores location information. The network interface allows communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the various method embodiments.
[0051] Those skilled in the art will understand thatFigure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0052] In one embodiment, a computer-readable storage medium is provided, which may be non-volatile or volatile, having stored thereon a computer program that, when executed by a processor, implements the steps in the above method embodiments.
[0053] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.
[0054] It should be noted that the user personal information involved in the embodiments of this application is all authorized (with the knowledge and consent) by the relevant parties or fully authorized by all parties, and the executing entity can obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with the relevant laws and regulations of the relevant countries and regions, and do not violate public order and good morals. It should be noted that if any software tools or components other than those of this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use.
[0055] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, graphics processors, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.
[0056] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0057] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A video understanding method based on dynamic tokens, characterized in that, The method includes: Acquire the target video and inference task; The target video and the inference task are input into the video model, and the following steps are performed through the video model to complete the task inference: The target video is sampled and encoded frame by frame to obtain an initial visual token set. The tokens in the initial visual token set are eliminated based on similarity according to the reasoning task to obtain a core visual token set. After the core visual token set is sent to the shallow layer of the video model for processing, the processed core visual token set is passed to the middle layer of the video model for processing. During the processing of the core visual token set by the middle layer, the core visual token set is dynamically adjusted based on the real-time processing status to delete tokens in the core visual token set and / or revive eliminated tokens. The core visual token set obtained after the intermediate layer processing is passed to the deep layer of the large video model for further processing to determine the reasoning result of the reasoning task.
2. The method according to claim 1, characterized in that, The shallow, middle, and deep layers of the video large model each include multiple Transformer layers; the method further includes: During the deep processing of the core visual token set in the video big model, the prediction results and corresponding confidence scores of at least one preset Transformer layer in the deep layer are captured. When the confidence score of the captured prediction result meets the preset exit confidence score condition, the task inference of the video big model ends, and the prediction result that meets the preset exit confidence score condition is output as the inference result of the inference task.
3. The method according to claim 2, characterized in that, Before inputting the target video and the inference task into the video large model, the method further includes: In the pre-trained video large model, a shallow token selector is inserted in the shallow layer, a medium-level dynamic token regulator is inserted in the middle layer, and a deep exit mechanism executor is inserted in the deep layer. The shallow token selector is used to eliminate tokens in the initial visual token set based on similarity according to the reasoning task, so as to obtain the core visual token set. The mid-level dynamic token adjuster is used to dynamically adjust the core visual token set based on the real-time processing status, in order to delete tokens in the core visual token set and / or revive eliminated tokens. The deep exit mechanism executor is used to capture the prediction results and corresponding confidence scores of at least one preset Transformer layer in the deep layer, and when it is identified that the confidence score of the captured prediction result meets the preset exit confidence score condition, the task inference of the video large model is terminated, and the prediction result that meets the preset exit confidence score condition is output as the inference result of the inference task.
4. The method according to claim 3, characterized in that, The preset Transformer layer includes multiple Transformer layers in the deep layer, with a preset number of Transformer layers in between; Capturing the prediction results and corresponding confidence scores of at least one preset Transformer layer in the deep layer includes: The deep exit mechanism executor captures the prediction results and corresponding confidence scores of the preset Transformer layer in the deep layer; If the reasoning task is a selection task, the confidence scores of the prediction results are sorted from largest to smallest, the difference between the confidence scores of the first and second ranked results is calculated, and when the confidence score of the first ranked result is greater than a preset confidence threshold and the difference is greater than a preset difference threshold, the prediction result of the first ranked result is taken as the reasoning result of the reasoning task. If the reasoning task is a question-and-answer type task, then the prediction result is taken as the reasoning result of the reasoning task when the confidence level of the prediction result is greater than a preset confidence threshold.
5. The method according to claim 3, characterized in that, Based on the reasoning task, tokens in the initial visual token set are eliminated according to similarity to obtain a core visual token set, including: The shallow token selector calculates the similarity between the reasoning task and each token in the initial visual token set; it eliminates tokens in the initial visual token set whose similarity is lower than a preset similarity threshold, uses the eliminated initial visual token set as the core visual token set, and constructs an eliminated token set based on the eliminated tokens.
6. The method according to claim 5, characterized in that, The process of the middle layer processing the core visual token set involves dynamically adjusting the core visual token set based on the real-time processing status to delete tokens from the core visual token set and / or revive expired tokens, including: During the processing of the core visual token set by the middle layer, the middle layer dynamic token adjuster monitors the real-time reasoning status of the middle layer for the reasoning task. When the real-time reasoning status reflects that the uncertainty of the task reasoning based on the core visual token set is higher than a preset uncertainty threshold, the first correlation between the reasoning task and each token in the elimination token set is calculated, and the tokens in the elimination token set are revived based on the first correlation. The mid-level dynamic token regulator calculates the second correlation between the inference task and each token in the core visual token set, and deletes tokens from the core visual token set based on the second correlation.
7. The method according to claim 6, characterized in that, After the middle-layer dynamic token regulator monitors the real-time inference status of the middle layer for the inference task, the method further includes: When the real-time inference state reflects the lack of inference information for the inference task in the core visual token set, the third correlation between the inference task and each token in the discarded token set is calculated, and the tokens in the discarded token set are revived based on the third correlation.
8. A video understanding device based on dynamic tokens, characterized in that, The device includes: The task acquisition module is used to acquire the target video and the inference task; The task reasoning module is used to input the target video and the reasoning task into the video large model, so as to complete the task reasoning by performing the following steps through the video large model: The target video is sampled and encoded frame by frame to obtain an initial visual token set. The tokens in the initial visual token set are eliminated based on similarity according to the reasoning task to obtain a core visual token set. After the core visual token set is sent to the shallow layer of the video model for processing, the processed core visual token set is passed to the middle layer of the video model for processing. During the processing of the core visual token set by the middle layer, the core visual token set is dynamically adjusted based on the real-time processing status to delete tokens in the core visual token set and / or revive eliminated tokens. The core visual token set obtained after the intermediate layer processing is passed to the deep layer of the large video model for further processing to determine the reasoning result of the reasoning task.
9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.
10. A computer device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.