Method and device for enhancing active response capability of video understanding model

Abstract

The invention discloses a method and a device for enhancing the active response capability of a video understanding model. The method comprises the following steps: acquiring a video stream and a query text for the video stream; inputting the query text into a text encoder to obtain a query vector; for each frame in the video stream, inputting the current frame into a visual encoder to obtain a feature vector of the current frame; obtaining the similarity between the feature vector and the query vector of the current frame; counting the number of pixels of which the value is a preset value in the pruning mask of the current frame; in response to the fact that the similarity of any frame in the video stream is larger than a first similarity threshold value and the ratio of the number of pixels to the total number of pixels of the current frame meets a preset condition, pruning the feature vectors of the any frame and the first preset number of frames before the any frame by using pruning masks of the any frame and the first preset number of frames before the any frame, feature vectors after pruning are obtained; and inputting the pruned feature vector and the query vector into a large language model to obtain response information for the query text.

Description

Technical Field

[0001] This disclosure generally relates to the field of artificial intelligence, and more specifically, to methods and apparatus for enhancing the proactive response capabilities of video understanding models. Background Technology

[0002] With the rapid development of artificial intelligence technology, the paradigm of video understanding is undergoing a fundamental shift from offline, post-event analysis to online, interactive streaming scenarios (such as embodied intelligence, autonomous driving, and real-time event monitoring). Although Video Large Language Models (Video-LLMs) have made significant progress in handling visual tasks recently, their design architecture is mainly designed for offline processing, typically treating video as static, finite batches of frames. However, in streaming data processing, facing continuous video streams with extremely high temporal redundancy, the traditional frame-by-frame full-processing method leads to excessively high computational costs and unacceptable response latency, making it difficult to meet the needs of real-time interaction.

[0003] Existing streaming video understanding technologies are mainly divided into passive response models and active response models. Passive models focus on efficient memory management for on-demand queries, but they only generate responses when explicitly prompted by the user, lacking the initiative for real-time interaction. In contrast, active response models aim to autonomously decide when to respond, but existing active response mechanisms are usually based on simple visual change detection or complex training schedulers, making it difficult to accurately grasp the timing of interactions. Models often generate false triggers due to irrelevant visual abrupt changes, or miss triggers due to subtle but semantically crucial visual changes, resulting in untimely responses or illusions, failing to meet the high requirements of context sensitivity and timeliness in real-time interaction. Summary of the Invention

[0004] The embodiments of this disclosure provide a method and apparatus for enhancing the active response capability of a video understanding model, which can effectively solve the problem that active response models in the prior art often fail to respond in a timely manner or produce illusions.

[0005] In one general aspect, a method is provided to enhance the active response capability of a video understanding model, which includes a text encoder, a visual encoder, and a large language model. The method includes: acquiring a video stream and a query text for the video stream; inputting the query text into the text encoder to obtain a query vector; for each frame in the video stream, performing the following processing: inputting the current frame into the visual encoder to obtain a feature vector for the current frame; obtaining a first similarity between the feature vector of the current frame and the query vector; counting the number of pixels with a predetermined value in the pruning mask of the current frame; in response to the first similarity of any frame in the video stream being greater than a first similarity threshold and the ratio of the number of pixels to the total number of pixels in the current frame satisfying a preset condition, pruning the feature vectors of any frame and the first predetermined number of frames preceding any frame using the pruning mask of any frame and the first predetermined number of frames preceding any frame to obtain a pruned feature vector; inputting the pruned feature vector and the query vector into the large language model to obtain response information for the query text.

[0006] Optionally, the pruning mask of the current frame is obtained as follows: the current frame is divided into a second predetermined number of image blocks; for each image block in the second predetermined number of image blocks, the following processing is performed sequentially: the current image block is input into a visual encoder to obtain the feature vector of the current image block; the semantic mask of the current image block is obtained based on the similarity between the feature vector of the current image block and the query vector; the pruning mask of the current image block is determined based on the semantic mask of the current image block; and the pruning mask of the second predetermined number of image blocks is used as the pruning mask of the current frame.

[0007] Optionally, based on the similarity between the feature vector of the current image patch and the query vector, a semantic mask for the current image patch is obtained, including: in response to the similarity between the feature vector of the current image patch and the query vector being greater than a second similarity threshold, setting the pixel values ​​of all pixel positions of the current image patch to 1 as the semantic mask for the current image patch; in response to the similarity between the feature vector of the current image patch and the query vector being less than or equal to the second similarity threshold, setting the pixel values ​​of all pixel positions of the current image patch to 0 as the semantic mask for the current image patch.

[0008] Optionally, determining the pruning mask of the current image patch based on its semantic mask includes: obtaining a novelty mask of the current image patch based on the similarity between the feature vector of the current image patch and the historical feature vector of the current image patch, wherein the historical feature vector is determined based on the feature vector of a predetermined image patch, and the predetermined image patch is the image patch in all frames preceding the current frame in the video stream that corresponds to the position of the current image patch; and determining the pruning mask of the current image patch based on its semantic mask and novelty mask.

[0009] Optionally, a novelty mask for the current image patch is obtained based on the similarity between the feature vector of the current image patch and the historical feature vector of the current image patch, including: in response to the similarity between the feature vector of the current image patch and the historical feature vector of the current image patch being less than a third similarity threshold, setting the pixel value of all pixel positions of the current image patch to 1 as the novelty mask of the current image patch; in response to the similarity between the feature vector of the current image patch and the historical feature vector of the current image patch being greater than or equal to the third similarity threshold, setting the pixel value of all pixel positions of the current image patch to 0 as the novelty mask of the current image patch.

[0010] Optionally, after obtaining the novelty mask of the current image block based on the similarity between the feature vector of the current image block and the historical feature vector of the current image block, the method further includes: performing a weighted summation of the feature vector of the current image block and the historical feature vector to obtain the historical feature vector of the image block corresponding to the position of the current image block in the next frame of the current frame in the video stream.

[0011] In another general aspect, an apparatus is provided to enhance the active response capability of a video understanding model, the video understanding model including a text encoder, a visual encoder, and a large language model. The apparatus includes: a first acquisition unit configured to acquire a video stream and query text for the video stream; a second acquisition unit configured to input the query text into the text encoder to obtain a query vector; a processing unit configured to perform the following processing for each frame in the video stream: inputting the current frame into the visual encoder to obtain a feature vector of the current frame; acquiring a first similarity between the feature vector of the current frame and the query vector; counting the number of pixels in the pruning mask of the current frame that have a predetermined value; a pruning unit configured to, in response to a first similarity greater than a first similarity threshold and the ratio of the number of pixels to the total number of pixels in the current frame satisfying a preset condition, prune the feature vectors of any frame and the first predetermined number of frames preceding any frame using the pruning mask of any frame and the first predetermined number of frames preceding any frame to obtain a pruned feature vector; and a response unit configured to input the pruned feature vector and the query vector into the large language model to obtain response information for the query text.

[0012] Optionally, the above apparatus further includes a third acquisition unit, configured to acquire the pruning mask of the current frame by: dividing the current frame into a second predetermined number of image blocks; for each image block in the second predetermined number of image blocks, performing the following processing in sequence: inputting the current image block into a visual encoder to obtain the feature vector of the current image block; acquiring the semantic mask of the current image block based on the similarity between the feature vector of the current image block and the query vector; determining the pruning mask of the current image block based on the semantic mask of the current image block; and using the pruning mask of the second predetermined number of image blocks as the pruning mask of the current frame.

[0013] Optionally, the third acquisition unit is further configured to, in response to the similarity between the feature vector of the current image block and the query vector being greater than a second similarity threshold, set the pixel values ​​of all pixel positions of the current image block to 1 as a semantic mask of the current image block; and in response to the similarity between the feature vector of the current image block and the query vector being less than or equal to the second similarity threshold, set the pixel values ​​of all pixel positions of the current image block to 0 as a semantic mask of the current image block.

[0014] Optionally, the third acquisition unit is further configured to acquire a novelty mask of the current image block based on the similarity between the feature vector of the current image block and the historical feature vector of the current image block, wherein the historical feature vector is determined based on the feature vector of a predetermined image block, and the predetermined image block is the image block in all previous frames of the video stream that corresponds to the position of the current image block; and to determine a pruning mask of the current image block based on the semantic mask and the novelty mask of the current image block.

[0015] Optionally, the third acquisition unit is further configured to, in response to the similarity between the feature vector of the current image block and the historical feature vector of the current image block being less than a third similarity threshold, set the pixel value of all pixel positions of the current image block to 1 as a novelty mask of the current image block; and in response to the similarity between the feature vector of the current image block and the historical feature vector of the current image block being greater than or equal to the third similarity threshold, set the pixel value of all pixel positions of the current image block to 0 as a novelty mask of the current image block.

[0016] Optionally, the third acquisition unit is further configured to, after acquiring the novelty mask of the current image block based on the similarity between the feature vector of the current image block and the historical feature vector of the current image block, perform a weighted summation of the feature vector of the current image block and the historical feature vector to obtain the historical feature vector of the image block corresponding to the position of the current image block in the next frame of the current frame in the video stream.

[0017] In another general aspect, a computer-readable storage medium is provided for storing instructions, wherein when the instructions are executed by at least one computing device, they cause at least one computing device to perform a method for the active response capability of any of the above-described enhanced video understanding models.

[0018] In another general aspect, a system is provided that includes at least one computing device and at least one storage device for storing instructions, wherein the instructions, when executed by the at least one computing device, cause the at least one computing device to perform a method for the active response capability of any of the above-described enhanced video understanding models.

[0019] In another general aspect, a computer program product is provided, including computer instructions that, when executed by a processor, implement the active responsiveness of any of the above-described enhanced video understanding models.

[0020] The method and apparatus for enhancing the active response capability of a video understanding model according to embodiments of this disclosure utilize a dual-gated active response mechanism of semantic relevance and retained information density. This enables the enhanced video understanding model to autonomously schedule response timing based on semantic relevance and retained information density, overcoming the problems of false triggering due to irrelevant visual mutations or missed triggering due to subtle key events in related technologies. This significantly improves the naturalness and timeliness of human-computer interaction. Furthermore, the lightweight architecture of this disclosure, which requires no training, has extremely strong generalization capabilities. It is not only suitable for online streaming tasks but also effectively improves the accuracy of offline long video understanding, providing solid technical support for building efficient, intelligent, and robust real-time video understanding systems. Therefore, this disclosure effectively solves the problem that active response models in the prior art often respond untimely or produce illusions.

[0021] Further aspects and / or advantages of the general concept of this disclosure will be set forth in part in the description which follows, and in part will be clear from the description or may be learned by practice of the general concept of this disclosure. Attached Figure Description

[0022] The above and other objects and features of the embodiments of this disclosure will become clearer from the following description taken in conjunction with the accompanying drawings illustrating the embodiments, wherein: Figure 1 This is a flowchart illustrating a method for enhancing the proactive response capability of a video understanding model according to an embodiment of the present disclosure; Figure 2 This is a flowchart illustrating an embodiment of the present disclosure of a method for enhancing streaming video understanding efficiency and proactive interaction capabilities. Figure 3 This is a schematic diagram illustrating the overall structure of a system for enhancing streaming video understanding efficiency and proactive interaction capabilities according to an embodiment of the present disclosure; Figure 4 This is a comparison diagram showing the response results of the query-aware active response model of this disclosure and the passive response model and active response model in related technologies, illustrating an embodiment of this disclosure. Figure 5 This is a block diagram illustrating the active response capability of an enhanced video understanding model according to an embodiment of the present disclosure. Detailed Implementation

[0023] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but may be changed as will become clear upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for clarity and conciseness, descriptions of features known in the art may be omitted.

[0024] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein are provided only to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein, which will become clear upon understanding the disclosure of this application.

[0025] As used herein, the term “and / or” includes any one of the associated listed items and any combination of any two or more.

[0026] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, assemblies, regions, layers, or parts, these components, assemblies, regions, layers, or parts should not be limited by these terms. Rather, these terms are used only to distinguish one component, assembly, region, layer, or part from another. Thus, without departing from the teaching of the examples described herein, the first component, first assembly, first region, first layer, or first part referred to as the first component, first assembly, first region, first layer, or first part may also be referred to as the second component, second assembly, second region, second layer, or second part.

[0027] In the specification, when an element (such as a layer, region, or substrate) is described as being "on" another element, "connected to," or "bonded to" another element, the element may be directly "on" another element, directly "connected to," or "bonded to" the other element, or one or more other elements may be present in between. Conversely, when an element is described as being "directly on" another element, "directly connected to," or "directly bonded to" another element, no other elements may be present in between.

[0028] The terminology used herein is for the purpose of describing various examples only and is not intended to limit disclosure. Unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. The terms “comprising,” “including,” and “having” indicate the presence of the described features, quantities, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0029] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains upon understanding this disclosure. Unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and in this disclosure, and shall not be interpreted in an idealized or overly formalistic manner.

[0030] Furthermore, in the description of the examples, detailed descriptions of well-known related structures or functions will be omitted when it is believed that such detailed descriptions would lead to a vague interpretation of this disclosure.

[0031] Existing proactive response models aim to autonomously decide when to respond, but they often rely on a flawed, query-agnostic "change is important" principle. For example, some existing techniques simultaneously perform token pruning and response triggering by detecting visual changes. This approach incorrectly equates raw visual dynamics with semantic relevance, causing the model to fail to effectively distinguish visual noise from key information when processing video streams containing a large amount of irrelevant background motion or scene transitions. This results in a significant waste of computational resources and introduces irrelevant interference, affecting inference accuracy.

[0032] Moreover, the lack of query awareness in existing proactive response models leads to numerous challenges and limitations when processing complex streaming video. For example, the models are easily misled by semantically irrelevant visual dynamics (such as abrupt scene changes or the actions of irrelevant people in the background), resulting in wasted computational resources and false triggers. At the same time, the models struggle to separate visually subtle but crucial events that are essential to the user's query from surrounding visual noise, leading to missed triggers.

[0033] Furthermore, while existing visual token pruning strategies have introduced adaptive mechanisms, most are still based on inter-frame dissimilarity and fail to incorporate the user's query intent into the core processing flow. This can lead to the loss of key semantic clues when filtering redundant information, making it impossible to build a truly efficient and robust online video understanding system.

[0034] This disclosure aims to provide a method for enhancing the efficiency and proactive interaction capabilities of streaming video understanding, referred to as QueryStream. Specifically, it introduces a Relevance-Triggered Active Response (RTAR) strategy and a Query-Aware Differential Pruning (QDP) mechanism. The RTAR strategy addresses the "when to speak" challenge in streaming interaction through a dual-gating mechanism. This mechanism, using both semantic relevance and retained information density as gates, enables the enhanced video understanding model to autonomously schedule response times based on semantic relevance and retained information density. This overcomes the problems of false triggering due to irrelevant visual mutations or missed triggering due to subtle key events in relevance technologies, significantly improving the naturalness and timeliness of human-computer interaction. The QDP mechanism, by jointly evaluating the novelty of semantic relevance and dynamically smoothed history, achieves precise filtering of redundant visual information. This significantly reduces computational overhead while also serving as contextual denoising, ensuring the model focuses on key visual cues. This allows it to maintain high inference performance even with minimal visual data retention, effectively addressing the shortcomings of traditional methods that equate visual dynamics with semantic relevance.

[0035] Therefore, this disclosure solves the problem of confusion between visual dynamics and semantic relevance in the prior art, enabling the model to dynamically filter redundant visual streams and accurately schedule response times based on user intent, thereby significantly improving the computational efficiency and response accuracy of the model in streaming video scenarios.

[0036] The method and apparatus for enhancing the active response capability of the video understanding model disclosed herein will be described in detail below with reference to the accompanying drawings.

[0037] This disclosure proposes a method to enhance the proactive response capability of video understanding models. Figure 1 This is a flowchart illustrating a method for enhancing the proactive response capability of a video understanding model according to embodiments of the present disclosure. (Refer to...) Figure 1 The video understanding model includes a text encoder, a visual encoder, and a large language model. The method for enhancing the active response capability of the video understanding model includes the following steps: In step S101, the video stream and the query text for the video stream are obtained.

[0038] In step S102, the query text is input into the text encoder to obtain the query vector.

[0039] In step S103, for each frame in the video stream, the following processing is performed: input the current frame into the visual encoder to obtain the feature vector of the current frame; obtain the first similarity between the feature vector of the current frame and the query vector; count the number of pixels with a predetermined value in the pruning mask of the current frame.

[0040] In step S104, in response to the first similarity of any frame in the video stream being greater than the first similarity threshold and the ratio of the number of pixels to the total number of pixels in the current frame satisfying a preset condition, the feature vectors of any frame and the first predetermined number of frames preceding any frame are pruned using the pruning mask of any frame and the first predetermined number of frames preceding any frame to obtain the pruned feature vector.

[0041] In step S105, the pruned feature vector and query vector are input into the large language model to obtain response information for the query text.

[0042] As an example, the video stream mentioned above can be a real-time video stream, such as a surveillance video stream, or a pre-stored video stream, such as a video tutorial on opening a bottle. This disclosure does not limit the scope of the video stream.

[0043] As an example, the query text can vary depending on the content of the video stream. For instance, when the video stream is a surveillance video stream, the query text could be "monitoring the time when the child appears"; or, when the video stream is a bottle-opening tutorial video, the query text could be "how to open a bottle correctly". This disclosure does not limit the scope of the query text.

[0044] As an example, the above-mentioned pruning mask can be obtained using pruning strategies in related technologies, or other strategies can be used, such as determining the pruning mask based on semantic relevance, etc. This disclosure does not limit the scope of the pruning mask.

[0045] As an example, when the preset value is 1, the above preset condition is that the ratio of the number of pixels to the total number of pixels in the current frame exceeds a preset density threshold; when the preset value is 0, the above preset condition is that the ratio of the number of pixels to the total number of pixels in the current frame is less than a preset density threshold, and this disclosure does not limit this.

[0046] As an example, the value of the first predetermined quantity mentioned above is related to the input length of the large language model, that is, the first predetermined quantity can be the number of video frames corresponding to the input length of the large language model minus the length of any of the above frames.

[0047] As an example, the aforementioned first similarity can be cosine similarity, and this disclosure does not limit this. The aforementioned first similarity threshold can be set as needed, and this disclosure does not limit this.

[0048] As an example, this embodiment provides a Relevance Triggered Active Response (RTAR) strategy, which solves the problem of "when to speak" in streaming interaction through a dual gating mechanism. The dual gating consists of a relevance condition gating and an information density condition gating, specifically: Relevance conditional gating ( This is used to prevent the model from being misled by visually dramatic but semantically irrelevant events (such as camera cuts or passersby). First, for each frame in the video stream, the feature vector of the current frame is calculated. and query vector The cosine similarity is calculated, and then it is determined whether the cosine similarity is greater than a preset similarity threshold. If the cosine similarity is greater than a preset similarity threshold, it means the current frame matches the user's intent, and the relevance condition is met. At this point, a setting can be made. .

[0049] Information density conditional gating ( This is used to ensure that the response is triggered by a substantial influx of new information. First, for each frame in the video stream, the number of pixels retained in the current frame is counted. Then, it is determined whether this number exceeds a preset density threshold. If the number exceeds the preset density threshold... This indicates the emergence of numerous and important new visual cues, fulfilling the information density condition, at which point we can set... .

[0050] Finally, based on the simultaneous fulfillment of the relevance and information density conditions in any frame, the response information for the query text is triggered. That is, when the relevance and information density conditions in any frame are simultaneously fulfilled, the feature vectors of any frame and the first predetermined number of frames preceding it are pruned to obtain the pruned feature vectors. The pruned feature vectors and the query vector are then input into the large language model to obtain the response information for the query text.

[0051] Here, whether the correlation condition and the information density condition are simultaneously met can be determined by the logical AND operation result of the two gating states mentioned above, which is the trigger signal that triggers the subsequent response action. It can be represented as follows: (1) Only when When the condition is true, the large language model (such as the backbone Video-LLM) is activated, and the pruned feature vectors are input into the large language model to generate a natural language response. This mechanism effectively avoids false triggers and missed triggers.

[0052] According to embodiments of this disclosure, the pruning mask of the current frame in step S103 can be obtained by, but is not limited to, the following methods: dividing the current frame into a second predetermined number of image blocks; for each image block in the second predetermined number of image blocks, performing the following processing in sequence: inputting the current image block into a visual encoder to obtain the feature vector of the current image block; obtaining the semantic mask of the current image block based on the similarity between the feature vector of the current image block and the query vector; determining the pruning mask of the current image block based on the semantic mask of the current image block; and using the pruning mask of the second predetermined number of image blocks as the pruning mask of the current frame.

[0053] This embodiment identifies and retains image regions that are semantically highly relevant to the user's query text, ensuring that the model can automatically focus on the core visual objects related to the question and exclude irrelevant backgrounds.

[0054] As an example, the aforementioned first predetermined quantity can be set as needed, and this disclosure does not limit it. The aforementioned similarity can be cosine similarity or other similarities, and this disclosure does not limit it.

[0055] As an example, firstly, the current frame is divided into a second predetermined number of image blocks, and each image block is input into a visual encoder to obtain the feature vector of each image block; then, the feature vector of each image block is calculated. respectively with query vector The semantic mask for each image block is determined based on the cosine similarity. For example, the semantic mask of an image block with a similarity greater than a certain threshold is set to 1; this disclosure does not limit this. Then, the semantic mask of each image block can be used as the pruning mask for each image block individually, or the pruning mask for each image block can be determined by combining the semantic mask with other information, such as the novelty mask mentioned below; this disclosure does not limit this either. Finally, the pruning mask of each image block is used as the pruning mask for the corresponding frame.

[0056] According to embodiments of this disclosure, obtaining the semantic mask of the current image patch based on the similarity between the feature vector and the query vector can include: in response to the similarity between the feature vector and the query vector being greater than a second similarity threshold, setting the pixel values ​​of all pixel positions in the current image patch to 1, as the semantic mask of the current image patch; and in response to the similarity between the feature vector and the query vector being less than or equal to the second similarity threshold, setting the pixel values ​​of all pixel positions in the current image patch to 0, as the semantic mask of the current image patch. This embodiment allows for the rapid and accurate acquisition of the semantic mask of an image patch.

[0057] As an example, the aforementioned second similarity threshold can be set as needed, and this disclosure does not limit it. For example, to adapt to the changing complexity of different video scenes, a dynamic adaptive threshold strategy can be adopted, that is, the average similarity of all image blocks in the current frame can be used as the second similarity threshold.

[0058] As an example, assuming the first similarity threshold is the average similarity of all image patches within the current frame, semantic mask... The calculation logic can be as follows: (2) in, For the first The total number of image blocks within a frame; This is an indicator function that sets the value to 1 when the condition is met. For the first Intra-frame Feature vectors of image patches; For the first Intra-frame Feature vectors of image patches; For query vector; To calculate the similarity function, this mechanism ensures that the model can automatically focus on the core visual objects relevant to the problem, excluding irrelevant background.

[0059] According to embodiments of this disclosure, determining the pruning mask of the current image block based on the semantic mask of the current image block may include: obtaining a novelty mask of the current image block based on the similarity between the feature vector of the current image block and the historical feature vector of the current image block, wherein the historical feature vector is determined based on the feature vector of a predetermined image block, and the predetermined image block is the image block in all frames preceding the current frame in the video stream that corresponds to the position of the current image block; and determining the pruning mask of the current image block based on the semantic mask and the novelty mask of the current image block.

[0060] This embodiment also introduces a Dynamic Smoothing History (DSH) mechanism to accurately capture real visual dynamics and enhance robustness to transient noise. Furthermore, this embodiment jointly evaluates the novelty of semantic relevance and dynamic smoothing history, achieving precise filtering of redundant visual information. This significantly reduces computational overhead while playing a role in contextual denoising, ensuring that the model focuses on key visual cues. As a result, it can maintain a high level of inference performance while retaining very little visual data, thus effectively solving the defect of traditional methods that equate visual dynamics with semantic relevance.

[0061] As an example, when the current frame is the first frame of the video stream, the historical feature vector of each image block in the current frame is an empty vector. When the current frame is the second frame of the video stream, the historical feature vector of each image block in the current frame is the feature vector of the corresponding image block in the first frame.

[0062] Specifically, this embodiment abandons simple frame-to-frame comparison, and instead performs a comparison based on the position of each image block. Maintain a historical feature vector First, calculate the current feature vector for each image patch. With historical feature vectors The cosine similarity is used to determine the novelty mask for each image patch. For example, the novelty mask of image patches with a cosine similarity less than a certain threshold is set to 1, which is not limited in this disclosure; then, the pruning mask of each image patch can be determined based on the semantic mask and the novelty mask of each image patch. At this point, the final pruning decision is a logical combination of two dimensions: semantic relevance and novelty; only when an image patch (i.e., a visual token) is both semantically relevant to the query text and temporally novel relative to its historical context will it be retained. Thus, after applying this final pruning mask, the visual stream sent to the backend large language model achieves efficient "contextual denoising". The final pruning mask The calculation logic can be as follows: (3) According to embodiments of this disclosure, obtaining the novelty mask of the current image patch based on the similarity between the feature vector of the current image patch and its historical feature vectors may include: in response to the similarity between the feature vector of the current image patch and its historical feature vectors being less than a third similarity threshold, setting the pixel values ​​of all pixel positions in the current image patch to 1 as the novelty mask of the current image patch; and in response to the similarity between the feature vector of the current image patch and its historical feature vectors being greater than or equal to the third similarity threshold, setting the pixel values ​​of all pixel positions in the current image patch to 0 as the novelty mask of the current image patch. Through this embodiment, the semantic mask of an image patch can be obtained quickly and accurately.

[0063] As an example, the aforementioned third similarity threshold can be set as needed, and this disclosure does not limit it.

[0064] As an example, after obtaining the current feature vector of each image patch... With historical feature vectors After calculating the cosine similarity, it can be determined whether the cosine similarity of each image patch is lower than a preset similarity threshold. If the judgment result of a certain image patch is yes, then the novelty mask of that image patch is... All element values ​​are set to 1. If the judgment result of a certain image patch is negative, then the novelty mask of that image patch is... All element values ​​are set to 0.

[0065] According to embodiments of this disclosure, after obtaining the novelty mask of the current image block based on the similarity between the feature vector of the current image block and the historical feature vector of the current image block, the feature vector of the current image block and the historical feature vector can be weighted and summed to obtain the historical feature vector of the image block corresponding to the position of the current image block in the next frame of the current frame in the video stream.

[0066] This embodiment updates historical feature vectors in real time and continues processing the next frame of video, thereby achieving efficient, accurate, and user-intent-compliant streaming interaction.

[0067] As an example, after obtaining the novelty mask for each image patch in the current frame, the following exponential moving average formula combined with a smoothing factor can be used. The historical feature vector of each image patch in the next frame is updated using the feature vector of each image patch in the current frame. That is, the historical feature vector of each image patch in the next frame is obtained using the Dynamic Smooth History (DSH) mechanism. (4) in, Indicates the first t Intra-frame i Historical feature vectors of image patches Indicates the first t Intra-frame i Feature vectors of image patches.

[0068] This embodiment introduces a smoothing factor. It can effectively smooth out minor visual jitters, thereby enhancing the model's robustness to slow visual drift and transient noise, and ensuring that only significant and persistent state changes are recorded.

[0069] To better understand this disclosure, the following is in conjunction with... Figure 2 and Figure 3 Provide a systematic explanation.

[0070] Figure 2 A flowchart illustrating a method for enhancing the efficiency and proactive interactivity of streaming video understanding is shown, such as... Figure 2 As shown, the method includes the following steps: Step S201: Receive continuous video stream input and user query text, use a lightweight visual-text encoder to extract features from image blocks in each frame of the video stream to obtain feature vectors for each image block, and encode the user query text to obtain query embedding vectors.

[0071] Step S202: Define a query-aware differential pruning (QDP) strategy, which includes two orthogonal evaluation dimensions: semantic relevance filtering and time novelty filtering based on dynamic smooth history (DSH); and define a relevance-triggered proactive response (RTAR) strategy to dynamically schedule response timing.

[0072] Step S203: Using the extracted feature vectors of the image patches and the encoded query embedding vectors, perform the query-aware differential pruning (QDP) process, that is, calculate the semantic similarity between the feature vector of each image patch and the query embedding vector to generate a block semantic mask for each image; at the same time, calculate the difference between the current feature vector and the historical feature vector of each image patch to generate a novelty mask; take the logical intersection of the semantic mask and the novelty mask to generate the final pruning mask, so that only visual tokens that have both semantic relevance and temporal novelty are retained.

[0073] Step S204: Using the feature vector of the extracted image patch and the encoded query embedding vector, perform the Relevance Triggered Active Response (RTAR) process. Introduce a dual gating mechanism to evaluate the current frame, that is, calculate the overall correlation between the feature vector and the query embedding vector of each frame and determine the retained information density based on the pruning mask. A trigger signal will only be generated when the overall correlation and the retained information density simultaneously meet their respective conditions.

[0074] Step S205: Based on the trigger signal generated above, at the trigger time, the accumulated original visual tokens (i.e., the feature vectors of image blocks of any frame and a predetermined number of frames preceding them that simultaneously satisfy their respective conditions) are filtered using the pruning mask generated above. The filtered visual tokens, together with the query embedding vector, are input into the frozen backbone video large language model (Video-LLM) to generate an immediate and context-sensitive natural language response.

[0075] Figure 3 This diagram illustrates the overall structure of a system for enhancing the efficiency of streaming video understanding and its proactive interactive capabilities. Figure 3 As shown, this embodiment aims to learn how to properly open a bottle of champagne from a video stream. The entire process mainly includes the following parts: 1) System initialization and feature encoding module The system input consists of a video stream for teaching and the user's query text. First, the system preprocesses each frame of the video stream and the query text using a lightweight visual-text encoder (e.g., OpenCLIP-ViT-L / 14). Specifically, for each frame in the video stream, the visual encoder extracts the feature vectors of its divided image patches, while the text encoder encodes the user's input query text into a query vector. This step provides the basic semantic representation for subsequent intent perception and information filtering, ensuring that the model can understand the relationship between visual content and user intent from the underlying features.

[0076] 2) Query-Aware Differential Pruning (QDP) Module To efficiently filter redundant information, this embodiment designs a Query-Aware Differential Pruning (QDP) module. This module abandons the traditional filtering principle based solely on pixel changes or inter-frame differences, and instead employs a dual evaluation mechanism based on semantic relevance and temporal novelty. Specifically: Semantic relevance filtering involves calculating the cosine similarity between the feature vector of each image patch and the query vector, and then determining the semantic mask for each image patch based on this cosine similarity. .

[0077] Temporal novelty filtering involves calculating the cosine similarity between the current feature vector and the historical feature vectors of each image patch, and then determining the novelty mask for each image patch based on this cosine similarity. .

[0078] Finally, based on semantic masks and novelty mask Determine the final pruning mask for each image patch. .

[0079] 3) Relevance Triggered Active Response (RTAR) module The RTAR module is responsible for the response timing of the intelligent scheduling model. It solves the problem of "when to speak" in streaming interaction through a dual gating mechanism. Specifically: Relevance conditional gating ( This involves calculating the cosine similarity between the feature vector and the query vector for each frame. The relevance condition for a frame is only met when the cosine similarity exceeds a preset threshold. .

[0080] Information density conditional gating ( The final pruning mask can be obtained by using the output of the QDP module. To quantify the retained information density, the proportion of pixels retained in each frame is statistically analyzed. The information density condition for a frame is only met when this proportion exceeds a preset density threshold. .

[0081] Finally, a frame can only generate a trigger signal, i.e., trigger the large language model to output response information, when both the relevance condition and the information density condition of a particular frame are met simultaneously.

[0082] 4) Sparse Inference and Response Generation Module After generating the trigger signal, this embodiment enters the final inference and response generation stage. Specifically, the cached sparse visual token sequence, filtered by the QDP mechanism, is extracted. This is done by filtering using the pruning mask of each frame, along with multimodal rotation position coding (M-ROPE). During this process, the system discards the position codes of pruned visual tokens, while strictly preserving the original time, height, and width coordinates of the remaining visual tokens to maintain the integrity and accuracy of the spatiotemporal information. Subsequently, these extracted sparse visual tokens carrying position information, along with the user's query text (as shown in the query tokens), are input into a video large language model (such as Qwen2.5-VL) with frozen parameters. The large language model generates a natural language response based on this highly refined visual context and provides it back to the user.

[0083] It should be noted that if no trigger signal is generated at a certain moment, the system remains silent and only updates the historical feature vector of each image block. For example, it uses the Dynamic Smooth History (DSH) mechanism to update the historical feature vector of each image block in the next frame and continues to process the next frame, thereby achieving efficient, accurate and user-intent-compliant streaming interaction.

[0084] In summary, this disclosure constructs a query-aware differential pruning mechanism and a relevance-triggered proactive response strategy, enabling the video understanding model to accurately filter redundant visual information based on joint evaluation of user intent and dynamic history. Furthermore, it automatically schedules the optimal response time through a dual gating mechanism, significantly improving the computational efficiency and response accuracy of the streaming video understanding model in real-time interactive scenarios.

[0085] To evaluate the effectiveness of the embodiments of this disclosure, a series of experiments were conducted to verify the effectiveness of the embodiments of this disclosure (such as the QueryStream method). The test environment was built based on the officially recommended configuration, with the backbone model Qwen2.5-VL-7B integrated with the Query Aware Differential Pruning (QDP) module and the Relevance Triggered Active Response (RTAR) module. For the baseline models used in comparison, their officially released inference code and pre-trained weights were employed. To comprehensively verify the effectiveness of the QueryStream method of this disclosure, experiments were conducted on several widely recognized video language benchmarks, including StreamingBench and OVO-Bench, which focus on real-time streaming understanding, and LongVideoBench and VideoMME, which focus on offline long-form video understanding. These benchmarks cover a variety of scenarios, from short-term real-time interaction to long-term contextual reasoning, and can comprehensively evaluate the performance of large-scale video language models (Video-LLMs).

[0086] Table 1. Overall performance score of the QueryStream method disclosed in this paper compared with other representative models.

[0087] As shown in Table 1, the QueryStream method of this disclosure is compared with several powerful multimodal models, including the proprietary closed-source models GPT-4o and Gemini-1.5-Pro, the open-source full-processing models InternVL2-8B and Qwen2.5-VL-7B, and the existing state-of-the-art streaming understanding model TimeChat-Online-7B. GPT-4o and Gemini-1.5-Pro ​​represent the upper limit of current multimodal inference capabilities, capable of handling extremely complex long video tasks. InternVL2-8B and Qwen2.5-VL-7B, as powerful open-source foundations, demonstrate performance benchmarks without pruning by processing full video frames. Of particular note is TimeChat-Online-7B, which employs a query-independent pruning strategy based on visual variations and is the most direct competitor of this disclosure. In addition, a variant of "Qwen2.5-VL-7B w / QDP" is introduced to verify the zero-shot generalization capability of the QDP module of this disclosure on offline models. These comparisons can verify the effectiveness of the QueryStream method from multiple dimensions, including computational efficiency, inference accuracy, and generalization ability. As shown in Table 1, the QueryStream method of this disclosure outperforms the base model under the same conditions on all benchmarks. Compared with related technologies, this disclosure significantly improves the inference accuracy and response time of the model while greatly reducing computational overhead (pruning more than 70% of visual tokens).

[0088] Furthermore, this disclosure also compares passive response (such as ReKV) models and active response models (such as TimeChat-Online) in related technologies with the query-aware active response model (QueryStream) of this disclosure, and the comparison results are as follows: Figure 4 As shown, the query-aware proactive response model QueryStream disclosed in this invention has better response capabilities than passive response (such as ReKV) models and proactive response models (such as TimeChat-Online).

[0089] Figure 5 This is a block diagram illustrating an apparatus for enhancing the active response capability of a video understanding model according to an embodiment of the present disclosure, such as... Figure 5 As shown, the video understanding model includes a text encoder, a visual encoder, and a large language model, and the device includes: The first acquisition unit 50 is configured to acquire the video stream and the query text for the video stream; the second acquisition unit 52 is configured to input the query text into a text encoder to obtain a query vector; the processing unit 54 is configured to perform the following processing for each frame in the video stream: input the current frame into a visual encoder to obtain the feature vector of the current frame; acquire the first similarity between the feature vector of the current frame and the query vector; count the number of pixels with a predetermined value in the pruning mask of the current frame; the pruning unit 56 is configured to, in response to the first similarity of any frame in the video stream being greater than a first similarity threshold and the ratio of the number of pixels to the total number of pixels in the current frame satisfying a preset condition, prune the feature vectors of any frame and the first predetermined number of frames preceding any frame using the pruning mask of any frame and the first predetermined number of frames preceding any frame to obtain the pruned feature vector; the response unit 58 is configured to input the pruned feature vector and the query vector into a large language model to obtain response information for the query text.

[0090] Optionally, the above apparatus further includes a third acquisition unit, configured to acquire the pruning mask of the current frame by: dividing the current frame into a second predetermined number of image blocks; for each image block in the second predetermined number of image blocks, performing the following processing in sequence: inputting the current image block into a visual encoder to obtain the feature vector of the current image block; acquiring the semantic mask of the current image block based on the similarity between the feature vector of the current image block and the query vector; determining the pruning mask of the current image block based on the semantic mask of the current image block; and using the pruning mask of the second predetermined number of image blocks as the pruning mask of the current frame.

[0091] Optionally, the third acquisition unit is further configured to, in response to the similarity between the feature vector of the current image block and the query vector being greater than a second similarity threshold, set the pixel values ​​of all pixel positions of the current image block to 1 as a semantic mask of the current image block; and in response to the similarity between the feature vector of the current image block and the query vector being less than or equal to the second similarity threshold, set the pixel values ​​of all pixel positions of the current image block to 0 as a semantic mask of the current image block.

[0092] Optionally, the third acquisition unit is further configured to acquire a novelty mask of the current image block based on the similarity between the feature vector of the current image block and the historical feature vector of the current image block, wherein the historical feature vector is determined based on the feature vector of a predetermined image block, and the predetermined image block is the image block in all previous frames of the video stream that corresponds to the position of the current image block; and to determine a pruning mask of the current image block based on the semantic mask and the novelty mask of the current image block.

[0093] Optionally, the third acquisition unit is further configured to, in response to the similarity between the feature vector of the current image block and the historical feature vector of the current image block being less than a third similarity threshold, set the pixel value of all pixel positions of the current image block to 1 as a novelty mask of the current image block; and in response to the similarity between the feature vector of the current image block and the historical feature vector of the current image block being greater than or equal to the third similarity threshold, set the pixel value of all pixel positions of the current image block to 0 as a novelty mask of the current image block.

[0094] Optionally, the third acquisition unit is further configured to, after acquiring the novelty mask of the current image block based on the similarity between the feature vector of the current image block and the historical feature vector of the current image block, perform a weighted summation of the feature vector of the current image block and the historical feature vector to obtain the historical feature vector of the image block corresponding to the position of the current image block in the next frame of the current frame in the video stream.

[0095] According to embodiments of the present disclosure, a computer-readable storage medium for storing instructions is provided, wherein when the instructions are executed by at least one computing device, they cause at least one computing device to perform a method for enhancing the active response capability of a video understanding model as described in any of the above embodiments.

[0096] According to embodiments of the present disclosure, a system is provided that includes at least one computing device and at least one storage device storing instructions, wherein the instructions, when executed by at least one computing device, cause at least one computing device to perform a method for enhancing the active response capability of a video understanding model as described in any of the above embodiments.

[0097] According to embodiments of this disclosure, a computer program product is provided, including computer instructions that, when executed by a processor, implement the active response capability of any of the above-described enhanced video understanding models.

[0098] While some embodiments of this disclosure have been shown and described, those skilled in the art will understand that modifications may be made to these embodiments without departing from the principles and spirit of this disclosure, which are defined by the claims and their equivalents.

Claims

1. A method for enhancing the active response capability of a video understanding model, characterized in that, The video understanding model includes a text encoder, a visual encoder, and a large language model; the method includes: Obtain the video stream and the query text for the video stream; The query text is input into the text encoder to obtain the query vector; For each frame in the video stream, the following processing is performed: The current frame is input into the visual encoder to obtain the feature vector of the current frame; Obtain the first similarity between the feature vector and the query vector of the current frame; Count the number of pixels in the current frame whose pruning mask has a predetermined value; In response to the first similarity of any frame in the video stream being greater than the first similarity threshold and the ratio of the number of pixels to the total number of pixels in the current frame satisfying a preset condition, the feature vectors of any frame and the first predetermined number of frames preceding the frame are pruned using the pruning mask of the first predetermined number of frames preceding the frame to obtain the pruned feature vectors. The pruned feature vector and the query vector are input into the large language model to obtain response information for the query text.

2. The method as described in claim 1, characterized in that, The pruning mask for the current frame is obtained in the following way: Divide the current frame into a second predetermined number of image blocks; For each image block in the second predetermined number of image blocks, the following processing is performed sequentially: The current image patch is input into the visual encoder to obtain the feature vector of the current image patch; Based on the similarity between the feature vector of the current image patch and the query vector, obtain the semantic mask of the current image patch; Based on the semantic mask of the current image patch, determine the pruning mask of the current image patch; The pruning mask of the second predetermined number of image blocks is used as the pruning mask of the current frame.

3. The method as described in claim 2, characterized in that, The step of obtaining the semantic mask of the current image patch based on the similarity between the feature vector of the current image patch and the query vector includes: In response to the fact that the similarity between the feature vector of the current image block and the query vector is greater than a second similarity threshold, the pixel values ​​of all pixel positions of the current image block are set to 1, which is used as the semantic mask of the current image block; In response to the similarity between the feature vector of the current image block and the query vector being less than or equal to the second similarity threshold, the pixel values ​​of all pixel positions of the current image block are set to 0, which serves as the semantic mask of the current image block.

4. The method as described in claim 2, characterized in that, Determining the pruning mask of the current image patch based on its semantic mask includes: Based on the similarity between the feature vector of the current image block and the historical feature vector of the current image block, a novelty mask of the current image block is obtained. The historical feature vector is determined based on the feature vector of a predetermined image block, which is the image block in the video stream that corresponds to the position of the current image block in all frames preceding the current frame. Based on the semantic mask and novelty mask of the current image patch, determine the pruning mask of the current image patch.

5. The method as described in claim 4, characterized in that, The step of obtaining the novelty mask of the current image patch based on the similarity between the feature vector of the current image patch and the historical feature vector of the current image patch includes: In response to the fact that the similarity between the feature vector of the current image block and the historical feature vector of the current image block is less than the third similarity threshold, the pixel values ​​of all pixel positions of the current image block are set to 1, which is used as the novelty mask of the current image block. In response to the fact that the similarity between the feature vector of the current image block and the historical feature vector of the current image block is greater than or equal to the third similarity threshold, the pixel values ​​of all pixel positions of the current image block are set to 0, which serves as the novelty mask of the current image block.

6. The method as described in claim 4, characterized in that, After obtaining the novelty mask of the current image patch based on the similarity between the feature vector of the current image patch and the historical feature vector of the current image patch, the method further includes: The feature vector of the current image block and the historical feature vector are weighted and summed to obtain the historical feature vector of the image block corresponding to the position of the current image block in the next frame of the video stream.

7. An apparatus for enhancing the active response capability of a video understanding model, characterized in that, The video understanding model includes a text encoder, a visual encoder, and a large language model; the device includes: The first acquisition unit is configured to acquire a video stream and a query text for the video stream; The second acquisition unit is configured to input the query text into the text encoder to obtain a query vector; The processing unit is configured to perform the following processing for each frame in the video stream: The current frame is input into the visual encoder to obtain the feature vector of the current frame; Obtain the first similarity between the feature vector and the query vector of the current frame; Count the number of pixels in the current frame whose pruning mask has a predetermined value; The pruning unit is configured to prune the feature vectors of any frame and the first predetermined number of frames preceding any frame in the video stream in response to a first similarity greater than a first similarity threshold and the ratio of the number of pixels to the total number of pixels in the current frame satisfying a preset condition, by using the pruning mask of any frame and the first predetermined number of frames preceding any frame to obtain the pruned feature vector. The response unit is configured to input the pruned feature vector and the query vector into the large language model to obtain response information for the query text.

8. A computer-readable storage medium for storing instructions, characterized in that, When the instruction is executed by at least one computing device, it causes the at least one computing device to perform the method for enhancing the active response capability of the video understanding model as described in any one of claims 1 to 6.

9. A system comprising at least one computing device and at least one storage device for storing instructions, characterized in that, When the instruction is executed by the at least one computing device, it causes the at least one computing device to perform the method for enhancing the active response capability of the video understanding model as described in any one of claims 1 to 6.

10. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the method for enhancing the active response capability of the video understanding model as described in any one of claims 1 to 6.

Images