Training of a streaming video understanding model, streaming video understanding method and apparatus

Abstract

The application provides a kind of streaming video understanding model training, streaming video understanding method and device, wherein the method comprises: video frame features and word features are arranged in time sequence to obtain input sequence, thereby simulating the actual input mode of video data frame by frame continuous arrival in online scene, so that the model can accurately grasp the dynamic change of video content;Further based on the observable historical input corresponding to the current video frame determined by the initial streaming video understanding model and the output prediction description result, thereby under the streaming constraint of strictly prohibiting access to future video frames, the probability mapping relationship between the visual evidence at the current time and the historical semantic description is effectively established, the time sequence logic in streaming process is robustly modeled, avoid the model in actual online operation due to lack of active and reliable time decision basis and produce redundant repeated output for each incoming frame, guarantee accurate and timely streaming video understanding effect.

Description

Technical Field

[0001] This invention relates to the field of streaming video understanding technology, and in particular to a training method and apparatus for streaming video understanding. Background Technology

[0002] In recent years, large-visual language models have demonstrated remarkable capabilities in video processing, thanks to continuous advancements in spatial-temporal reasoning and long-context understanding. With the rapid development of applications such as live streaming, online monitoring, and sports commentary, the industry has an urgent technical demand for intelligent systems for online video understanding. These systems must be constantly operational, processing continuously arriving video streams over time and providing real-time, autonomous feedback.

[0003] To meet the aforementioned needs of online video understanding, existing technologies typically rely on video-based large language models trained offline. These approaches are primarily based on static image-text alignment targets, utilizing a limited number of pre-recorded complete video sequences for iterative model optimization. When applied to online scenarios, existing technologies often employ sliding window strategies to extract video segments or force the model to learn specific silence commands, attempting to make the model determine its response state when processing continuous frame-by-frame input.

[0004] However, training methods based on pre-recorded complete video sequences or static image-text alignment differ significantly from the actual input pattern of continuously arriving video data frame by frame in online scenarios. This makes it difficult for the model to accurately grasp the dynamic changes of video content when faced with constantly evolving long-term contexts. Secondly, under the streaming constraint of strictly prohibiting access to future video frames, existing solutions cannot effectively establish a probabilistic mapping relationship between visual evidence at the current moment and historical semantic descriptions, making it difficult to robustly model the temporal logic in the streaming process. This results in the model lacking proactive and reliable temporal decision-making basis in actual online operation, easily generating redundant and repetitive outputs for each incoming frame, or experiencing response delays due to target misalignment, failing to achieve accurate and timely streaming video understanding. Summary of the Invention

[0005] This invention provides a training method and apparatus for streaming video understanding, which addresses the shortcomings of existing streaming video understanding technologies. These shortcomings include a mismatch between the distribution of training data and online frame-by-frame input, difficulty in robustly modeling cross-frame temporal dependencies without the constraint of available future frames, lack of effective time decision-making mechanisms during online inference, and susceptibility to duplicate outputs or response delays, thus failing to achieve accurate and timely real-time understanding.

[0006] This invention provides a training method for a streaming video understanding model, wherein the streaming video understanding model is obtained by iteratively executing the following steps until a preset iteration termination condition is met: Acquire training sample data from streaming video; the training sample data includes multiple semantic segments divided according to semantic consistency, each semantic segment corresponding to a continuous video frame sequence and label description text aligned with the continuous video frame sequence; The video frame features of each video frame in the continuous video frame sequence are interleaved with the lexical features of the tag description text in chronological order to obtain the input sequence; The input sequence is input into the initial streaming video understanding model, which determines the observable historical input corresponding to each current video frame in the continuous video frame sequence based on the input sequence, and outputs the prediction description result based on the observable historical input corresponding to each current video frame. Based on the predicted description results and the labeled description text, the description generation loss is determined; The model parameters of the initial streaming video understanding model are updated based on the loss generated from the description.

[0007] According to a training method for a streaming video understanding model provided by the present invention, the step of determining the description generation loss based on the predicted description result and the labeled description text includes: For each word in the tag description text, based on the observable historical input and the generated words in the tag description text preceding each word, the conditional probability of each word in the predicted description result is determined; The joint probability is determined based on the conditional probabilities of all lexical units in the predicted description results; Based on the joint probability, the description generation loss is determined.

[0008] According to a training method for a streaming video understanding model provided by the present invention, the step of inputting the input sequence into an initial streaming video understanding model, wherein the initial streaming video understanding model determines the observable historical input corresponding to each current video frame in the continuous video frame sequence based on the input sequence, and outputs a prediction description result based on the observable historical input corresponding to each current video frame, includes: In the attention calculation of the initial streaming video understanding model, within the same semantic segment, attention connections are blocked between the video frame features of the current video frame and the lexical features corresponding to the current video frame as queries, and the video frame features of the observed frames and the lexical features corresponding to the observed frames as keys; the observed frames are video frames preceding the current video frame in the corresponding continuous video frame sequence within the same semantic segment. Between different semantic segments, an attention connection is maintained between the video frame features of the current video frame and the lexical features corresponding to the current video frame as the query, and the lexical features corresponding to the terminating descriptive lexical features of each semantic segment that is temporally earlier than the semantic segment to which the current video frame belongs as the key. The video frame features and word features corresponding to the historical positions in the input sequence where attention connections were not blocked are determined as the observable historical input of the current video frame. The predicted description result is obtained by performing autoregressive inference based on the observable historical input.

[0009] According to a training method for a streaming video understanding model provided by the present invention, the step of obtaining the lexical features of the tag description text includes: According to the preset word segmentation rules, the tag description text is segmented to obtain a word sequence; Based on a pre-built vocabulary, each lexical in the lexical sequence is mapped to a lexical index; The lexical indexes are vectorized and mapped to obtain vector mapping results. The positional codes of each lexical in the lexical sequence are superimposed on the vector mapping results to obtain the lexical features of the tag description text.

[0010] The present invention also provides a streaming video understanding method, comprising: Acquire streaming video data to be understood; the streaming video data to be understood includes a sequence of streaming video frames of the target to be understood, and task prompts in the streaming video frame sequence; The streaming video frame sequence and the task prompt are input into the streaming video understanding model to obtain the description prediction result output by the streaming video understanding model. The description prediction result is obtained by the streaming video understanding model after understanding the streaming video frame sequence based on the task type specified by the task prompt and guided by the task prompt. The streaming video understanding model is obtained by performing the training method described above.

[0011] According to a streaming video understanding method provided by the present invention, the method further includes a hierarchical memory management step, the hierarchical memory management step comprising: When the total number of lexical units in the context window of the streaming video understanding model reaches the preset length limit, the semantic segment to which the context window belongs is determined to be the target semantic segment. Determine the verification perplexity corresponding to each video frame within the target semantic segment, and based on the verification perplexity, determine the semantic divergence of the current video frame in the streaming video frame sequence; Determine the median of the semantic divergence of each video frame within the target semantic segment, and identify video frames within the target semantic segment with a semantic divergence lower than the median as frames to be compressed; Remove the video frame features and the corresponding lexical features of the frame to be compressed from the context window, and retain only the end-description lexical features of the target semantic segment to obtain the compressed target semantic segment; the end-description lexical features are the lexical features corresponding to the end-description lexical, and the end-description lexical is the lexical located at the end position of the tag description text in the target semantic segment.

[0012] According to a streaming video understanding method provided by the present invention, the method further includes: The remaining video frames in the target semantic segment, excluding the frame to be compressed, and the lexical features corresponding to the remaining video frames are taken as historical semantic units, and the historical semantic units are inserted into the recursive event tree. Based on the semantic similarity between the feature vector of the historical semantic unit and the feature vector of the existing tree node in the recursive event tree, the highest semantic similarity is determined; If the highest semantic similarity exceeds a preset semantic similarity threshold, the historical semantic unit is attached as a child node to the existing tree node corresponding to the highest semantic similarity; otherwise, a new root node is created to store the historical semantic unit. Update the semantic embedding of the parent node corresponding to the child node.

[0013] According to a streaming video understanding method provided by the present invention, updating the semantic embedding of the parent node corresponding to the child node includes: The existing tree node corresponding to the highest semantic similarity is determined as the parent node to be updated; Obtain the semantic embedding of the parent node to be updated, and the feature vector of the historical semantic unit attached as a child node; The preset momentum update rate is used as the first weight of the feature vector of the historical semantic unit, and the second weight of the semantic embedding of the parent node to be updated is determined based on the momentum update rate. Based on the first weight and the second weight, the feature vector of the historical semantic unit and the semantic embedding of the parent node to be updated are weighted and summed to obtain the updated semantic embedding of the parent node.

[0014] The present invention also provides a training apparatus for a streaming video understanding model, comprising the following units: The first acquisition unit is used to acquire training sample data of the streaming video; the training sample data includes multiple semantic segments divided according to semantic consistency, each semantic segment corresponds to a continuous video frame sequence and a tag description text aligned with the continuous video frame sequence; An arrangement unit is used to cross-arrange the video frame features of each video frame in the continuous video frame sequence with the lexical features of the tag description text in chronological order to obtain the input sequence; The prediction unit is used to input the input sequence into the initial streaming video understanding model, and the initial streaming video understanding model determines the observable historical input corresponding to each current video frame in the continuous video frame sequence based on the input sequence, and outputs the prediction description result based on the observable historical input corresponding to each current video frame. A loss determination unit is used to determine the description generation loss based on the predicted description result and the label description text; An update unit is used to update the model parameters of the initial streaming video understanding model based on the loss generated by the description.

[0015] The present invention also provides a streaming video understanding device, comprising the following units: The second acquisition unit is used to acquire streaming video data to be understood; the streaming video data to be understood includes a continuous video frame sequence of the target to be understood, and task prompt words in the continuous video frame sequence; An input unit is used to input the continuous video frame sequence and the task prompt words into a target streaming video understanding model to obtain a description prediction result output by the streaming video understanding model; the description prediction result is obtained by the streaming video understanding model after understanding the continuous video frame sequence based on the task type specified by the task prompt words and guided by the task prompt words. The streaming video understanding model is obtained by performing the training method described above.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a training method for a streaming video understanding model as described above, or implements a streaming video understanding method as described above.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a training method for a streaming video understanding model as described above, or implements a streaming video understanding method as described above.

[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements a training method for a streaming video understanding model as described above, or implements a streaming video understanding method as described above.

[0019] The present invention provides a training method and apparatus for a streaming video understanding model, which obtains an input sequence by cross-arranging video frame features and lexical features in chronological order. This simulates the actual input pattern of video data arriving frame by frame in an online scenario, solving the problem of significant data distribution differences between training methods based on pre-recorded complete video sequences or static image-text alignment and the actual input pattern. This enables the model to accurately grasp the dynamic changes of video content. Furthermore, based on the observable historical input corresponding to the current video frame determined by the initial streaming video understanding model, a predicted description result is output. Thus, under the streaming constraint of strictly prohibiting access to future video frames, a probabilistic mapping relationship between visual evidence at the current moment and historical semantic description is effectively established. The temporal logic in the streaming process is robustly modeled, avoiding redundant repeated outputs for each incoming frame due to the lack of proactive and reliable time decision-making basis in actual online operation, or response delays caused by target misalignment, ensuring accurate and timely streaming video understanding results. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the training method of the streaming video understanding model provided by the present invention.

[0022] Figure 2 This is a flowchart illustrating the streaming video understanding method provided by the present invention.

[0023] Figure 3 This is a schematic diagram of the streaming video understanding method provided by the present invention in a real-time narration task application.

[0024] Figure 4 This is a schematic diagram of the memory management process provided by the present invention.

[0025] Figure 5 This is a schematic diagram of the structure of the training device for the streaming video understanding model provided by the present invention.

[0026] Figure 6 This is a schematic diagram of the structure of the streaming video understanding device provided by the present invention.

[0027] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0029] In recent years, the rapid development of large visual language models has dramatically changed the field of video large language models, with related architectures demonstrating remarkable capabilities in complex multimodal tasks. However, these achievements have primarily been obtained in offline environments, where models process limited and pre-recorded video sequences. Unlike offline scenarios, agents such as live streaming assistants for online video understanding must remain constantly active, continuously inferring over time and providing real-time feedback. The core challenge lies in the fact that models need to process continuous, frame-by-frame video input while autonomously determining the most appropriate response time under strict streaming constraints that prohibit access to future frames. Training models solely through mute token prediction can lead to data imbalance and misalignment of training objectives; if models are allowed to focus on all historical information without restriction during training, they are prone to "copying" synonyms from previous frames when predicting the description of the current video frame, rather than relying on the visual evidence of the current video frame itself for reasoning. This results in unreliable probability distributions output by the trained model during the streaming inference phase.

[0030] This invention provides a training method for a streaming video understanding model. The subject executing the training method for the streaming video understanding model provided by this invention can be an electronic device, a component in an electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device, such as a server, a server cluster, a personal computer, a graphics processing unit computing platform, etc. This invention does not specifically limit the specific type of device. Figure 1 This is a flowchart illustrating the training method for the streaming video understanding model provided by this invention, as shown below. Figure 1As shown, the streaming video understanding model is obtained by iteratively executing the following steps until a preset iteration termination condition is met. The preset iteration termination condition is used to determine whether the training process of the streaming video understanding model can end. In specific implementation, the preset iteration termination condition can be one or more of the following conditions combined: the number of training iterations reaches a preset upper limit; the loss value describing the generation loss is less than a preset loss threshold; the loss value describing the generation loss no longer decreases significantly in consecutive iterations; the total training time reaches a preset upper limit.

[0031] Step 110: Obtain training sample data of the streaming video; the training sample data includes multiple semantic segments divided according to semantic consistency, each semantic segment corresponds to a continuous video frame sequence and a tag description text aligned with the continuous video frame sequence.

[0032] Specifically, firstly, training sample data of streaming video can be obtained, which includes multiple semantic segments divided according to semantic consistency, each semantic segment corresponding to a continuous video frame sequence and label description text aligned with the continuous video frame sequence.

[0033] Streaming video refers to video data where video frames arrive frame by frame over time, and only historical content is visible at any given moment. In practical applications, streaming video can be live video streams, such as live sports broadcasts, e-commerce live streams, and online classroom live streams; it can also be replay video streams played back in a streaming manner, such as online surveillance video playback and long-duration film and television content playback. Unlike pre-recorded offline videos that are visible in their entirety, streaming video arrives continuously and its duration may be unbounded. The model can only access previously played historical segments at any given moment, and access to any future frames is strictly prohibited. During the training phase, to ensure that the trained streaming video understanding model can adapt to the above streaming constraints, this invention uses streaming video as the source of training sample data and organizes the training sample data in a way that can simulate streaming input.

[0034] Here, training sample data refers to the samples input to the initial streaming video understanding model in one iteration of training, which includes multiple semantic segments divided according to semantic consistency. A semantic segment is a continuous piece of video content in a streaming video that maintains consistent semantic content; that is, each video frame within the same semantic segment describes the same event or the same semantic content. For example, in a streaming video of a kitchen scene, each video frame in the first semantic segment corresponds to the event "the chef is chopping onions," and each video frame in the second semantic segment corresponds to the event "the chef pours the chopped onions into the pan and stir-fries them." The boundary between the two semantic segments is the moment when the video's semantic content changes.

[0035] As an optional embodiment, semantic consistency partitioning can be achieved based on the semantic similarity between descriptive texts corresponding to consecutive video content. For example, the semantic similarity between descriptive texts corresponding to adjacent video content can be calculated. When the semantic similarity is lower than an adaptive threshold, it is determined that a semantic change has occurred, thereby partitioning a new semantic segment at that location; all video frames within the same semantic segment share the same event description. In addition, semantic consistency partitioning can also be achieved using other methods such as manual annotation partitioning or partitioning based on shot transition detection, and this embodiment of the invention does not limit the specific methods used.

[0036] Here, the continuous video frame sequence corresponding to each semantic segment refers to the ordered set of video frames obtained by continuously sampling the semantic segment in the time dimension. In specific implementation, the streaming video can be sampled uniformly in time at a fixed frequency to obtain a continuous video frame sequence. The fixed frequency can be, for example, 1 frame / second, thus obtaining a temporally continuous and uniformly distributed video frame sequence. If necessary, a maximum frame count limit can also be set to balance computational cost and context coverage. By using uniform temporal sampling, the continuity of visual context can be guaranteed.

[0037] Each semantic segment also corresponds to a labeled descriptive text aligned with a sequence of consecutive video frames. The labeled descriptive text refers to the actual semantic description corresponding to the sequence of consecutive video frames, i.e., the supervision signal during training. Alignment here means that the labeled descriptive text and the sequence of consecutive video frames correspond to each other in both the temporal and semantic dimensions; that is, the video frames within the same semantic segment match the text describing the event content of that semantic segment. For example, for the first semantic segment mentioned above, its corresponding labeled descriptive text could be "The chef is chopping onions."

[0038] As an optional implementation, to improve the diversity and robustness of the training sample data, the label description text corresponding to each video frame can be randomly sampled from multiple predefined pools of paraphrased variant captions. The paraphrased variant caption pool refers to a pre-constructed set of description texts (paraphrases) with the same semantics but different expressions for the same event content, denoted by M. For example, for the event "a chef is chopping onions," the paraphrased variant caption pool could also include paraphrases such as "a chef is chopping onions on a cutting board." In a preferred implementation, M is set to 1 to ensure optimal time alignment between the label description text and the video frame.

[0039] Step 120: The video frame features of each video frame in the continuous video frame sequence and the lexical features of the tag description text are cross-arranged in chronological order to obtain the input sequence.

[0040] Specifically, the video frame features of each video frame in a continuous video frame sequence can be interleaved with the lexical features of the label description text in chronological order to obtain the input sequence.

[0041] In this context, video frame features refer to the feature representation of visual information within a video frame. In practice, video frame features can be obtained by extracting features from each video frame using a visual encoder. The visual encoder can employ models such as the Vision Transformer (ViT), convolutional neural networks, or other encoding structures capable of mapping images into feature vectors or sequences of feature tokens. Each video frame, after processing by the visual encoder, can be represented as one or more video frame features in the form of visual tokens, thus enabling unified processing within the same sequence as text-based features.

[0042] Here, token features refer to the feature representation of the semantic information of each token in the tag description text. A token is a basic unit of text obtained after segmenting the tag description text; tokens can be at the granularity of characters, words, sub-words, etc. In practice, the tag description text can first be segmented to obtain a token sequence, and then each token can be mapped to a corresponding feature vector to obtain token features.

[0043] Here, the input sequence refers to the serialized data used as input to the initial streaming video understanding model, obtained by cross-arranging video frame features and lexical features in chronological order. Cross-arranging in chronological order means alternating the video frame features of each video frame with the lexical features of its corresponding tag description text, according to the arrival time order of the video frames in the streaming video. For example, if a continuous video frame sequence includes video frames F1, F2, and F3 arriving in chronological order, and the lexical features of the tag description text corresponding to each video frame are C1, C2, and C3, then the input sequence obtained after cross-arranging can be represented as [video frame features of F1, C1, video frame features of F2, C2, video frame features of F3, C3]. It can be understood that when the training sample data includes multiple semantic segments, the cross-arranged subsequences corresponding to each semantic segment are sequentially concatenated according to the chronological order of the semantic segments to form a complete input sequence.

[0044] The purpose of constructing the input sequence by the above cross-arrangement is to simulate the input pattern of "video frames arriving one by one and descriptions being updated as visual content evolves" during the training phase of streaming inference. That is, each video frame is followed by its corresponding descriptive word, so that the sequence structure faced by the initial streaming video understanding model during training is consistent with the streaming data structure actually received during the inference phase, thereby eliminating the distribution difference between training and inference.

[0045] Step 130: Input the input sequence into the initial streaming video understanding model. Based on the input sequence, the initial streaming video understanding model determines the observable historical input corresponding to each current video frame in the continuous video frame sequence, and outputs the prediction description result based on the observable historical input corresponding to each current video frame.

[0046] Specifically, after obtaining the input sequence, the input sequence can be fed into the initial streaming video understanding model. Based on the input sequence, the initial streaming video understanding model determines the observable historical input corresponding to each current video frame in the continuous video frame sequence, and outputs the prediction description result based on the observable historical input corresponding to each current video frame.

[0047] Here, the initial streaming video understanding model refers to a streaming video understanding model that has not yet completed training and whose model parameters are yet to be updated. In specific implementations, the initial streaming video understanding model can adopt an autoregressive generative model based on the Transformer architecture, such as a video large language model or a large visual language model. It includes at least an input embedding layer for receiving an input sequence composed of video frame features and word features, a multi-layer attention module for performing attention calculations on the input sequence, such as stacked Transformer decoder layers, and an output layer for outputting prediction results word by word.

[0048] Here, the model parameters of the initial streaming video understanding model can be randomly initialized, or they can be initialized using the parameters of a pre-trained video large language model before executing the training process of this invention. This embodiment of the invention does not limit this.

[0049] Here, the current video frame refers to the video frame in the continuous video frame sequence that is used as the current processing object during training. Since the input sequence is arranged in chronological order and interleaved, each video frame in the continuous video frame sequence is used as the current video frame in turn during training, and the initial streaming video understanding model performs descriptive prediction for each current video frame.

[0050] The observable historical input reflects the set of historical input information that the initial streaming video understanding model is allowed to focus on when making descriptive predictions for the current video frame. This includes video frame features and lexical features in the input sequence that are temporally earlier than the current video frame and visible in the prediction process for the current video frame. It should be noted that in streaming video understanding scenarios, the model is strictly prohibited from accessing future frames at any given time; therefore, the observable historical input does not contain any input information that is temporally later than the current video frame. Furthermore, to force the model to reason based on visual evidence from the current video frame itself, rather than simply repeating historical descriptions, the observable historical input does not need to include all historical inputs. Instead, it can be determined by the model from the historical input according to preset visibility constraints. In specific implementations, determining the observable historical input corresponding to each current video frame can be achieved by applying an attention mask during the model's attention calculation. This means controlling which historical positions in the input sequence the query position corresponding to the current video frame can focus on. Alternatively, it can be achieved by truncating, filtering, or gating the input sequence; this embodiment of the invention does not limit this approach.

[0051] The predicted description result reflects the prediction information of the initial streaming video understanding model based on the observable historical input corresponding to each current video frame and the descriptive text output for each current video frame. In specific implementation, the initial streaming video understanding model can generate the descriptive word sequence corresponding to the current video frame word by word in an autoregressive manner based on the observable historical input. That is, when generating each word, the model predicts the next word based on the observable historical input and the generated words, and the generated word sequence constitutes the predicted description result. For example, for a certain current video frame in the first semantic segment mentioned above, the predicted description result output by the model can be the text "The chef is chopping vegetables" corresponding to the word sequence.

[0052] In this way, the training objective of the initial streaming video understanding model is redefined as modeling the probability of the current semantic description under the historical context of evolution. That is, the initial streaming video understanding model learns the conditional probability distribution of which semantic description the current video frame corresponds to given observable historical input. This is completely consistent with the task form required by the model in the streaming inference stage.

[0053] Step 140: Determine the description generation loss based on the predicted description result and the label description text.

[0054] Specifically, after obtaining the predicted description results, the description generation loss can be determined based on the predicted description results and the label description text.

[0055] The description generation loss reflects the degree of difference between the predicted description and the labeled text; essentially, it measures the deviation between the current predictive ability of the initial streaming video understanding model and the ground truth annotations. Understandably, the greater the difference between the predicted description and the labeled text, the larger the description generation loss, indicating that the current model parameters of the initial streaming video understanding model need adjustment. Conversely, the smaller the difference between the predicted description and the labeled text, the smaller the description generation loss, indicating that the predictive ability of the initial streaming video understanding model is closer to the training objective.

[0056] In practice, the description generation loss can be determined using the loss function commonly used in generative model training. For example, the cross-entropy loss between the probability distribution of the model output at each word position in the predicted description result and the real word at the corresponding word position in the label description text can be used as the description generation loss. Alternatively, the negative log-likelihood loss can be used, which is to determine the description generation loss based on the likelihood of the label description text under the conditional probability distribution of the model output. Other auxiliary loss terms can also be superimposed on the above losses.

[0057] Step 150: Update the model parameters of the initial streaming video understanding model based on the loss generated by the description.

[0058] Specifically, in implementation, the backpropagation algorithm can be used. Starting with the description of the generation loss, it propagates backward along the computational graph of the initial streaming video understanding model, using the chain rule to calculate the gradient of the description of the generation loss relative to each trainable parameter in the initial streaming video understanding model. Then, the optimizer adjusts the model parameters according to the gradient. The optimizer can use optimization algorithms such as stochastic gradient descent, Adam, and AdamW, and perform parameter updates in conjunction with preset hyperparameters such as learning rate and learning rate scheduling strategy.

[0059] After updating the model parameters, return to step 110 to obtain a new batch of training sample data from the training data. Repeat steps 110 to 150 iteratively. In each iteration, the predictive ability of the initial streaming video understanding model will be improved until a preset iteration termination condition is met. At this point, the model with the last parameter update is determined as the successfully trained streaming video understanding model. The successfully trained streaming video understanding model can be deployed in online understanding scenarios for long-duration streaming videos, such as real-time live narration, online monitoring event description, and real-time sports commentary. It is used to understand and output descriptions of streaming videos as each video frame arrives.

[0060] The method provided in this invention cross-arranges video frame features and lexical features in chronological order to obtain an input sequence. This simulates the actual input pattern of video data arriving frame by frame in an online scenario, solving the problem of significant data distribution differences between training methods based on pre-recorded complete video sequences or static image-text alignment and the actual input pattern. This enables the model to accurately grasp the dynamic changes of video content. Furthermore, based on the observable historical input corresponding to the current video frame determined by the initial streaming video understanding model, a predicted description result is output. Thus, under the streaming constraint of strictly prohibiting access to future video frames, a probabilistic mapping relationship between visual evidence at the current moment and historical semantic description is effectively established. The temporal logic in the streaming process is robustly modeled, avoiding redundant repeated outputs for each incoming frame due to the lack of proactive and reliable time decision-making basis in actual online operation, or response delays caused by target misalignment, ensuring accurate and timely streaming video understanding results.

[0061] Based on the above embodiments, step 140 includes: Step 1401: For each word in the tag description text, based on the observable historical input and the generated words in the tag description text that precede each word, determine the conditional probability of each word in the predicted description result; Step 1402: Determine the joint probability based on the conditional probabilities of all lexical units in the predicted description result; Step 1403: Determine the description generation loss based on the joint probability.

[0062] Specifically, firstly, for each word in the label description text, based on observable historical input and the generated words preceding each word in the label description text, the conditional probability of each word in the predicted description result can be determined.

[0063] Understandably, the initial streaming video understanding model outputs the predicted description results word by word in an autoregressive manner. That is, when the initial streaming video understanding model outputs the nth word in the predicted description result, its input conditions include two parts: one part is the observable historical input corresponding to the current video frame, and the other part is the word in the label description text that is located before the nth word and is input into the model as a generated word.

[0064] In this context, "generated lexical units" specifically refers to the actual lexical sequence at the corresponding position in the label description text during the training phase, rather than the lexical sequence sampled by the model in the predicted description result. For example, when the lexical sequence corresponding to the label description text is "chef / currently / cutting / onion", and the lexical unit to be predicted is "cutting", the generated lexical units that the initial streaming video understanding model uses when generating the lexical unit at that position are "chef" and "currently" in the label text.

[0065] The conditional probability of each word corresponding to the tag description text in the predicted description result refers to the probability that the initial streaming video understanding model outputs that word given the observable historical input and the words preceding that word in the tag description text. In practice, the output layer of the initial streaming video understanding model can output a probability distribution covering the entire vocabulary at each word position. The probability value of the position of the actual word corresponding to the tag description text in this probability distribution is determined as the conditional probability of that word.

[0066] Secondly, the joint probability can be determined based on the conditional probabilities of all lexical units in the predicted description result. The joint probability refers to the probability that the predicted description result, as a whole, is generated by the initial streaming video understanding model given observable historical input. In practice, according to the chain rule of the autoregressive generation process, the joint probability can be obtained by multiplying the conditional probabilities of all lexical units in the predicted description result.

[0067] Finally, the descriptive generation loss can be determined based on the joint probability. In practice, the negative log-likelihood of the joint probability can be used as the descriptive generation loss.

[0068] Understandably, a higher joint probability indicates a higher likelihood of the labeled description text under the probability distribution of the model output, meaning a smaller difference between the predicted description and the labeled description text, and thus a smaller loss value for description generation loss. Conversely, a lower joint probability indicates a lower likelihood of the labeled description text under the probability distribution of the model output, meaning a greater difference between the predicted description and the labeled description text, and thus a greater loss value for description generation loss.

[0069] The method provided in this invention determines the conditional probability of each word based on observable historical input and previously generated words, then determines the joint probability based on the conditional probabilities of all words and uses this to determine the description generation loss. This makes the training objective explicitly set as maximizing the joint probability of the current semantic description under the conditions of evolutionary historical context. This is strictly consistent with the task form of the streaming inference stage model, which evaluates the confidence of the description text based on historical context. As a result, the trained streaming video understanding model can output a more reliable and robust probability distribution.

[0070] Based on the above embodiments, step 130 includes: Step 1301: In the attention calculation of the initial streaming video understanding model, within the same semantic segment, the attention connection between the video frame features of the current video frame and the word features corresponding to the current video frame as queries and the video frame features of the observed frames and the word features corresponding to the observed frames as keys is blocked; the observed frames are video frames preceding the current video frame in the corresponding continuous video frame sequence within the same semantic segment. Step 1302: Between different semantic segments, retain the attention connection between the video frame features of the current video frame and the lexical features corresponding to the current video frame as the query, and the lexical features corresponding to the terminating descriptive lexicals of each preceding semantic segment that is earlier in time than the semantic segment to which the current video frame belongs as the key; the terminating descriptive lexical refers to the lexical at the end position in the tag description text corresponding to the preceding semantic segment. Step 1303: Determine the video frame features and word features corresponding to the historical positions in the input sequence where attention connections were not blocked as the observable historical input of the current video frame; Step 1304: Perform autoregressive inference based on the observable historical input to obtain the prediction description result.

[0071] Specifically, firstly, the initial streaming video understanding model can, within the same semantic segment, block the attention connection between the video frame features and the corresponding word features of the current video frame as queries and the video frame features and the corresponding word features of the observed frames as keys, during the attention calculation.

[0072] Attention computation refers to the process performed by the multi-layer attention modules in the initial streaming video understanding model to determine attention weights based on the similarity between queries and keys, and then weighted aggregate values ​​based on these attention weights. Here, attention connectivity refers to the attention weight calculation path between query positions and key positions in the input sequence. If the attention connectivity between a query position and a key position is blocked, the query position cannot focus on the feature information corresponding to the key position during attention computation.

[0073] Here, observed frames refer to video frames within the same semantic segment that precede the current video frame in time sequence; that is, video frames that have arrived and been processed by the initial streaming video understanding model within the semantic segment to which the current video frame belongs. It should be noted that since all video frames within the same semantic segment share the same event description, the lexical features corresponding to observed frames and the lexical features corresponding to the current video frame are semantically synonymous. If queries corresponding to the current video frame are allowed to focus on the lexical features corresponding to observed frames, the initial streaming video understanding model can directly reuse the synonymous expressions of previous frames when predicting the description of the current video frame, without relying on the visual evidence of the current video frame itself.

[0074] In practice, blocking can be achieved by overlaying a streaming causal attention mask on the standard causal attention of the initial streaming video understanding model's autoregressive training. This means forcibly masking the attention weights of descriptive lemmas generated in previous frames within the same semantic segment. For example, the attention score corresponding to the blocked attention connection can be set to negative infinity or set to the mask value at the corresponding position in the mask matrix, so that the attention weights after being normalized by the Softmax function approach zero, thereby achieving selective visibility constraint.

[0075] Secondly, between different semantic segments, attention connections are maintained between the video frame features and the lexical features corresponding to the current video frame as queries, and the lexical features corresponding to the terminating descriptive lexical features of each semantic segment that is temporally earlier than the semantic segment to which the current video frame belongs as keys.

[0076] Among them, the preceding semantic segment refers to the semantic segment in the training sample data that precedes the semantic segment to which the current video frame belongs in time.

[0077] Here, the terminology for termination description refers to the terminology at the end of the tag description text corresponding to the preceding semantic segment. Since all video frames within the same semantic segment share the same event description, the terminology at the end can serve as a summary representation of the event content corresponding to that semantic segment. The lexical features corresponding to the terminology for termination description are the lexical features obtained after feature representation of the terminology for termination description.

[0078] Understandably, unlike the blocking operation within the same semantic segment, this step preserves full visibility of the terminating descriptive terms of all preceding semantic segments between different semantic segments. The purpose is to enable the model to still obtain summary information of previous events when predicting the description of the current video frame, thereby maintaining the coherence of the narrative and avoiding the loss of long-term contextual coherence due to excessive blocking of historical information.

[0079] Furthermore, the video frame features and lexical features corresponding to the historical positions in the input sequence where attention connections were not blocked can be identified as the observable historical input of the current video frame.

[0080] The historical position reflects the sequence location of video frame features and lexical features that precede the current video frame in the input sequence. After the blocking and preservation operations described above, the historical positions of the attention connections in the input sequence that are not blocked specifically include: the positions of video frame features of observed frames within the semantic segment to which the current video frame belongs, i.e., visual features are still visible, only synonymous descriptive lexical units within the same semantic segment are blocked, and the positions of lexical features corresponding to the terminating descriptive lexical units of each preceding semantic segment. The set of video frame features and lexical features corresponding to these unblocked historical positions constitutes the observable historical input of the current video frame.

[0081] Finally, autoregressive inference can be performed based on observable historical input to obtain the predicted description. In this setting, the training objective of the initial streaming video understanding model under the constraint of streaming causal attention masks can be expressed as: ; in, This represents the k-th semantic segment in the training sample data; Represents the k-th semantic segment The current moment corresponding to the current video frame. This indicates that the training objective is related to semantic segments. The current video frame at every moment is valid; This represents the tag description text corresponding to the k-th semantic fragment. Each word element is a descriptive word element corresponding to the current video frame; Indicates time The video frame features of the current video frame; Represents the time in the input sequence Previous historical input; Indicates the application applied at the deadline. The streaming causal attention mask on the historical input achieves the aforementioned selective visibility constraint, namely, masking the attention weights of descriptive terms generated in previous frames within the same semantic segment, while preserving full visibility of terminating descriptive terms of all preceding semantic segments. Constrained Together, they constitute the observable historical input corresponding to the current video frame.

[0082] The method provided in this invention, by blocking the attention connection between the descriptive lexical units of the current video frame and the observed frame within the same semantic segment during attention calculation, and retaining the attention connection to the terminating descriptive lexical units of each preceding semantic segment between different semantic segments, prevents the model from reusing synonymous expressions of previous frames within the same semantic segment when predicting the description of the current video frame. Instead, it can only rely on the visual features of the current video frame and the summary information of each previous semantic segment for reasoning. In other words, the model is forced to reason based on visual evidence rather than relying on language repetition, thereby providing a more reliable and robust probability distribution for confidence assessment in the streaming reasoning stage, while also taking into account the coherence of long-term narratives.

[0083] Based on the above embodiments, the step of obtaining the lexical features of the tag description text includes: Step 21: According to the preset word segmentation rules, the tag description text is segmented to obtain a word sequence; Step 22: Based on the pre-constructed vocabulary, map each word in the word sequence to a word index; Step 23: Perform vectorized mapping on each of the lexical indices to obtain vector mapping results, and superimpose the positional codes of each lexical in the lexical sequence on the vector mapping results to obtain the lexical features of the tag description text.

[0084] Specifically, firstly, the tag description text can be segmented according to preset word segmentation rules to obtain a word sequence.

[0085] The preset word segmentation rules refer to the rules governing the granularity and method of dividing the tag description text into word units. In practice, the preset word segmentation rules can employ sub-word level rules such as Byte Pair Encoding (BPE), or character-level or word-level word segmentation rules. Preferably, using sub-word level rules can break down unseen words during training into known, smaller sub-word units, thus preserving the semantic information of the text to the greatest extent while controlling the size of the vocabulary.

[0086] Here, a lexical sequence refers to an ordered set of lexical units arranged in the text order after the tag description text is segmented. For example, the lexical sequence obtained after segmenting the tag description text "The chef is cutting onions" can be [chef, is, cutting, onions].

[0087] Then, based on the pre-built vocabulary, each word in the word sequence is mapped to a word index. The pre-built vocabulary reflects the one-to-one mapping relationship between words and word indices, and it can be statistically constructed based on a large-scale corpus according to preset word segmentation rules before model training.

[0088] Here, the lexical index refers to the unique numerical number of each lexical in a pre-built vocabulary. By looking up the vocabulary, the lexical sequence is converted into an integer sequence. For example, the above lexical sequence can be mapped to [1058, 327, 4291, 7763], thereby converting the text into a numerical format that the model can directly process.

[0089] Finally, vectorization mapping can be performed on each lexical index to obtain the vector mapping result. The positional encoding of each lexical in the lexical sequence can then be superimposed on the vector mapping result to obtain the lexical features of the tag description text.

[0090] Vectorization mapping refers to the process of mapping discrete lexical indices into continuous high-dimensional dense vectors by performing a lookup operation based on a learnable embedding matrix and using lexical indices as row indices; the vector mapping result refers to the set of dense vectors obtained after vectorization mapping of each lexical index.

[0091] In practical implementation, the positional encoding of each word in the word sequence can adopt a fixed sine-cosine positional encoding, a learnable positional encoding, or a rotated positional encoding, etc. The superposition of positional encoding can be achieved by adding the positional encoding of each word in the word sequence to the vector mapping result element by element, or by fusing positional information in attention calculation. This embodiment of the invention does not limit the specific method used.

[0092] The method provided in this invention obtains the lexical features of the tag description text through word segmentation, vocabulary index mapping, vectorization mapping, and sequence position encoding injection. This ensures that the lexical features carry both the semantic information of each lexical and the positional information of each lexical in the temporal order, thereby guaranteeing that the lexical features and video frame features can be processed uniformly and orderly in the cross-arranged input sequence.

[0093] Considering that after training a streaming video understanding model, it can be applied to perform streaming video understanding.

[0094] Based on any of the above embodiments, the present invention provides a streaming video understanding method. Figure 2 This is a flowchart illustrating the streaming video understanding method provided by the present invention, as shown below. Figure 2 As shown, the method includes: Step 210: Obtain the streaming video data to be understood; the streaming video data to be understood includes the streaming video frame sequence of the target to be understood, and the task prompt words of the streaming video frame sequence; Step 220: Input the streaming video frame sequence and the task prompt words into the streaming video understanding model to obtain the description prediction result output by the streaming video understanding model; the description prediction result is obtained by the streaming video understanding model after understanding the streaming video frame sequence based on the task type specified by the task prompt words and guided by the task prompt words. The streaming video understanding model is obtained by performing the training method described above.

[0095] Specifically, firstly, the streaming video data to be understood can be acquired, which includes the streaming video frame sequence of the target to be understood, and the task prompt words of the streaming video frame sequence.

[0096] Here, "streaming video data to be understood" refers to the relevant data of streaming video that needs to be understood during the inference phase. "Target to be understood" refers to the video object corresponding to the streaming video data to be understood, such as a live video stream that is currently playing (e.g., a sports event stream, an e-commerce live stream), an online monitoring video stream, or a replay video stream that is being streamed back to its original format.

[0097] Here, a streaming video frame sequence refers to an ordered set of video frames obtained by sampling historical segments that have been played at the current moment for the target to be understood. In specific implementations, only the historical segments that have already been played can be extracted as visible input at any given moment, strictly prohibiting access to any future frames. The historical segments are then sampled uniformly over time at a fixed frequency, such as 1 frame / second, to obtain the streaming video frame sequence. If necessary, a maximum frame count can be set to balance computational cost and context coverage.

[0098] Among them, task prompts are used to reflect structured prompts for the guided streaming video understanding model to perform specific understanding tasks. The task prompts are constructed according to the task type and are used to specify the output structure of the model, such as whether a timestamp needs to be output or whether a bounding box needs to be given.

[0099] The task types can include at least one of real-time narrative generation, online time localization, frame-level intensive question answering, context-based online question answering, and multi-turn interactive question answering. For example, for the real-time narrative generation task, the task prompt could be "Please provide a real-time narrative of the current video stream content"; for the online time localization task, the task prompt could be "Please indicate the time when the target event occurred and output it in timestamp format." Simultaneously with acquiring the streaming video data to be understood, the memory buffer can be initialized and the upper limit of the context length can be set to complete context initialization.

[0100] Next, the streaming video frame sequence and task prompts are input into the streaming video understanding model to obtain the descriptive prediction results output by the model. These descriptive prediction results are obtained by the streaming video understanding model based on the task type specified by the task prompts, guided by the prompts.

[0101] The streaming video understanding model is obtained by performing the training method of the streaming video understanding model as described in any of the above embodiments. The specific training process can be found in the descriptions of the above embodiments, and will not be repeated here.

[0102] Here, the description prediction results are used to reflect the understanding results output by the streaming video understanding model for the streaming video frame sequence according to the task type specified by the task prompt words. For example, for the real-time narrative generation task, the description prediction results are the narrative text that is updated as the video content evolves; for the online time localization task, the description prediction results are the localization text containing timestamps.

[0103] As an optional embodiment, a streaming key-value caching mechanism can also be used in the above process to avoid repeated calculation of historical representations due to frequent context appending and logical exchange operations. The streaming key-value caching mechanism includes two levels: in-dialogue caching and inter-dialogue streaming caching. In-dialogue caching serves frame-level processing, while inter-dialogue streaming caching serves long-term contexts across dialogues.

[0104] As an optional embodiment, when outputting the prediction results, if the task type specified by the task prompt requires temporal or spatial evidence, the timestamp of the corresponding supporting frame can be extracted, and optionally, a spatial bounding box can be generated on the corresponding frame using the target localization model. Subsequently, the answer, timestamp, and bounding box coordinates are formatted into structured labels for output, and the structured labels at least include the answer label. <answer>With timestamp tags <time>Optionally, it may also include bounding box labels. <bbox>For example, formatted as <answer> ...< / answer> <time>...< / time> <bbox> ...< / bbox> The structured results are then transmitted to the front-end player, Q&A panel, or downstream automatic editing system.

[0105] The method provided in this invention, by inputting streaming video frame sequences and task prompts into a streaming video understanding model obtained based on the above training method, enables the model to flexibly adapt to various online video understanding tasks under the guidance of task prompts without retraining for each task. Simultaneously, because the streaming video understanding model establishes a reliable probability estimate of the current semantic description under the context of evolutionary history during the training phase, the model transforms the silence decision from a generation target to a confidence-based verification state during the inference phase. It can autonomously decide the response timing with only a single forward propagation, completely bypassing the data imbalance, time inconsistency, target misalignment, and vocabulary degradation problems of traditional solutions. This significantly reduces the computational latency of streaming inference, achieving accurate and timely streaming video understanding.

[0106] Based on the above embodiments, the method further includes a hierarchical memory management step, which includes: Step 31: When the total number of lexical units in the context window of the streaming video understanding model reaches the preset length limit, the semantic segment to which the context window belongs is determined to be the target semantic segment. Step 32: Determine the verification perplexity corresponding to each video frame within the target semantic segment, and based on the verification perplexity, determine the semantic divergence of the current video frame in the streaming video frame sequence; Step 33: Determine the median of the semantic divergence of each video frame in the target semantic segment, and determine the video frames in the target semantic segment with a semantic divergence lower than the median as frames to be compressed. Step 34: Remove the video frame features and the lexical features corresponding to the frame to be compressed from the context window, and retain only the end description lexical features of the target semantic segment to obtain the compressed target semantic segment; the end description lexical features are the lexical features corresponding to the end description lexical, and the end description lexical is the lexical located at the end position of the tag description text in the target semantic segment.

[0107] Specifically, firstly, when the total number of lexical units in the context window of the streaming video understanding model reaches the preset length limit, the semantic segment to which the context window belongs is determined to be the target semantic segment.

[0108] The context window refers to the input feature buffer that the streaming video understanding model can focus on during the inference phase, i.e., short-term working memory, which contains the video frame features of the video frames that have arrived and the lexical features of the historical descriptive text.

[0109] Here, the total number of lexical units refers to the sum of the number of visual lexical units corresponding to the video frame features and the number of textual lexical units corresponding to the descriptive text within the context window. (Preset length limit) The maximum value used to reflect the total number of tokens that the context window can hold is, in the preferred implementation, The value is set to 8192 tokens. During inference, the total number of tokens in the context window is monitored in real time, and a compression loop is triggered when the total number of tokens reaches the preset length limit.

[0110] The target semantic fragment is used to reflect the semantic fragment that needs to be compressed within the context window when the compression loop is triggered.

[0111] Secondly, the verification perplexity of each video frame within the target semantic segment can be determined, and based on the verification perplexity, the semantic divergence of the current video frame in the streaming video frame sequence can be determined.

[0112] The verification perplexity is used to reflect the perplexity of the existing descriptive text under the updated historical context conditions at the arrival time of the corresponding video frame, and the formula is as follows: ; in, This indicates that there is already a description text. At the present moment Verification perplexity; This indicates that there is already descriptive text in the historical context after the current update. The probability value under the given conditions, i.e., the joint probability; The perplexity is the length of the existing descriptive text. In other words, the validation perplexity is obtained by taking the Nth root of the Nth root of the probability value of the existing descriptive text under the current updated historical context.

[0113] Subsequently, the verification perplexity is compared with the reference perplexity, and the activation condition for the gating decision is: ; in, This indicates the reference time, which is the time when the existing descriptive text was last generated; This indicates the degree of confusion in reference, which is the degree of confusion of the existing descriptive text under the historical context conditions at the reference time. >1 is an adjustable sensitivity factor used to control the sensitivity of the response activation. In the preferred implementation, The value range is from 1.02 to 1.04, and more preferably 1.03. Experimental results show that... The optimal balance is achieved in the range of 1.02 to 1.04. The overall performance is best when the value is 1.03.

[0114] When the above activation conditions are met, it indicates that there has been a significant semantic discrepancy between the current visual input and the existing descriptive text. In this case, the decoding gate is activated, and the streaming video understanding model performs a complete autoregressive generation to output a new descriptive prediction result and updates the reference perplexity accordingly. Conversely, if the verification perplexity remains stable, generation is suppressed to avoid redundant output. At this time, in order to ensure that the context window reflects the continuous passage of time, a logical swap operation is performed to move the existing descriptive text to the end of the context buffer, so that its temporal validity extends to the current moment.

[0115] Here, the semantic divergence of the current video frame reflects the degree of semantic deviation between the visual input of the current video frame and the existing descriptive text. In practice, the verification perplexity can be directly determined as the semantic divergence, i.e., for the target semantic segment... Inner Time t The semantic divergence of the video frames is as follows: ; in, Indicates time t The degree of semantic divergence, This indicates that there is already a description text. At any moment t Verification perplexity.

[0116] Understandably, the higher the semantic divergence, the more significant the semantic change in the visual content of the video frame relative to the existing descriptive text, the greater the amount of information carried by the video frame, and the more likely it is to be a significant peak frame within the target semantic segment; conversely, the lower the semantic divergence, the more consistent the semantics of the video frame with the existing descriptive text, and the higher its information redundancy.

[0117] Furthermore, the median of the semantic divergence of each video frame within the target semantic segment can be determined, and video frames with a semantic divergence below the median within the target semantic segment are identified as frames to be compressed. The median of the semantic divergence... Used to reflect target semantic fragments The semantic divergence score of all video frames is the middle value after being sorted numerically.

[0118] Here, the frames to be compressed reflect video frames with high information redundancy within the target semantic segment that need to be removed from the context window. Based on the median, the subset of frames retained within the target semantic segment can be represented as: ; in, Represents the k-th target semantic segment The set of times corresponding to the video frames that are retained within the content.

[0119] Since the median divides the video frames within a segment into two roughly equal parts based on semantic divergence, this operation can remove about 50% of the video frames within the target semantic segment, while significant peak frames with semantic divergence no less than the median are retained.

[0120] Finally, the video frame features and corresponding lexical features of the frame to be compressed can be removed from the context window, retaining only the end-description lexical features of the target semantic fragment, thus obtaining the compressed target semantic fragment. Here, the end-description lexical features are the lexical features corresponding to the end-description lexical, which is the lexical at the end of the tag description text in the target semantic fragment.

[0121] Among them, the lexical features corresponding to the frame to be compressed refer to the feature representation of the descriptive text lexical units corresponding to the frame to be compressed in the input sequence.

[0122] Here, the end position of the tag description text refers to the end position of the description text corresponding to the target semantic fragment. The lexical unit located at this end position is the end description lexical unit, which can serve as a summary description of the event content corresponding to the target semantic fragment. The end description lexical unit feature is the lexical feature corresponding to the end description lexical unit, which is permanently preserved during the compression process.

[0123] The compressed target semantic segment is used to reflect the semantic segment obtained after performing the above-mentioned peak-end rule-based compression operation. It is compressed into a compact form of "significant peak frame plus summary description", which is composed of video frame features of the retained high semantic divergence video frame and end description lexical features.

[0124] The method provided in this invention determines the semantic divergence of each video frame based on the verification perplexity when the total number of lexical units in the context window reaches a preset length limit. It then removes redundant frames to be compressed, retaining only significant peak frames and termination descriptive lexical features, using the median of the semantic divergence as the boundary. This achieves on-demand compression of the context window, ensuring that the streaming video understanding model maintains bounded computational costs when facing infinitely long sequences of streaming video frames. At the same time, it maximizes the preservation of the most informative visual evidence and summary semantics in each semantic segment, providing a guarantee for the continuous operation of long-term streaming video understanding.

[0125] Based on the above embodiments, the method further includes: Step 41: Take the remaining video frames in the target semantic segment, excluding the frame to be compressed, and the lexical features corresponding to the remaining video frames as historical semantic units, and insert the historical semantic units into the recursive event tree; Step 42: Determine the highest semantic similarity based on the semantic similarity between the feature vector of the historical semantic unit and the feature vector of the existing tree node in the recursive event tree; Step 43: If the highest semantic similarity exceeds a preset semantic similarity threshold, the historical semantic unit is attached as a child node to the existing tree node corresponding to the highest semantic similarity; otherwise, a new root node is created to store the historical semantic unit. Step 44: Update the semantic embedding of the parent node corresponding to the child node.

[0126] Specifically, firstly, the remaining video frames in the target semantic segment, excluding the frame to be compressed, and the corresponding lexical features of the remaining video frames can be used as historical semantic units, and these historical semantic units can be inserted into the recursive event tree.

[0127] Among them, the remaining video frames refer to the video frames retained after compression within the target semantic segment, that is, significant peak frames with semantic divergence not lower than the median.

[0128] Here, a historical semantic unit refers to a memory unit that needs to be archived and stored after leaving the active region of the context window, and is composed of the remaining video frames and their corresponding lexical features. A recursive event tree is a long-term retrieval memory structure that organizes historical semantic units in a tree-like structure. Each tree node stores one or more historical semantic units and their semantic embeddings. Historical semantic units with similar semantics are recursively aggregated in the tree in a parent-child relationship, thereby organizing the expelled history into a retrievalable structured memory.

[0129] Then, the highest semantic similarity can be determined based on the semantic similarity between the feature vector of the historical semantic unit and the feature vector of the existing tree node in the recursive event tree.

[0130] Among them, the feature vector of a historical semantic unit refers to the vectorized representation of the overall semantic content of the historical semantic unit. In practice, this can be achieved by pooling, averaging, or other aggregation methods on the video frame features and word features in the historical semantic units.

[0131] Among them, an existing tree node refers to a tree node that already exists in the recursive event tree before the current historical semantic unit is inserted.

[0132] Here, semantic similarity reflects the degree of semantic closeness between the feature vector of a historical semantic unit and the feature vector of an existing tree node. In practice, cosine similarity can be used for calculation. The highest semantic similarity refers to the maximum value among the semantic similarities between the feature vector of the historical semantic unit and the feature vectors of all existing tree nodes in the recursive event tree.

[0133] Furthermore, if the highest semantic similarity exceeds a preset semantic similarity threshold, the historical semantic unit is attached as a child node to the existing tree node corresponding to the highest semantic similarity; otherwise, a new root node is created to store the historical semantic unit.

[0134] Among them, the preset semantic similarity threshold The decision boundary used to determine whether a historical semantic unit is semantically similar to an existing tree node, and thus to decide whether to attach or create a new one, is used in the preferred implementation. The value is 0.75. When the highest semantic similarity exceeds... If the historical semantic unit is found to be semantically similar to the corresponding existing tree node, it is attached as a child node to the existing tree node; otherwise, it indicates that the historical semantic unit corresponds to newly emerging event content, and a new root node is created to store the historical semantic unit.

[0135] Finally, update the semantic embedding of the parent node corresponding to the child node. The semantic embedding of the parent node refers to the vectorized representation of the semantic content stored in the existing tree node after the attachment operation. Since attaching a new child node changes the semantic scope represented by the parent node, the semantic embedding of the parent node needs to be updated, causing the parent node's semantic embedding to evolve towards the semantic center of all its descendant nodes.

[0136] As an optional implementation, when the aforementioned decoding gate is activated, a query vector can be constructed based on the current task, and a hierarchical bundle search can be performed on the recursive event tree to retrieve the historical event chain: starting from the root node, each layer retains only the top-k nodes with the highest cosine similarity to the query vector as the leading edge of the next layer, proceeding downwards layer by layer until the leaf node. Since only child nodes along the top-k root-to-leaf paths are evaluated, the expected retrieval cost of the hierarchical bundle search is... ,in, k The number of nodes to retain at each level. b For the branching factor of the recursive event tree, n Let be the total number of storage units in the recursive event tree, meaning the retrieval cost grows sublinearly with respect to the total number of storage units. The retrieved historical events can be represented as: ; in, This represents the set of historical events retrieved. Indicates the first The lexical features of the historical descriptive text stored in each retrieved node; Indicates the first The video frame features (visual terms) stored in the retrieved node; I represents the index set of the nodes finally hit by the hierarchical beam search; Path(I) represents the index set of parent nodes on the retrieval path from the root node to each node in set I. The retrieved historical events, that is, the historical descriptive text of the parent nodes and immediate child nodes extracted along the retrieval path, and the video frame features are re-injected into the current generation context and jointly decoded with the currently generated description prediction results to ensure that the output statement contains both current screen information and historical context.

[0137] The method provided in this invention inserts the remaining compressed video frames and their corresponding lexical features as historical semantic units into a recursive event tree. Based on semantic similarity and a preset semantic similarity threshold, it determines whether to attach them as child nodes or create new root nodes. The historical information that has been expelled from the context window is organized into a semantically aggregated, searchable, structured long-term memory. Combined with hierarchical bundle search with sublinear retrieval costs, the streaming video understanding model can recall long-term historical information at any time under bounded context conditions, effectively alleviating the catastrophic forgetting problem caused by the sliding window strategy and significantly improving the memory capacity of long-term streaming video understanding.

[0138] Based on the above embodiments, step 44 includes: Step 441: Determine the existing tree node corresponding to the highest semantic similarity as the parent node to be updated; Step 442: Obtain the semantic embedding of the parent node to be updated, and the feature vector of the historical semantic unit attached as a child node; Step 443: Using the preset momentum update rate as the first weight of the feature vector of the historical semantic unit, determine the second weight of the semantic embedding of the parent node to be updated based on the momentum update rate. Step 444: Based on the first weight and the second weight, perform a weighted summation of the feature vector of the historical semantic unit and the semantic embedding of the parent node to be updated to obtain the updated parent node semantic embedding.

[0139] Specifically, firstly, the existing tree node corresponding to the highest semantic similarity can be identified as the parent node to be updated. Here, the parent node to be updated refers to the existing tree node that needs to be semantically embedded and updated due to the attachment of new child nodes, that is, the existing tree node to which the historical semantic unit is attached.

[0140] Secondly, obtain the semantic embedding of the parent node to be updated, as well as the feature vector of the historical semantic unit attached to the child node.

[0141] Among them, the semantic embedding of the parent node to be updated This refers to the vectorized representation of the semantic content stored in the parent node to be updated before the current attachment operation. It serves as the feature vector of the historical semantic unit for attaching to the child node. That is, the feature vector of the historical semantic unit in the above steps. At this point, the historical semantic unit has been attached as a child node to the parent node to be updated.

[0142] Secondly, the preset momentum update rate is used as the first weight of the feature vector of the historical semantic unit, and the second weight of the semantic embedding of the parent node to be updated is determined based on the momentum update rate.

[0143] Among them, the preset momentum update rate This refers to the control parameter for the speed at which the semantic embedding of a parent node evolves towards the semantic center of its descendant nodes. In the preferred implementation, The value is 0.3.

[0144] Here, the first weight is used to reflect the weight of the feature vector of the historical semantic unit in the weighted summation, which is the momentum update rate. The second weight reflects the weight of the semantic embedding of the parent node to be updated in the weighted summation. The second weight is determined based on the momentum update rate, i.e., 1 minus the momentum update rate, denoted as... .

[0145] Finally, based on the first and second weights, the feature vectors of historical semantic units and the semantic embeddings of the parent node to be updated can be weighted and summed to obtain the updated semantic embeddings of the parent node.

[0146] Specifically, the semantic embedding of the parent node to be updated is updated through momentum aggregation, that is: ; Among them, the one on the right This represents the semantic embedding of the parent node to be updated before the update. The feature vector representing the historical semantic unit attached as a child node; This represents the preset momentum update rate, i.e., the first weight; This indicates the second weight; Normalize(·) is a function that normalizes the weighted summation result, ensuring the updated parent node semantic embedding has a unit length to eliminate the impact of modulus differences on subsequent semantic similarity calculations; the left side of the arrow... This is the updated semantic embedding of the parent node.

[0147] Among them, the updated parent node semantic embedding refers to the vectorized representation of the semantic content stored in the parent node to be updated after incorporating the semantic information of the newly attached child nodes.

[0148] Understandably, the higher the momentum update rate, the faster the semantic embedding of the parent node evolves semantically towards the newly attached child node; conversely, the lower the momentum update rate, the more the semantic embedding of the parent node tends to maintain its original semantic center. By taking... The preferred value of 0.3 can achieve a balance between the stability of the semantic center and its adaptability to the semantics of new events.

[0149] The method provided in this invention uses a preset momentum update rate as the first weight and a factor minus the momentum update rate as the second weight to perform a weighted summation and normalization of the feature vector of the historical semantic unit and the semantic embedding of the parent node to be updated. This allows the semantic embedding of each parent node in the recursive event tree to smoothly evolve towards the semantic center of its descendant nodes as child nodes are continuously attached. This ensures that the semantic representation of each layer of nodes in the recursive event tree always accurately reflects the overall semantics of all historical semantic units under it, thereby improving the accuracy of subsequent attachment determination based on semantic similarity and hierarchical bundle search retrieval, and further ensuring the reliability of long-term historical information recall.

[0150] Based on any of the above embodiments Figure 3 This is a schematic diagram illustrating the application of the streaming video understanding method provided by this invention in real-time narration tasks, as shown below. Figure 3 As shown, for frame-by-frame input of borderless video streams, continuous processing can be performed based on the prompt: "Generate streaming narration in real time." This contrasts with existing methods that overly rely on [the specific method] during the training and inference phases. <eos>The labeling process leads to temporal inconsistencies and data imbalances. Furthermore, traditional Video-LLM suffers from numerous shortcomings by forcibly decoding and outputting narration when faced with the current frame. This method introduces a streaming causal attention masking mechanism during the training phase to ensure pre-training consistency. As shown in the training perspective, images and narration sequences are interleaved for learning. During the inference phase, this method performs a single forward propagation based on the input logical values ​​to autonomously decide whether to activate decoding and output narration or remain silent (no output). Simultaneously, a memory module manages historical events, for example... Figure 3 The search results for "kitten on the grass" and "kitten playing indoors" are dynamically retrieved and updated.

[0151] Figure 4 This is a schematic diagram of the memory management process provided by the present invention, such as... Figure 4 As shown, in the specific memory management process, when faced with a continuously arriving video stream, the system calculates the perplexity of each frame in the context, such as the specific values ​​shown in the diagram: 1.85, 2.46, 2.11, 1.73, 2.68, 2.02, 2.29, 2.51, 1.94, etc. Redundant information is filtered through compression operations, forming a compact memory that retains highly divergent frames and subtitles, which is then divided into memory units. When a new frame arrives and triggers the splitting and archiving of a memory unit, the system traverses downwards from the initial state in the tree structure. If the similarity between the current unit and an existing node is ≥ a threshold, an association is established; if the similarity is < a threshold, a new branch is created. The node features in the tree specifically include query embedding (Q), subtitle embedding (C), and frame marker / keyword (KV) data. (F); When generating a new response, a query is constructed based on the current frame, Top-K related historical nodes are retrieved in the recursive tree, and input into the video big language model. Finally, the video big language model outputs accurate current subtitles, thus forming a coherent result as shown in the complete narrative: "Initially, a gray kitten was running on the green grass. Soon after, another gray kitten entered the frame and approached the first one. Then the scene changed, and they are now indoors." The training apparatus for the streaming video understanding model provided by the present invention will be described below. The training apparatus for the streaming video understanding model described below can be referred to in correspondence with the training method for the streaming video understanding model described above.

[0152] Based on any of the above embodiments, the present invention provides a training apparatus for a streaming video understanding model. Figure 5 This is a schematic diagram of the structure of the training device for the streaming video understanding model provided by the present invention, as shown below. Figure 5 As shown, the device includes: The first acquisition unit 510 is used to acquire training sample data of streaming video; the training sample data includes multiple semantic segments divided according to semantic consistency, each semantic segment corresponds to a continuous video frame sequence and a tag description text aligned with the continuous video frame sequence; The arrangement unit 520 is used to cross-arrange the video frame features of each video frame in the continuous video frame sequence with the lexical features of the tag description text in chronological order to obtain the input sequence; The prediction unit 530 is used to input the input sequence into the initial streaming video understanding model, and the initial streaming video understanding model determines the observable historical input corresponding to each current video frame in the continuous video frame sequence based on the input sequence, and outputs the prediction description result based on the observable historical input corresponding to each current video frame. The loss determination unit 540 is used to determine the description generation loss based on the predicted description result and the label description text; The update unit 550 is used to update the model parameters of the initial streaming video understanding model based on the loss generated by the description.

[0153] The apparatus provided in this invention cross-arranges video frame features and lexical features in chronological order to obtain an input sequence, thereby simulating the actual input pattern of video data arriving frame by frame in an online scenario. This solves the problem of significant data distribution differences between training methods based on pre-recorded complete video sequences or static image-text alignment and the actual input pattern, enabling the model to accurately grasp the dynamic changes of video content. Furthermore, based on the observable historical input corresponding to the current video frame determined by the initial streaming video understanding model, a predicted description result is output. Thus, under the streaming constraint of strictly prohibiting access to future video frames, a probabilistic mapping relationship between visual evidence at the current moment and historical semantic description is effectively established. Robust modeling of the temporal logic in the streaming process is performed, avoiding redundant repeated outputs for each incoming frame due to the lack of proactive and reliable time decision-making basis in actual online operation, or response delays caused by target misalignment, ensuring accurate and timely streaming video understanding results.

[0154] Based on any of the above embodiments, the loss determination unit 540 is specifically used for: For each word in the tag description text, based on the observable historical input and the generated words in the tag description text preceding each word, the conditional probability of each word in the predicted description result is determined; The joint probability is determined based on the conditional probabilities of all lexical units in the predicted description results; Based on the joint probability, the description generation loss is determined.

[0155] Based on any of the above embodiments, the prediction unit 530 is specifically used for: In the attention calculation of the initial streaming video understanding model, within the same semantic segment, attention connections are blocked between the video frame features of the current video frame and the lexical features corresponding to the current video frame as queries, and the video frame features of the observed frames and the lexical features corresponding to the observed frames as keys; the observed frames are video frames preceding the current video frame in the corresponding continuous video frame sequence within the same semantic segment. Between different semantic segments, an attention connection is maintained between the video frame features of the current video frame and the lexical features corresponding to the current video frame as the query, and the lexical features corresponding to the terminating descriptive lexical features of each semantic segment that is temporally earlier than the semantic segment to which the current video frame belongs as the key. The video frame features and word features corresponding to the historical positions in the input sequence where attention connections were not blocked are determined as the observable historical input of the current video frame. The predicted description result is obtained by performing autoregressive inference based on the observable historical input.

[0156] Based on any of the above embodiments, a lexical feature acquisition unit is further included, wherein the lexical feature acquisition unit is specifically used for: According to the preset word segmentation rules, the tag description text is segmented to obtain a word sequence; Based on a pre-built vocabulary, each lexical in the lexical sequence is mapped to a lexical index; The lexical indexes are vectorized and mapped to obtain vector mapping results. The positional codes of each lexical in the lexical sequence are superimposed on the vector mapping results to obtain the lexical features of the tag description text.

[0157] The streaming video understanding device provided by the present invention is described below. The streaming video understanding device described below can be referred to in correspondence with the streaming video understanding method described above.

[0158] Based on any of the above embodiments, the present invention provides a streaming video understanding device. Figure 6 This is a schematic diagram of the structure of the streaming video understanding device provided by the present invention, as shown below. Figure 6 As shown, the device includes: The second acquisition unit 610 is used to acquire streaming video data to be understood; the streaming video data to be understood includes a continuous video frame sequence of the target to be understood, and task prompt words of the continuous video frame sequence. The input unit 620 is used to input the continuous video frame sequence and the task prompt words into the target streaming video understanding model to obtain the description prediction result output by the streaming video understanding model; the description prediction result is obtained by the streaming video understanding model after understanding the continuous video frame sequence based on the task type specified by the task prompt words and guided by the task prompt words. The streaming video understanding model is obtained by performing the training method described above.

[0159] The apparatus provided in this invention, by inputting streaming video frame sequences and task prompts into a streaming video understanding model obtained based on the above training method, enables the model to flexibly adapt to various online video understanding tasks under the guidance of task prompts without retraining for each task. Simultaneously, because the streaming video understanding model establishes a reliable probability estimate of the current semantic description under the context of evolutionary history during the training phase, the model transforms the silence decision from a generation target to a confidence-based verification state during the inference phase. It can autonomously decide the response timing with only a single forward propagation, completely bypassing the data imbalance, time inconsistency, target misalignment, and vocabulary degradation problems of traditional schemes. This significantly reduces the computational latency of streaming inference, achieving accurate and timely streaming video understanding.

[0160] Based on any of the above embodiments, a monitoring unit is further included, wherein the monitoring unit is specifically used for: When the total number of lexical units in the context window of the streaming video understanding model reaches the preset length limit, the semantic segment to which the context window belongs is determined to be the target semantic segment. Determine the verification perplexity corresponding to each video frame within the target semantic segment, and based on the verification perplexity, determine the semantic divergence of the current video frame in the streaming video frame sequence; Determine the median of the semantic divergence of each video frame within the target semantic segment, and identify video frames within the target semantic segment with a semantic divergence lower than the median as frames to be compressed; Remove the video frame features and the corresponding lexical features of the frame to be compressed from the context window, and retain only the end-description lexical features of the target semantic segment to obtain the compressed target semantic segment; the end-description lexical features are the lexical features corresponding to the end-description lexical, and the end-description lexical is the lexical located at the end position of the tag description text in the target semantic segment.

[0161] Based on any of the above embodiments, a semantic embedding update unit is further included, wherein the semantic embedding update unit specifically includes: An insertion unit is used to take the remaining video frames in the target semantic segment, excluding the frame to be compressed, and the lexical features corresponding to the remaining video frames as historical semantic units, and insert the historical semantic units into the recursive event tree. A similarity unit is determined to identify the highest semantic similarity based on the semantic similarity between the feature vector of the historical semantic unit and the feature vector of an existing tree node in the recursive event tree. A creation unit is used to attach the historical semantic unit as a child node to the existing tree node corresponding to the highest semantic similarity if the highest semantic similarity exceeds a preset semantic similarity threshold; otherwise, a new root node is created to store the historical semantic unit. Update the sub-unit, which is used to update the semantic embedding of the parent node corresponding to the child node.

[0162] Based on any of the above embodiments, the updating subunit is specifically used for: The existing tree node corresponding to the highest semantic similarity is determined as the parent node to be updated; Obtain the semantic embedding of the parent node to be updated, and the feature vector of the historical semantic unit attached as a child node; The preset momentum update rate is used as the first weight of the feature vector of the historical semantic unit, and the second weight of the semantic embedding of the parent node to be updated is determined based on the momentum update rate. Based on the first weight and the second weight, the feature vector of the historical semantic unit and the semantic embedding of the parent node to be updated are weighted and summed to obtain the updated semantic embedding of the parent node.

[0163] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include: a processor 710, a communications interface 720, a memory 730, and a communications bus 740, wherein the processor 710, the communications interface 720, and the memory 730 communicate with each other through the communications bus 740. The processor 710 can call logic instructions in the memory 730 to execute a training method for a streaming video understanding model. This method includes: acquiring training sample data from a streaming video; the training sample data includes multiple semantic segments divided according to semantic consistency, each semantic segment corresponding to a continuous video frame sequence and tag description text aligned with the continuous video frame sequence; arranging the video frame features of each video frame in the continuous video frame sequence and the lexical features of the tag description text in chronological order to obtain an input sequence; inputting the input sequence into an initial streaming video understanding model, whereby the initial streaming video understanding model, based on the input sequence, determines the observable historical input corresponding to each current video frame in the continuous video frame sequence, and outputs a predicted description result based on the observable historical input corresponding to each current video frame; determining a description generation loss based on the predicted description result and the tag description text; and updating the model parameters of the initial streaming video understanding model based on the description generation loss.

[0164] The processor 710 can call logic instructions in the memory 730 to execute a streaming video understanding method, the method comprising: acquiring streaming video data to be understood; the streaming video data to be understood includes a streaming video frame sequence of the target to be understood, and task prompts for the streaming video frame sequence; inputting the streaming video frame sequence and the task prompts into a streaming video understanding model to obtain a description prediction result output by the streaming video understanding model; the description prediction result is obtained by the streaming video understanding model after understanding the streaming video frame sequence based on the task type specified by the task prompts and guided by the task prompts; the streaming video understanding model is obtained by executing the training method of the above-mentioned streaming video understanding model.

[0165] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0166] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the training method of the streaming video understanding model provided by the above methods. The method includes: acquiring training sample data of streaming video; the training sample data includes multiple semantic segments divided according to semantic consistency, each semantic segment corresponding to a continuous video frame sequence and a tag description text aligned with the continuous video frame sequence; arranging the video frame features of each video frame in the continuous video frame sequence and the lexical features of the tag description text in chronological order to obtain an input sequence; inputting the input sequence into an initial streaming video understanding model, whereby the initial streaming video understanding model determines the observable historical input corresponding to each current video frame in the continuous video frame sequence based on the input sequence, and outputs a predicted description result based on the observable historical input corresponding to each current video frame; determining a description generation loss based on the predicted description result and the tag description text; and updating the model parameters of the initial streaming video understanding model based on the description generation loss.

[0167] When the computer program is executed by the processor, the computer can execute the streaming video understanding method provided by the above methods. The method includes: acquiring streaming video data to be understood; the streaming video data to be understood includes a streaming video frame sequence of the target to be understood, and task prompts for the streaming video frame sequence; inputting the streaming video frame sequence and the task prompts into a streaming video understanding model to obtain a description prediction result output by the streaming video understanding model; the description prediction result is obtained by the streaming video understanding model after understanding the streaming video frame sequence based on the task type specified by the task prompts and guided by the task prompts; the streaming video understanding model is obtained by executing the training method of the above-mentioned streaming video understanding model.

[0168] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a training method for the streaming video understanding model provided by the above methods. The method includes: acquiring training sample data of streaming video; the training sample data includes multiple semantic segments divided according to semantic consistency, each semantic segment corresponding to a continuous video frame sequence and tag description text aligned with the continuous video frame sequence; arranging the video frame features of each video frame in the continuous video frame sequence and the lexical features of the tag description text in chronological order to obtain an input sequence; inputting the input sequence into an initial streaming video understanding model, whereby the initial streaming video understanding model, based on the input sequence, determines the observable historical input corresponding to each current video frame in the continuous video frame sequence, and outputs a predicted description result based on the observable historical input corresponding to each current video frame; determining a description generation loss based on the predicted description result and the tag description text; and updating the model parameters of the initial streaming video understanding model based on the description generation loss.

[0169] When executed by a processor, this computer program implements the streaming video understanding methods provided by the above methods. The method includes: acquiring streaming video data to be understood; the streaming video data to be understood includes a sequence of streaming video frames of the target to be understood, and task prompts for the streaming video frame sequence; inputting the streaming video frame sequence and the task prompts into a streaming video understanding model to obtain a description prediction result output by the streaming video understanding model; the description prediction result is obtained by the streaming video understanding model understanding the streaming video frame sequence based on the task type specified by the task prompts and guided by the task prompts; the streaming video understanding model is obtained based on the training method described above.

[0170] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0171] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.< / eos> < / bbox> < / time> < / answer>

Claims

1. A training method for a streaming video understanding model, characterized in that, The streaming video understanding model is obtained by iteratively executing the following steps until a preset iteration termination condition is met: Acquire training sample data from streaming video; the training sample data includes multiple semantic segments divided according to semantic consistency, each semantic segment corresponding to a continuous video frame sequence and label description text aligned with the continuous video frame sequence; The video frame features of each video frame in the continuous video frame sequence are interleaved with the lexical features of the tag description text in chronological order to obtain the input sequence; The input sequence is input into the initial streaming video understanding model, which determines the observable historical input corresponding to each current video frame in the continuous video frame sequence based on the input sequence, and outputs the prediction description result based on the observable historical input corresponding to each current video frame. Based on the predicted description results and the labeled description text, the description generation loss is determined; The model parameters of the initial streaming video understanding model are updated based on the loss generated from the description.

2. The training method for the streaming video understanding model according to claim 1, characterized in that, The step of determining the description generation loss based on the predicted description result and the labeled description text includes: For each word in the tag description text, based on the observable historical input and the generated words in the tag description text preceding each word, the conditional probability of each word in the predicted description result is determined; The joint probability is determined based on the conditional probabilities of all lexical units in the predicted description results; Based on the joint probability, the description generation loss is determined.

3. The training method for the streaming video understanding model according to any one of claims 1 to 2, characterized in that, The step of inputting the input sequence into an initial streaming video understanding model, whereby the initial streaming video understanding model determines the observable historical input corresponding to each current video frame in the continuous video frame sequence based on the input sequence, and outputs a prediction description result based on the observable historical input corresponding to each current video frame, includes: In the attention calculation of the initial streaming video understanding model, within the same semantic segment, attention connections are blocked between the video frame features of the current video frame and the lexical features corresponding to the current video frame as queries, and the video frame features of the observed frames and the lexical features corresponding to the observed frames as keys; the observed frames are video frames preceding the current video frame in the corresponding continuous video frame sequence within the same semantic segment. Between different semantic segments, an attention connection is maintained between the video frame features of the current video frame and the lexical features corresponding to the current video frame as the query, and the lexical features corresponding to the terminating descriptive lexical features of each semantic segment that is temporally earlier than the semantic segment to which the current video frame belongs as the key. The video frame features and word features corresponding to the historical positions in the input sequence where attention connections were not blocked are determined as the observable historical input of the current video frame. The predicted description result is obtained by performing autoregressive inference based on the observable historical input.

4. The training method for the streaming video understanding model according to any one of claims 1 to 2, characterized in that, The steps for obtaining the lexical features of the tag description text include: According to the preset word segmentation rules, the tag description text is segmented to obtain a word sequence; Based on a pre-built vocabulary, each lexical in the lexical sequence is mapped to a lexical index; The lexical indexes are vectorized and mapped to obtain vector mapping results. The positional codes of each lexical in the lexical sequence are superimposed on the vector mapping results to obtain the lexical features of the tag description text.

5. A streaming video understanding method, characterized in that, include: Acquire streaming video data to be understood; the streaming video data to be understood includes a sequence of streaming video frames of the target to be understood, and task prompts in the streaming video frame sequence; The streaming video frame sequence and the task prompt are input into the streaming video understanding model to obtain the description prediction result output by the streaming video understanding model. The description prediction result is obtained by the streaming video understanding model after understanding the streaming video frame sequence based on the task type specified by the task prompt and guided by the task prompt. The streaming video understanding model is obtained by performing the training method of the streaming video understanding model as described in any one of claims 1 to 4.

6. The streaming video understanding method according to claim 5, characterized in that, The method further includes a hierarchical memory management step, which includes: When the total number of lexical units in the context window of the streaming video understanding model reaches the preset length limit, the semantic segment to which the context window belongs is determined to be the target semantic segment. Determine the verification perplexity corresponding to each video frame within the target semantic segment, and based on the verification perplexity, determine the semantic divergence of the current video frame in the streaming video frame sequence; Determine the median of the semantic divergence of each video frame within the target semantic segment, and identify video frames within the target semantic segment with a semantic divergence lower than the median as frames to be compressed; Remove the video frame features and the corresponding lexical features of the frame to be compressed from the context window, and retain only the end-description lexical features of the target semantic segment to obtain the compressed target semantic segment; the end-description lexical features are the lexical features corresponding to the end-description lexical, and the end-description lexical is the lexical located at the end position of the tag description text in the target semantic segment.

7. The streaming video understanding method according to claim 6, characterized in that, The method further includes: The remaining video frames in the target semantic segment, excluding the frame to be compressed, and the lexical features corresponding to the remaining video frames are taken as historical semantic units, and the historical semantic units are inserted into the recursive event tree. Based on the semantic similarity between the feature vector of the historical semantic unit and the feature vector of the existing tree node in the recursive event tree, the highest semantic similarity is determined; If the highest semantic similarity exceeds a preset semantic similarity threshold, the historical semantic unit is attached as a child node to the existing tree node corresponding to the highest semantic similarity; otherwise, a new root node is created to store the historical semantic unit. Update the semantic embedding of the parent node corresponding to the child node.

8. The streaming video understanding method according to claim 7, characterized in that, Updating the semantic embedding of the parent node corresponding to the child node includes: The existing tree node corresponding to the highest semantic similarity is determined as the parent node to be updated; Obtain the semantic embedding of the parent node to be updated, and the feature vector of the historical semantic unit attached as a child node; The preset momentum update rate is used as the first weight of the feature vector of the historical semantic unit, and the second weight of the semantic embedding of the parent node to be updated is determined based on the momentum update rate. Based on the first weight and the second weight, the feature vector of the historical semantic unit and the semantic embedding of the parent node to be updated are weighted and summed to obtain the updated semantic embedding of the parent node.

9. A training device for a streaming video understanding model, characterized in that, include: The first acquisition unit is used to acquire training sample data of the streaming video; The training sample data includes multiple semantic segments divided according to semantic consistency, each semantic segment corresponding to a continuous video frame sequence and a label description text aligned with the continuous video frame sequence; An arrangement unit is used to cross-arrange the video frame features of each video frame in the continuous video frame sequence with the lexical features of the tag description text in chronological order to obtain the input sequence; The prediction unit is used to input the input sequence into the initial streaming video understanding model, and the initial streaming video understanding model determines the observable historical input corresponding to each current video frame in the continuous video frame sequence based on the input sequence, and outputs the prediction description result based on the observable historical input corresponding to each current video frame. A loss determination unit is used to determine the description generation loss based on the predicted description result and the label description text; An update unit is used to update the model parameters of the initial streaming video understanding model based on the loss generated by the description.

10. A streaming video understanding device, characterized in that, include: The second acquisition unit is used to acquire the streaming video data to be understood. The streaming video data to be understood includes a continuous video frame sequence of the target to be understood, and task prompt words in the continuous video frame sequence; An input unit is used to input the continuous video frame sequence and the task prompt words into a target streaming video understanding model to obtain a description prediction result output by the streaming video understanding model; the description prediction result is obtained by the streaming video understanding model after understanding the continuous video frame sequence based on the task type specified by the task prompt words and guided by the task prompt words. The streaming video understanding model is obtained by performing the training method of the streaming video understanding model as described in any one of claims 1 to 4.

Images