A video understanding method and device, electronic equipment and storage medium

Abstract

This invention relates to the field of video understanding technology, and discloses a video understanding method, apparatus, electronic device, and storage medium. The method includes: constructing sample data; performing visual prediction pre-training on a visual understanding model based on the sample data to obtain a pre-trained model; the sample data includes a text-image interwoven sample reasoning chain; performing supervised fine-tuning of the pre-trained model for text-image reasoning based on the sample data to obtain a fine-tuned model; optimizing the fine-tuned model through reinforcement learning oriented towards text-image reasoning to obtain a target model; and using the target model, performing text-image reasoning based on a target video and a target question to obtain a target answer. This invention optimizes the model step-by-step through three stages: visual prediction pre-training, supervised fine-tuning, and reinforcement learning, enabling the model to learn to generate text-image interwoven reasoning chains and perform reasoning in a single-round, end-to-end manner. This allows the model to maintain reasoning accuracy while possessing the characteristics of low cost and high interpretability.

Description

Technical Field

[0001] This invention relates to the field of video understanding technology, and more specifically to a video understanding method, apparatus, electronic device, and storage medium. Background Technology

[0002] Video understanding refers to answering related questions given a video as input. Current mainstream video understanding methods typically follow this process: First, the video is processed by frame extraction, extracting a fixed set of N frames, and each frame is sequentially input into an image encoder to obtain image features; then, a neural network maps the image features to the feature space of a large model, generating image tokens; simultaneously, the text-based question is input into the large model to obtain text tokens; finally, the image tokens and text tokens are combined and input into the large model via autoregression to generate the answer.

[0003] However, the above technical solutions have the following drawbacks: First, the method essentially converts visual information into text representation and then relies entirely on a large language model for reasoning, which is significantly different from the cognitive mechanism humans use when processing video comprehension tasks—humans often watch videos repeatedly to locate key evidence before making a response; Second, the method lacks an effective reasoning chain and has poor response capabilities when faced with complex questions. Summary of the Invention

[0004] The present invention provides a video understanding method, apparatus, electronic device and storage medium to solve one of the technical problems existing in the prior art.

[0005] In a first aspect, the present invention provides a video understanding method, the method comprising: Construct sample data, and perform visual prediction pre-training on the visual understanding model based on the sample data to obtain the pre-trained model. The sample data includes a sample inference chain interwoven with images and text. Supervised fine-tuning of the pre-trained model using graph-text reasoning based on sample data yields the fine-tuned model. The target model is obtained by optimizing the fine-tuned model through reinforcement learning oriented towards graph and text reasoning. Using a target model, text-based reasoning is performed based on the target video and the target question to obtain the target answer.

[0006] This invention constructs sample data and pre-trains a visual understanding model for visual prediction, enabling the model to predict video frame content. Then, supervised fine-tuning of the pre-trained model using graph-text reasoning further teaches it to perform multi-step reasoning according to a graph-text interwoven format, linking textual reasoning with visual evidence. This addresses the problem of poor answering ability for complex questions due to the separation of visual and textual information and the lack of reasoning chains. Next, reinforcement learning for graph-text reasoning is used to optimize the fine-tuned model, using rewards to guide the model to further improve the quality of its generated graph-text interwoven reasoning chains, enhancing the accuracy of the final answer and the interpretability of the reasoning process. Finally, the target video and target question are input into the target model for reasoning, achieving single-round, end-to-end video understanding and reducing computational costs. These steps work together to enable the model to maintain reasoning accuracy while possessing low cost and high interpretability.

[0007] In one alternative implementation, the sample data further includes sample videos, and the visual understanding model includes a backbone module, a visual encoder, and a projection module. The visual understanding model is pre-trained for visual prediction based on sample data, resulting in a pre-trained model, including: The sample video is sampled to obtain multiple sampled images; Construct pre-trained cue words, which include the index of the predicted frame; The core module of the visual understanding model is used to infer the predictive features of the predicted frame based on multi-frame sampled images and pre-trained prompt words. The sample image corresponding to the index of the predicted frame is obtained from the sample video, and the sample features are obtained by the visual encoder of the visual understanding model. Pooling is performed on the predicted features and sample features respectively, and the projection module of the visual understanding model is used to align the pooled predicted features and the pooled sample features. The pre-training loss is calculated based on the difference between the aligned sample features and the predicted features. The visual understanding model is updated based on the pre-training loss, and the process returns to the step of sampling the sample video to obtain multiple sampled images, until the pre-training loss is less than the first preset threshold, thus obtaining the pre-trained model.

[0008] This invention enables the model to learn the temporal correlation and content evolution rules between video frames through a pre-training process, and has the ability to predict the content of unobserved frames based on observed frames. Thus, it can autonomously generate images of unsampled frames during subsequent inference, while avoiding the problem of losing key information in long videos due to fixed frame sampling.

[0009] In one alternative implementation, the sample data also includes sample questions, a graph-text interwoven sample reasoning chain, and sample answers, and the pre-trained model includes a visual classifier; Supervised fine-tuning of the pre-trained model based on sample data for text-image reasoning yields the fine-tuned model, including: The sample inference chain is rendered as an image canvas, and a visual encoder is used to obtain the training visual features. Using the backbone module, based on sample videos and sample questions, we infer the training inference chain, the training answer, and the training hidden state corresponding to the training inference chain. A visual classifier is used to classify the training hidden states to obtain the corresponding categories; A projection module is used to align the training hidden state with the training visual features; The supervised loss is calculated based on the sample answer, the training answer, the image canvas, the category of each training hidden state, and the training visual features. The main module, projection module, and visual classifier are updated based on the supervised loss. The process then returns to the step of rendering the sample inference chain into an image canvas and using a visual encoder to obtain training visual features, until the supervised loss is less than a second preset threshold, thus obtaining a fine-tuned model.

[0010] This invention addresses the problem of direct supervision between sample inference chains and training inference chains due to their different modalities by rendering the image-text interwoven sample inference chains as an image canvas and extracting training visual features. The main module infers the training inference chain and its corresponding training hidden states, a visual classifier performs classification, and a projection module aligns the hidden states with canvas features, enabling the model to learn to generate image-text interwoven inference chains. Visual evidence is bound to each inference step, resolving the issues of fragmented visual and textual information and poor ability to answer complex questions due to the lack of inference chains.

[0011] In one alternative implementation, a supervised loss is calculated based on the sample answer, the training answer, the image canvas, the category of each training hidden state, and the training visual features, including: The language loss is calculated based on the difference between the training answer and the sample answer; The class loss is calculated based on the difference between the class of each trained hidden state and the class of its corresponding region in the image canvas; The visual loss is calculated based on the difference between the aligned trained hidden state and the trained visual features. By combining language loss, category loss, and visual loss, supervised loss is obtained.

[0012] This invention calculates language loss, category loss, and visual loss separately and merges them into a total supervised loss. It simultaneously supervises model learning from three dimensions: language accuracy, text-image structure rationality, and visual content authenticity. This ensures that the model guarantees correct answers, clear reasoning chain structure, and authentic and credible visual evidence during the reasoning process, thereby improving the model's accuracy in answering complex video questions and the interpretability of the reasoning process.

[0013] In one alternative implementation, the fine-tuned model is optimized through reinforcement learning for graph-text reasoning to obtain the target model, including: Using a backbone module, multiple prediction inference chains are derived based on sample videos and sample questions in the sample data. Calculate the reinforcement reward for each predictive inference chain; Based on the reinforcement rewards of all prediction inference chains corresponding to the sample videos and sample questions, calculate the relative advantage value of each prediction inference chain. The backbone module is updated based on the relative advantage value of each prediction inference chain. The process then returns to the backbone module, and based on the sample videos and sample questions in the sample data, the steps of multiple prediction inference chains are inferred until a preset stopping condition is met, resulting in the target model.

[0014] This invention generates multiple predictive inference chains through multiple samplings and calculates reinforcement rewards for each chain. Then, it calculates the relative advantage value based on the rewards of all inference chains in the same group, thereby updating the main module. This allows the model to gradually optimize the generation quality of inference chains during exploration, solving the performance bottleneck that is difficult to overcome by supervised fine-tuning alone, and improving the model's inference accuracy and answer correctness for complex problems.

[0015] In one alternative implementation, the reinforcement reward for each predictive inference chain is calculated, including: A visual classifier is used to classify the predicted hidden states of each prediction inference chain to obtain the category of each predicted hidden state. Calculate the interval between the predicted hidden states of each pair of adjacent images of the same category, and calculate the coefficient of variation for all intervals as the structural reward. The number of predicted images is determined based on the continuous state of the predicted hidden state of the category image, the expected number of images of the sample video is determined, and the quantity reward is calculated based on the difference between the number of predicted images and the expected number of images. The predicted hidden states of consecutive categories of images are divided into a state group, the state group is pooled into a predicted feature vector, each sampled image of the sample video is pooled into a sampled feature vector, the similarity matrix between the predicted feature vector and the sampled feature vector is calculated, bipartite graph matching is performed based on the similarity matrix, and the content reward is determined based on the bipartite graph matching result. The sequential reward is determined based on the inversions in the sequence formed by each sampled image in the bipartite graph matching results. By combining structural rewards, quantity rewards, content rewards, and sequence rewards, an enhanced reward is obtained.

[0016] This invention calculates rewards from four dimensions—structure, quantity, content, and sequence—to guide the model in optimizing the quality of the reasoning chain. This results in a reasoning chain with a compact structure, a reasonable number of images, authentic and well-founded content, and correct temporal logic, thereby improving the accuracy of the final answer and the interpretability of the reasoning process.

[0017] In one optional implementation, based on the reinforcement rewards of all prediction inference chains corresponding to the sample videos and sample questions, the relative advantage value of each prediction inference chain is calculated, including: The mean and standard deviation are calculated based on the reinforcement rewards of all predictive inference chains corresponding to the sample videos and sample questions. For each predictive inference chain, calculate the difference between the reinforcement reward of the predictive inference chain and the mean, and use the ratio of the difference to the standard deviation as the relative advantage value of the predictive inference chain.

[0018] This invention calculates the mean and standard deviation of the reinforcement rewards of all prediction inference chains in the same group, and divides the difference between the reward of each inference chain and the mean by the standard deviation to obtain the relative advantage value, thereby guiding the model to gradually generate better inference chains.

[0019] In a second aspect, the present invention provides a video understanding device, the device comprising: The pre-training module is used to construct sample data and perform visual prediction pre-training on the visual understanding model based on the sample data to obtain the pre-trained model. The sample data includes a sample inference chain that interweaves images and text. The supervised fine-tuning module is used to perform supervised fine-tuning of the pre-trained model for text-to-image reasoning based on sample data, resulting in a fine-tuned model. The reinforcement learning module is used to optimize the fine-tuned model through reinforcement learning oriented towards graph and text reasoning, so as to obtain the target model; The video understanding module is used to perform text-based reasoning based on the target video and the target question using the target model to obtain the target answer.

[0020] Thirdly, the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the video understanding method described in the first aspect or any corresponding embodiment thereof.

[0021] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the video understanding method described in the first aspect or any corresponding embodiment thereof. Attached Figure Description

[0022] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a video understanding method according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating the calculation of category loss and visual loss according to an embodiment of the present invention; Figure 3 This is a flowchart of supervised fine-tuning according to an embodiment of the present invention; Figure 4 This is a structural block diagram of a video understanding device according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] It is understood that before using the technical solutions disclosed in the various embodiments of the present invention, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in the present invention and their authorization should be obtained in accordance with relevant laws and regulations through appropriate means.

[0026] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0027] According to an embodiment of the present invention, a video understanding method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0028] This embodiment provides a video understanding method. Figure 1 This is a flowchart of a video understanding method according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps: Step S101: Construct sample data, perform visual prediction pre-training on the visual understanding model based on the sample data, and obtain a pre-trained model. The sample data includes a sample inference chain interwoven with images and text.

[0029] Specifically, the sample data includes sample videos, sample questions, image-text interwoven sample reasoning chains, and sample answers. The image-text interwoven sample reasoning chain refers to a sequence of multiple reasoning steps, each containing textual reasoning information and a corresponding video keyframe—a multimodal data format alternating between textual reasoning and visual evidence. For example, for a sample video of a kitchen scene with the sample question "What ingredient did this person take first?", the image-text interwoven sample reasoning chain can be represented as: C1 "I noticed the person in the video walked towards the stove" → I1 (video frame screenshot of the stove area) → C2 "The person's right hand reached towards a red object" → I2 (video frame screenshot of the red object) → C3 "Confirmed the red object is a tomato" → I3 (video frame screenshot of the tomato), and the sample answer is "tomato". This image-text interwoven reasoning chain directly corresponds to the human cognitive method of reasoning while watching, with each reasoning step bound to specific visual evidence, fundamentally solving the problem of the separation between visual and textual information.

[0030] Based on this, the visual understanding model is pre-trained using sample data to enable it to predict the content of video frames. This allows the model to autonomously predict and generate keyframe images corresponding to the reasoning steps based on the observed video content during subsequent reasoning, providing a basic capability to support the generation of image-text interwoven reasoning chains.

[0031] Step S102: Perform supervised fine-tuning of the pre-trained model for text-to-image reasoning based on sample data to obtain the fine-tuned model.

[0032] Specifically, by performing supervised fine-tuning of graph-text reasoning on the basis of the pre-trained model, the model can further learn to perform multi-step reasoning according to the interwoven graph-text format, which provides a strategy for subsequent reinforcement learning optimization.

[0033] Step S103: Optimize the fine-tuned model through reinforcement learning for graph-text reasoning to obtain the target model.

[0034] Specifically, the fine-tuning model is used as a policy network for reinforcement learning. By rewarding the model, the quality of the generated graph-text interwoven reasoning chain is further optimized, thereby improving the accuracy of the final answer and the interpretability of the reasoning process.

[0035] Step S104: Using the target model, perform text-based reasoning based on the target video and target question to obtain the target answer.

[0036] Specifically, the target video and corresponding target question are input into a pre-trained target model, which then autonomously performs video understanding and reasoning to output the target answer. After receiving the target video and target question, the model uses a single-round, end-to-end autoregressive generation method to gradually generate a reasoning chain interwoven with text and images and the final answer, realizing a complete reasoning process from video input to answer output, achieving low-cost and high-reliability video understanding and reasoning.

[0037] This invention constructs sample data and pre-trains a visual understanding model for visual prediction, enabling the model to predict video frame content. Then, supervised fine-tuning of the pre-trained model using graph-text reasoning further teaches it to perform multi-step reasoning according to a graph-text interwoven format, linking textual reasoning with visual evidence. This addresses the problem of poor answering ability for complex questions due to the separation of visual and textual information and the lack of reasoning chains. Next, reinforcement learning for graph-text reasoning is used to optimize the fine-tuned model, using rewards to guide the model to further improve the quality of its generated graph-text interwoven reasoning chains, enhancing the accuracy of the final answer and the interpretability of the reasoning process. Finally, the target video and target question are input into the target model for reasoning, achieving single-round, end-to-end video understanding and reducing computational costs. These steps work together to enable the model to maintain reasoning accuracy while possessing low cost and high interpretability.

[0038] This embodiment provides a video understanding method, which specifically includes the following steps: Step S201: Construct sample data, perform visual prediction pre-training on the visual understanding model based on the sample data, and obtain a pre-trained model. The sample data includes a sample inference chain with interwoven text and images and sample videos. The visual understanding model includes a backbone module, a visual encoder, and a projection module.

[0039] Specifically, step S201 above performs visual prediction pre-training on the visual understanding model based on sample data to obtain a pre-trained model, including: Step S2011: Sample the sample video to obtain multiple sampled images.

[0040] Specifically, several frames are sampled at equal intervals from the sample video to obtain corresponding sampled images, providing the model with sparse but globally comprehensive visual cues of the video content. For example, if the sample video has 100 frames, 16 sampled images can be uniformly extracted at a sampling interval of 6 frames.

[0041] Step S2012: Construct pre-trained cue words, which include the index of the predicted frame.

[0042] Specifically, the pre-training prompts are text instructions containing the total number of frames in the sample video, the number of frames in the sampled images, the sampling interval, and the index of the predicted frame. These prompts inform the model of the current temporal range of the video and the location of the target frame to be predicted. For example, if the video has 100 frames, 16 frames are input, the sampling interval is 6 frames, and the prediction is for the 33rd frame. The pre-training prompts explicitly define the prediction target for the model, guiding it to infer the content of unsampled video frames based on the observed sampled frames, thereby activating the model's ability to understand the temporal structure of the video.

[0043] Step S2013: Using the backbone module of the visual understanding model, based on multi-frame sampled images and pre-trained prompts, the predicted features of the predicted frame are inferred.

[0044] Specifically, the backbone module, namely the large language model within the Transformer architecture, is the core component of the visual understanding model, responsible for semantic understanding and autoregressive generation. Based on the input sampled frame images and pre-trained prompts, the backbone module generates predicted features for each prediction frame through autoregressive inference, token by token. Here, a token is the basic representation unit of text or image information within the model, and each token corresponds to a feature vector. To ensure a consistent output format, the model generates prediction results according to a predefined structure, i.e., sequentially outputting the starting labels. The system includes several feature tokens representing the content of the predicted frames, and an end tag. The main module predicts the content of unobserved video frames by comprehensively understanding the temporal relationships and content evolution patterns between observed frames.

[0045] Step S2014: Obtain the sample image corresponding to the index of the predicted frame from the sample video, and use the visual encoder of the visual understanding model to obtain the sample features.

[0046] Specifically, the corresponding sample image is obtained from the sample video according to the index of the predicted frame. For example, if the index of the predicted frame is 33, then the 33rd frame image is obtained as its corresponding sample image from the sample video. A visual encoder from the visual understanding model is used to encode this sample image to obtain sample features. The visual encoder can be a general visual model, such as a visual coding network based on the Transformer structure.

[0047] In step S2015, the predicted features and sample features are pooled separately, and the projection module of the visual understanding model is used to align the pooled predicted features and the pooled sample features.

[0048] Specifically, the predicted features output by the backbone module contain several tokens, and the sample features output by the visual encoder also contain several feature vectors. Pooling operations are performed on both the predicted and sample features, i.e., the average value of the two features is taken separately. To facilitate comparison between the two features, a projection module is used to map the pooled predicted features onto the feature space of the sample features, placing the two features in the same space, thus allowing direct calculation of their differences. The projection module is an MLP Projector (Multi-Layer Perceptron Projector).

[0049] Step S2016: Calculate the pre-training loss based on the difference between the aligned sample features and the predicted features.

[0050] Specifically, after aligning the two features, the mean squared error loss between them is calculated as the pre-training loss, which reflects the degree of difference between the video frame content predicted by the model and the actual video frame content. The smaller the loss value, the stronger the model's visual prediction ability.

[0051] Step S2017: Update the visual understanding model based on the pre-training loss, return to the step of sampling the sample video to obtain multiple frames of sampled images, until the pre-training loss is less than the first preset threshold, and obtain the pre-trained model.

[0052] Specifically, based on the pre-training loss, the parameters of the backbone module and projection module in the visual understanding model are updated using the backpropagation algorithm, while the other parameters are not updated. After the update is completed, the process returns to step S2011 to sample the next batch of video data and continue pre-training. This process is repeated cyclically, allowing the model to gradually learn the temporal relationships and content evolution patterns between video frames until the pre-training loss drops below a first preset threshold. At this point, the model is considered to have a relatively accurate visual prediction ability, and the pre-trained model is obtained.

[0053] Step S202: Based on the sample data, the pre-trained model is fine-tuned by supervised text-image reasoning to obtain the fine-tuned model. The sample data also includes sample questions, text-image interwoven sample reasoning chains, and sample answers. The pre-trained model includes a visual classifier.

[0054] Specifically, step S202 includes: In step S2021, the sample inference chain is rendered as an image canvas, and the training visual features are obtained using a visual encoder.

[0055] Specifically, because the textual reasoning information and visual evidence in the sample inference chain are arranged in a multimodal alternation structure, while the training inference chain generated by the model is a token sequence, the two have different lengths and modalities, making direct one-to-one supervision impossible. Even if the sample inference chain is forcibly converted into token form to match the length of the training inference chain, it will lead to a surge in the number of tokens, resulting in significant computational overhead and poor supervision performance. Therefore, this embodiment of the invention, through steps S2021-S2024, enables the originally unsupervised text-image interwoven inference chain to achieve effective supervision through a unified visual representation, while significantly reducing the number of tokens during training and lowering computational overhead.

[0056] First, the sample inference chain is rendered into a vertical striped image canvas using OCR (Optical Character Recognition). The resolution (height) of the entire image is controlled to not exceed a certain threshold, which in turn determines the scaling ratio of the images and the size of the text within the long image, ensuring the canvas visually presents the inference structure interwoven with text and images. OCR rendering refers to the process of compositing text and images onto a canvas according to a specified layout. For example, text descriptions from the inference steps are arranged sequentially with corresponding video frame screenshots to form a unified visual representation. A visual encoder from a visual understanding model is used to divide the image canvas into... Each patch corresponds to a token. The feature vector of each patch is extracted to form the training visual features of the image canvas, which are used as the visual supervision signal in the supervised fine-tuning stage.

[0057] Step S2022: Using the backbone module, based on the sample video and sample question, the training inference chain, training answer, and training hidden state corresponding to the training inference chain are obtained.

[0058] Specifically, sample videos and sample questions are input into the backbone module, which employs a 26-layer Transformer architecture. It generates training inference chains and training answers through autoregressive reasoning, inferring token by token. The inference chain is a sequence of interwoven text and image tokens generated by the model in the order of the reasoning steps before outputting the final answer. Its form is: text token sequence (corresponding to text reasoning steps), image token sequence (corresponding to video keyframes, i.e., visual evidence), text token sequence, image token sequence… arranged alternately. When generating each token, the backbone module generates a corresponding feature representation at its 25th layer (the penultimate layer). This layer's features retain rich semantic information but have not yet undergone the final layer's mapping to the vocabulary space. This feature representation serves as the training hidden state for the token. Each token in the training inference chain corresponds to a training hidden state.

[0059] Step S2023: Use a visual classifier to classify the training hidden states and obtain the corresponding categories.

[0060] Specifically, a visual classifier, specifically a visual token classifier, is used to classify each training hidden state to determine whether its corresponding token is an image or text. The visual classifier is a binary classification MLP structure, identical in structure to the projection module but with different functions. Its input is the training hidden state, and its output is the probability distribution of whether the token belongs to an image or text. The class with the highest probability value is then used as the final predicted class for that token.

[0061] Step S2024: Use the projection module to align the training hidden state and the training visual features.

[0062] Specifically, since the training hidden states come from the backbone module (the feature space of the language model), while the training visual features come from the visual encoder (the visual feature space), and the two reside in different feature spaces, they cannot be directly compared. Therefore, a projection module is used to map the training hidden states to the visual feature space where the training visual features reside, so that the hidden state corresponding to each token generated by the model is located in the same space as the visual features of the corresponding region in the canvas, thus allowing direct calculation of the difference between the two.

[0063] Step S2025: Calculate the supervised loss based on the sample answer, training answer, image canvas, category of each training hidden state, and training visual features.

[0064] In some optional implementations, step S2025 above includes: Step a1: Calculate the language loss based on the difference between the training answer and the sample answer.

[0065] Specifically, the training answers are generated by the backbone module and are in the form of text sequences; the sample answers are the pre-labeled correct results in the sample data, also in the form of text sequences. The training answers and sample answers are compared token-by-token, and the cross-entropy loss between them is calculated as the language loss. This loss is used to supervise whether the final answer generated by the model matches the sample answer, ensuring that the model can provide the correct answer based on the video content and the question.

[0066] Step a2: Calculate the category loss based on the difference between the category of each trained hidden state and the category of its corresponding region in the image canvas.

[0067] Specifically, each training hidden state is processed by a visual classifier to obtain its corresponding category. Simultaneously, when the image-text interwoven sample inference chain is rendered as an image canvas, the image and text in the sample inference chain occupy corresponding areas on the image canvas, thus obtaining the true category of each area on the image canvas. The training inference chain output by the model is aligned with the sample inference chain in terms of sequence arrangement. According to the correspondence of sequence positions, each training hidden state of the training inference chain can be matched to the corresponding area on the image canvas. The model's predicted category is compared with the true category in the canvas, and a binary cross-entropy loss is calculated as the category loss. This loss is used to enable the model to learn during fine-tuning which positions in the inference chain should generate image tokens and which should generate text tokens, thereby gradually mastering the ability to generate interwoven image-text structures.

[0068] Step a3: Calculate the visual loss based on the difference between the aligned training hidden state and the training visual features.

[0069] Specifically, the aligned training hidden states and training visual features are in the same visual feature space, and both are in the form of feature vectors. The mean squared error loss between the two is calculated as the visual loss, which is used to make the hidden state corresponding to each token generated by the model gradually approximate the visual features of the corresponding region in the canvas.

[0070] Step a4: Combine language loss, category loss, and visual loss to obtain supervised loss.

[0071] Specifically, language loss, category loss, and visual loss supervise the model's learning process from different dimensions. Weight coefficients are set for the three losses, and the three losses are weighted and summed according to their respective weights to obtain the supervised loss.

[0072] In some alternative implementations, Figure 2 This is a flowchart illustrating the calculation of category loss and visual loss according to an embodiment of the present invention, such as... Figure 2As shown, a visual encoder is used to obtain the training visual features of the image canvas. The training hidden states corresponding to the training inference chain obtained by the backbone module based on sample videos and sample questions are aligned with the training visual features, and the visual loss between the two is calculated after alignment. A visual classifier is used to classify the training hidden states to obtain the corresponding categories. Based on the true category of the corresponding region of the training hidden state in the image canvas, the category loss between the two is calculated.

[0073] Step S2026: Update the backbone module, projection module and visual classifier based on the supervised loss, return to the step of rendering the sample inference chain as an image canvas and using a visual encoder to obtain training visual features, until the supervised loss is less than the second preset threshold, and obtain the fine-tuned model.

[0074] Specifically, the parameters of the backbone module, projection module, and visual classifier are updated synchronously via backpropagation using supervised loss, while other parameters are not updated. After the update is complete, the process returns to step S2021 to render the inference chain for the next sample data and continue supervised fine-tuning. This process is repeated cyclically, allowing the model to simultaneously optimize text inference ability, image-text structure generation ability, and visual content prediction ability during fine-tuning, until the supervised loss decreases to less than a second preset threshold, resulting in a fine-tuned model. Simultaneously, through joint optimization, the model can utilize image-based inference chain information while learning the inference process, thereby improving the training efficiency and model performance of video inference tasks.

[0075] In some alternative implementations, Figure 3 This is a flowchart of supervised fine-tuning according to an embodiment of the present invention, such as... Figure 3 As shown, the sample inference chain is rendered into an image canvas using OCR, and its training visual features are extracted using a visual encoder. The backbone module performs inference on the sample video and sample question to obtain the training answer and the corresponding training hidden states of the training inference chain. The training hidden states are then classified using a visual classifier to obtain the categories. The training hidden states and training visual features are aligned using a projection module. Then, the language loss, category loss, and visual loss are calculated separately and fused into a supervised loss, which is used to update the model. The process continues with the next sample data, repeating the above steps until the supervised loss is less than a second preset threshold, completing the supervised fine-tuning and obtaining the fine-tuned model.

[0076] Step S203: Optimize the fine-tuned model through reinforcement learning for graph-text reasoning to obtain the target model.

[0077] Specifically, step S203 includes: Step S2031: Using the backbone module, multiple prediction inference chains are obtained based on sample videos and sample questions in the sample data.

[0078] Specifically, the fine-tuned model is optimized using reinforcement learning through the GRPO (Group Relative Policy Optimization) algorithm. Specifically, the supervised fine-tuned backbone module is first used as the reinforcement learning policy network. For the same sample video and sample question, randomness (such as a sampling strategy) is introduced into the backbone module, generating multiple different prediction inference chains through multiple samplings.

[0079] Step S2032: Calculate the reinforcement reward for each predictive inference chain.

[0080] In some optional implementations, step S2032 above includes: Step b1: Use a visual classifier to classify the predicted hidden state of each prediction inference chain to obtain the category of each predicted hidden state.

[0081] Specifically, for each prediction inference chain, the predicted hidden state corresponding to each token in the inference chain is extracted, input into the visual classifier trained in the supervised fine-tuning stage, and the category (image or text) corresponding to each predicted hidden state is output.

[0082] Step b2: Calculate the interval between the predicted hidden states of every two adjacent images of the same category, and calculate the coefficient of variation for all intervals as the structural reward.

[0083] Specifically, for each prediction inference chain, based on the classification result, the position indices of all predicted hidden states classified as images in that inference chain are obtained and arranged according to their order in the inference chain. Then, the positional interval between every two adjacent predicted hidden states is calculated. The standard deviation and mean of all intervals are calculated, and the ratio of the standard deviation to the mean is used as the coefficient of variation, i.e., the structural reward. The structural reward reflects the dispersion of image tokens in the inference chain. The larger the coefficient of variation, the more significant the difference in intervals between image tokens, that is, the more image tokens tend to cluster together, and the clearer and more compact the image-text interweaving structure.

[0084] For example, suppose a prediction inference chain has 10 tokens, where the tokens classified as images are located at positions 3, 4, 5, 8, 9, and 10. Then the interval between adjacent image tokens is [1, 1, 3, 1, 1], with a standard deviation of approximately 0.8, a mean of 1.4, and a coefficient of variation of 0.8 / 1.4 ≈ 0.57. If the image tokens are evenly distributed at positions 2, 4, 6, 8, and 10, then the interval is [2, 2, 2, 2], with a standard deviation of 0, a mean of 2, and a coefficient of variation of 0, indicating that the image tokens are scattered and have a loose structure.

[0085] Step b3: Determine the number of predicted images based on the continuity of the predicted hidden states of the category image, determine the expected number of images in the sample video, and calculate the quantity reward based on the difference between the number of predicted images and the expected number of images.

[0086] Specifically, based on the classification results of step b1, the number of consecutively arranged image token blocks in the predicted hidden states of all categories of images in the prediction inference chain is counted. This number of blocks represents the number of images predicted in the inference chain. Simultaneously, the expected number of images is determined according to the duration of the sample video: expect = log(max(1, video_length / 15s)) + 1, where video_length represents the duration of the sample video. Then, the predicted number of images is denoted as actual. The reward is calculated using the formula: reward_count = 1 - |actual - expect| / expect, which guides the model to adaptively generate a reasonable number of image evidences based on the video duration.

[0087] For example, when the sample video is 30 seconds long, the expected number of images is 1.69. If the model generates 2 images, the quantity reward is 1 - |2 - 1.69| / 1.69 ≈ 0.82; if the model generates 5 images, the quantity reward is 1 - |5 - 1.69| / 1.69 ≈ -0.96.

[0088] Step b4: Divide the predicted hidden states of consecutive categories of images into a state group, pool the state group into a predicted feature vector, pool each sampled image of the sample video into a sampled feature vector, calculate the similarity matrix between the predicted feature vector and the sampled feature vector, perform bipartite graph matching based on the similarity matrix, and determine the content reward based on the bipartite graph matching result.

[0089] Specifically, the predicted hidden states of consecutive categories being images are divided into state groups, with each state group corresponding to one image. All predicted hidden states within a state group are pooled to obtain a predicted feature vector. Simultaneously, each sampled image obtained in step S2011 of the sample video is pooled to obtain a corresponding sampled feature vector. The cosine similarity between each predicted feature vector and each sampled feature vector is calculated to construct a similarity matrix.

[0090] Based on this, bipartite graph matching is performed on the similarity matrix. Specifically, all predicted feature vectors are used as the first side nodes of the bipartite graph, and all sampled feature vectors are used as the second side nodes. Cosine similarity is used as the edge weights. The matching scheme that maximizes the sum of matching edge weights is solved, and this maximum sum of matching weights is used as the content reward. Each matching edge represents a one-to-one correspondence between a predicted feature vector and a sampled feature vector. A higher content reward indicates a better overall matching degree between the predicted image and the video sample image, meaning the image content generated by the model is more relevant to the real video footage, thus guiding the model to generate images supported by real visual evidence during reinforcement learning.

[0091] Step b5: Determine the sequential reward based on the inversion situation in the sequence formed by each sampled image in the bipartite graph matching result.

[0092] Specifically, based on the bipartite graph matching result of step b4, the frame number of the sampled image corresponding to the sampled feature vector matched by each predicted feature vector in the sample video is obtained. These frame numbers are then sorted according to the order of the corresponding predicted feature vectors in the prediction inference chain to obtain a frame number sequence. Next, the number of inversion pairs in this frame number sequence is calculated. An inversion pair is a pair where the frame number of the preceding element in the sequence is greater than the frame number of the following element. Fewer inversion pairs indicate that the order in which the predicted images appear in the inference chain is more consistent with the actual temporal order of the video frames; more inversion pairs indicate a more disordered order. The negative value of the number of inversion pairs is used as an order reward—fewer inversion pairs result in a higher reward, and more inversion pairs result in a lower reward. This guides the model to ensure that image evidence is presented sequentially according to the video's temporal order when generating the inference chain, thus guaranteeing the temporal logical correctness of the inference chain.

[0093] Step b6: Combine structure rewards, quantity rewards, content rewards, and sequence rewards to obtain enhanced rewards.

[0094] Specifically, structural rewards, quantity rewards, content rewards, and order rewards evaluate the quality of the prediction inference chain from different dimensions. Weight coefficients are set for each of the four rewards, and the four rewards are weighted and summed according to their respective weights to obtain the reinforcement reward. This enables the model to simultaneously optimize the generation quality of the four dimensions of structure, quantity, content, and order during the reinforcement learning process.

[0095] Step S2033: Based on the reinforcement rewards of all prediction inference chains corresponding to the sample videos and sample questions, calculate the relative advantage value of each prediction inference chain.

[0096] In some optional implementations, step S2033 above includes: Step c1: Calculate the mean and standard deviation based on the reinforcement rewards of all prediction inference chains corresponding to the sample videos and sample questions.

[0097] Specifically, the mean and standard deviation of the reinforcement rewards for all predictive inference chains generated from the same sample video and sample question are calculated.

[0098] Step c2: For each prediction inference chain, calculate the difference between the reinforcement reward of the prediction inference chain and the mean, and use the ratio of the difference to the standard deviation as the relative advantage value of the prediction inference chain.

[0099] Specifically, for each predictive inference chain, the difference between its reinforcement reward and the mean is calculated, and the ratio of this difference to the standard deviation is used as the relative advantage value of the predictive inference chain to measure its superiority or inferiority relative to other inference chains in the same group.

[0100] Step S2034: Update the backbone module based on the relative advantage value of each prediction inference chain, return to the backbone module, and infer multiple prediction inference chains based on sample videos and sample questions in the sample data until a preset stopping condition is reached to obtain the target model.

[0101] Specifically, the parameters of the backbone module are updated using the relative advantage value of each prediction inference chain through a policy gradient algorithm. For prediction inference chains with positive relative advantage values, their generation probability is increased; for prediction inference chains with negative relative advantage values, their generation probability is decreased, thereby causing the model to tend to generate higher-quality inference chains in subsequent generation. During the update process, the parameters of the backbone module are updated, while the parameters of the other modules remain frozen. After updating all prediction inference chains corresponding to the current sample video and sample question, the process returns to step S2031 to sample the sample video and sample question for the next sample data and continue reinforcement learning. The above process is iterated cyclically, allowing the model to gradually optimize the generation quality of the inference chains until a preset stopping condition is reached (such as reaching a preset number of iterations or the reinforcement reward tending to stabilize), resulting in the target model, which is used for subsequent video understanding and inference tasks.

[0102] Step S204: Using the target model, perform text-image reasoning based on the target video and target question to obtain the target answer. For details, please refer to [link to details]. Figure 1 Step S104 of the illustrated embodiment will not be described again here.

[0103] This invention constructs sample data and pre-trains a visual understanding model for visual prediction, enabling the model to predict video frame content. Then, supervised fine-tuning of the pre-trained model using graph-text reasoning further teaches it to perform multi-step reasoning according to a graph-text interwoven format, linking textual reasoning with visual evidence. This addresses the problem of poor answering ability for complex questions due to the separation of visual and textual information and the lack of reasoning chains. Next, reinforcement learning for graph-text reasoning is used to optimize the fine-tuned model, using rewards to guide the model to further improve the quality of its generated graph-text interwoven reasoning chains, enhancing the accuracy of the final answer and the interpretability of the reasoning process. Finally, the target video and target question are input into the target model for reasoning, achieving single-round, end-to-end video understanding and reducing computational costs. These steps work together to enable the model to maintain reasoning accuracy while possessing low cost and high interpretability.

[0104] The present invention has the following beneficial effects: (1) Compressed representation of multi-step inference information: The image-text interwoven inference chain is unified into an image canvas through OCR rendering, which reduces the number of inference tokens, improves training efficiency, and reduces memory usage.

[0105] (2) Joint visual and text training: Visual information and text reasoning steps are uniformly encoded, so that the model can learn text logic and visual evidence at the same time during the training process, which improves the accuracy and reliability of reasoning.

[0106] (3) Applicable to long videos and complex events: Through the representation of the image-based multi-step reasoning chain, it can efficiently handle complex reasoning tasks in long video scenarios, reduce training computation, and at the same time ensure the compact representation of multimodal supervision information.

[0107] This embodiment also provides a video understanding device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0108] This embodiment provides a video understanding device, such as... Figure 4 As shown, it includes: The pre-training module 401 is used to construct sample data and perform visual prediction pre-training on the visual understanding model based on the sample data to obtain the pre-trained model. The sample data includes a sample inference chain interwoven with images and text.

[0109] The supervised fine-tuning module 402 is used to perform supervised fine-tuning of the pre-trained model for text-image reasoning based on sample data, so as to obtain the fine-tuned model.

[0110] Reinforcement learning module 403 is used to optimize the fine-tuned model through reinforcement learning for graph-text reasoning to obtain the target model.

[0111] The video understanding module 404 is used to perform text-image reasoning based on the target video and the target question using the target model to obtain the target answer.

[0112] In some alternative implementations, the sample data also includes sample videos, and the visual understanding model includes a backbone module, a visual encoder, and a projection module. Pre-training module 401 includes: The sampling unit is used to sample the sample video to obtain multiple frames of sampled images.

[0113] The building unit is used to construct pre-trained cue words, which include the index of the prediction frame.

[0114] The first inference unit is used as the backbone module of the visual understanding model to infer the predictive features of the prediction frame based on multi-frame sampled images and pre-trained prompt words.

[0115] The first encoding unit is used to obtain the sample image corresponding to the index of the predicted frame from the sample video, and to obtain the sample features using the visual encoder of the visual understanding model.

[0116] The first alignment unit is used to pool the predicted features and sample features respectively, and uses the projection module of the visual understanding model to align the pooled predicted features and the pooled sample features.

[0117] The first computational unit is used to calculate the pre-training loss based on the difference between the aligned sample features and the predicted features.

[0118] The pre-training unit is used to update the visual understanding model based on the pre-training loss, and then return to the step of sampling the sample video to obtain multiple frames of sampled images until the pre-training loss is less than a first preset threshold, thus obtaining the pre-trained model.

[0119] In some alternative implementations, the sample data may also include sample questions, image-text interwoven sample reasoning chains, and sample answers, and the pre-trained model may include a visual classifier; The monitoring and fine-tuning module 402 includes: The second encoding unit is used to render the sample inference chain into an image canvas and use a visual encoder to obtain training visual features.

[0120] The second inference unit is used to infer the training inference chain, training answer, and training hidden state corresponding to the training inference chain based on the sample video and sample question using the backbone module.

[0121] The classification unit is used to classify the training hidden states using a visual classifier to obtain the corresponding categories.

[0122] The second alignment unit is used to align the training hidden state and the training visual features using the projection module.

[0123] The second computational unit is used to calculate the supervised loss based on the sample answer, the training answer, the image canvas, the category of each training hidden state, and the training visual features.

[0124] The supervised fine-tuning unit is used to update the backbone module, projection module and visual classifier based on the supervised loss, return to the step of rendering the sample inference chain into an image canvas and using a visual encoder to obtain training visual features, until the supervised loss is less than a second preset threshold, and the fine-tuned model is obtained.

[0125] In some alternative implementations, the second computing unit includes: The first computational subunit is used to calculate the language loss based on the difference between the training answer and the sample answer.

[0126] The second computational subunit is used to calculate the class loss based on the difference between the class of each trained hidden state and the class of its corresponding region in the image canvas.

[0127] The third computational subunit is used to calculate the visual loss based on the difference between the aligned training hidden state and the training visual features.

[0128] The first fusion subunit is used to fuse language loss, category loss, and visual loss to obtain supervised loss.

[0129] In some alternative implementations, reinforcement learning module 403 includes: The third inference unit is used to infer multiple predictive inference chains based on sample videos and sample questions in the sample data, using the backbone module.

[0130] The third computational unit is used to calculate the reinforcement reward for each predictive inference chain.

[0131] The fourth computational unit is used to calculate the relative advantage value of each prediction inference chain based on the reinforcement rewards of all prediction inference chains corresponding to the sample videos and sample questions.

[0132] The reinforcement learning unit is used to update the backbone module based on the relative advantage value of each prediction inference chain, and return to the backbone module to infer the steps of multiple prediction inference chains based on sample videos and sample questions in the sample data, until a preset stopping condition is reached to obtain the target model.

[0133] In some alternative implementations, the third computing unit includes: The classification subunit is used to classify the predicted hidden states of each prediction inference chain using a visual classifier to obtain the category of each predicted hidden state.

[0134] The fourth computational subunit is used to calculate the interval between the predicted hidden states of every two adjacent categories of images, and to calculate the coefficient of variation of all intervals as the structural reward.

[0135] The fifth calculation subunit is used to determine the number of predicted images based on the continuous situation of the predicted hidden state of the category image, determine the expected number of images in the sample video, and calculate the quantity reward based on the difference between the number of predicted images and the expected number of images.

[0136] The first determining subunit is used to divide the predicted hidden states of consecutive categories of images into a state group, pool the state group into a predicted feature vector, pool each sampled image of the sample video into a sampled feature vector, calculate the similarity matrix between the predicted feature vector and the sampled feature vector, perform bipartite graph matching based on the similarity matrix, and determine the content reward based on the bipartite graph matching result.

[0137] The second determining subunit is used to determine the order reward based on the inversion situation in the sequence formed by each sampled image in the bipartite graph matching result.

[0138] The second fusion subunit is used to fuse structural rewards, quantity rewards, content rewards, and sequence rewards to obtain enhanced rewards.

[0139] In some alternative implementations, the fourth computing unit includes: The sixth computational subunit is used to calculate the mean and standard deviation of the reinforcement rewards for all predictive inference chains corresponding to the sample videos and sample questions.

[0140] The seventh calculation subunit is used to calculate the difference between the reinforcement reward and the mean of each prediction inference chain, and the ratio of the difference to the standard deviation is used as the relative advantage value of the prediction inference chain.

[0141] The video understanding device provided in this embodiment of the invention can execute the video understanding method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the various modules and units described above are the same as in the corresponding embodiments described above, and will not be repeated here.

[0142] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0143] The following is a detailed reference. Figure 5The diagram illustrates a structural schematic suitable for implementing an electronic device according to embodiments of the present invention. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from memory 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device. The processor 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0144] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0145] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a memory 508, or installed from a ROM 502. When the computer program is executed by the processor 501, it performs the functions defined in the video understanding method of the embodiments of the present invention.

[0146] Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0147] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the video understanding method shown in the above embodiments is implemented.

[0148] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A video understanding method, characterized in that, The method includes: Construct sample data, and perform visual prediction pre-training on the visual understanding model based on the sample data to obtain a pre-trained model. The sample data includes a sample inference chain interwoven with images and text. Based on the sample data, supervised fine-tuning of the pre-trained model for text-image reasoning is performed to obtain the fine-tuned model; The target model is obtained by optimizing the fine-tuned model through reinforcement learning oriented towards graph and text reasoning. Using a target model, text-based reasoning is performed based on the target video and the target question to obtain the target answer.

2. The method according to claim 1, characterized in that, The sample data also includes sample videos, and the visual understanding model includes a backbone module, a visual encoder, and a projection module. The step of performing visual prediction pre-training on the visual understanding model based on the sample data to obtain a pre-trained model includes: The sample video is sampled to obtain multiple sampled images; Construct pre-trained cue words, wherein the pre-trained cue words include the index of the predicted frame; The core module of the visual understanding model is used to infer the predictive features of the predicted frame based on the multi-frame sampled images and the pre-trained prompt words. The sample image corresponding to the index of the predicted frame is obtained from the sample video, and the sample features are obtained by the visual encoder of the visual understanding model. The predicted features and the sample features are pooled respectively, and the projection module of the visual understanding model is used to align the pooled predicted features and the pooled sample features. The pre-training loss is calculated based on the difference between the aligned sample features and the predicted features. The visual understanding model is updated based on the pre-training loss, and the process returns to the step of sampling the sample video to obtain multiple sampled images, until the pre-training loss is less than a first preset threshold, thus obtaining the pre-trained model.

3. The method according to claim 2, characterized in that, The sample data also includes sample questions and sample answers, and the pre-trained model includes a visual classifier; The supervised fine-tuning of the pre-trained model based on the sample data to obtain the fine-tuned model includes: The sample inference chain is rendered into an image canvas, and a visual encoder is used to obtain the training visual features. Using the backbone module, based on the sample video and the sample question, the training inference chain, the training answer, and the training hidden state corresponding to the training inference chain are obtained; A visual classifier is used to classify the trained hidden states to obtain the corresponding categories; A projection module is used to align the training hidden state with the training visual features; Based on the sample answer, the training answer, the image canvas, the category of each training hidden state, and the training visual features, calculate the supervised loss; The main module, projection module, and visual classifier are updated based on the supervised loss. The process then returns to the step of rendering the sample inference chain into an image canvas and using a visual encoder to obtain training visual features, until the supervised loss is less than a second preset threshold, thus obtaining a fine-tuned model.

4. The method according to claim 3, characterized in that, The process of calculating supervised loss based on the sample answer, the training answer, the image canvas, the category of each training hidden state, and the training visual features includes: Calculate the language loss based on the difference between the training answer and the sample answer; The class loss is calculated based on the difference between the class of each trained hidden state and the class of the corresponding region in the image canvas; The visual loss is calculated based on the difference between the aligned trained hidden state and the trained visual features. The supervised loss is obtained by fusing the language loss, the category loss, and the visual loss.

5. The method according to claim 3, characterized in that, The optimization of the fine-tuned model through reinforcement learning oriented towards graph-text reasoning to obtain the target model includes: Using a backbone module, multiple prediction inference chains are derived based on sample videos and sample questions in the sample data; Calculate the reinforcement reward for each predictive inference chain; Based on the reinforcement rewards of all prediction inference chains corresponding to the sample video and the sample question, calculate the relative advantage value of each prediction inference chain. The backbone module is updated based on the relative advantage value of each predictive inference chain. The process then returns to the backbone module and, based on the sample videos and sample questions in the sample data, infers the steps of multiple predictive inference chains until a preset stopping condition is met, thus obtaining the target model.

6. The method according to claim 5, characterized in that, The calculation of the reinforcement reward for each predictive inference chain includes: A visual classifier is used to classify the predicted hidden states of each prediction inference chain to obtain the category of each predicted hidden state. Calculate the interval between the predicted hidden states of each pair of adjacent images of the same category, and calculate the coefficient of variation for all intervals as the structural reward. The number of predicted images is determined based on the continuous state of the predicted hidden state of the category image, the expected number of images of the sample video is determined, and a quantity reward is calculated based on the difference between the number of predicted images and the expected number of images. The predicted hidden states of consecutive categories of images are divided into a state group, the state group is pooled into a predicted feature vector, each sampled image of the sample video is pooled into a sampled feature vector, the similarity matrix between the predicted feature vector and the sampled feature vector is calculated, bipartite graph matching is performed based on the similarity matrix, and the content reward is determined based on the bipartite graph matching result. The sequential reward is determined based on the inversions in the sequence formed by each sampled image in the bipartite graph matching results. The enhancement reward is obtained by combining the structural reward, the quantity reward, the content reward, and the order reward.

7. The method according to claim 5, characterized in that, The reinforcement reward based on the sample video and the sample question for all predictive inference chains, calculating the relative advantage value of each predictive inference chain, includes: Calculate the mean and standard deviation based on the reinforcement rewards of all prediction inference chains corresponding to the sample video and the sample question; For each predictive inference chain, the difference between the reinforcement reward of the predictive inference chain and the mean is calculated, and the ratio of the difference to the standard deviation is taken as the relative advantage value of the predictive inference chain.

8. A video understanding device, characterized in that, The device includes: The pre-training module is used to construct sample data and perform visual prediction pre-training on the visual understanding model based on the sample data to obtain the pre-trained model. The sample data includes a sample inference chain that interweaves images and text. The supervised fine-tuning module is used to perform supervised fine-tuning of the pre-trained model for text-image reasoning based on the sample data, so as to obtain the fine-tuned model; The reinforcement learning module is used to optimize the fine-tuned model through reinforcement learning oriented towards graph and text reasoning, so as to obtain the target model; The video understanding module is used to perform text-based reasoning based on the target video and the target question using the target model to obtain the target answer.

9. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory storing computer instructions, and the processor executing the computer instructions to perform the video understanding method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the video understanding method according to any one of claims 1 to 7.

Images