Dynamic iterative long video understanding method based on large language model

Abstract

The invention provides a dynamic iterative long video understanding method based on a large language model. The method comprises the following steps: step 1, carrying out mathematical modeling and analysis on a video understanding task; 2, preprocessing a video input by a user, and performing preliminary reasoning through a question and answer Agent; 3, self-supervised information feedback is carried out, and an evaluation Agent is introduced in each step in the reasoning process to recognize answers; 4, quantitative analysis is carried out by adopting the question and answer accuracy and the average retrieval frame number as evaluation indexes, and the effectiveness of the method is verified; any video and question provided by the user are adopted for qualitative analysis, and whether the result meets the expectation or not is verified. According to the method, intelligent analysis and reasoning of video contents are realized by simulating a dynamic optimization process of a human logical thinking chain. The method is widely applied to the fields of video intelligent analysis, video question-answering systems, video content recommendation and the like, and is beneficial to promoting the technical development of rapid understanding and accurate reasoning of long videos.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and computer vision technology, and in particular to a dynamic iterative long video understanding method based on a large language model. Background Art

[0002] Traditional video understanding methods usually rely on dense sampling and global analysis, such as uniformly extracting video frames and extracting features using convolutional neural networks (CNNs) or time series models (such as LSTMs). Such methods need to process massive frame data, have high computational complexity, and have difficulty capturing the complex semantic logic of long videos. In addition, traditional methods rely on preset rules or fixed templates, lack flexibility, and cannot dynamically adapt to the diverse needs of users (such as open domain video question answering).

[0003] With the rise of large models, methods based on large language models (LLMs) have significant advantages in semantic understanding and dynamic reasoning. They can convert visual information into text descriptions through multimodal fusion and generate natural language answers. However, LLMs face two major bottlenecks when directly applied to long video understanding: first, the input text length is limited and it is difficult to carry all the key information of long videos; second, the mixing of redundant information and key fragments in long videos can easily lead to model reasoning deviations, and continuous retrieval of supplementary information will cause a surge in computing resource consumption.

[0004] The shortcomings of existing technologies can be summarized as follows: traditional methods are inefficient and lack dynamism, while existing LLM solutions are difficult to balance the efficiency and accuracy of long video understanding due to length limitations and redundant interference. Therefore, there is an urgent need for a method that simulates human progressive cognitive thinking to achieve efficient and accurate understanding of long videos under limited computing resources. Summary of the invention

[0005] Purpose of the invention: The technical problem to be solved by the present invention is to provide a dynamic iterative long video understanding method based on a large language model in view of the deficiencies of the prior art. Video understanding technology has a wide range of application scenarios in real life, especially in areas where key information needs to be quickly extracted from massive video data. By combining the powerful language understanding capabilities and rich prior knowledge of the large language model (LLM), the present invention can efficiently and accurately understand the content of long videos and generate answers to user questions.

[0006] The method comprises the following steps: Step 1: Based on the self-supervised dynamic iterative thinking chain, mathematical modeling and analysis of the video understanding task are carried out; Step 2: Preprocess the video input by the user, sample the video frames, generate text descriptions for each video frame and extract visual features of the video frame; perform preliminary reasoning through the question-answering agent (Agent is an abstract concept. It is usually defined as an entity that can perceive the environment and take actions in the environment to achieve goals), and generate preliminary answers based on the input text and video frames; use the cognitive adaptability evaluation mechanism to evaluate the answer, and output the answer if the cognitive adaptability meets the requirements, otherwise proceed to step 3; Step 3: A dual-agent system is used for self-supervisory information feedback. At each step of the reasoning process, a judging agent is introduced to recognize the answer. The judging agent helps confirm whether the answer is ambiguous or needs further details by detecting the consistency and accuracy of the answer. The question-answering agent will retrieve key frames and supplement information based on the feedback results, iteratively update the known information, and return to step 2 for the next round of reasoning until the cognitive adaptability reaches the preset standard, thereby ensuring the accuracy and consistency of the answer. Step 4: Evaluate the results: Use question-answering accuracy and average number of retrieved frames as evaluation indicators to conduct quantitative analysis on recognized open source datasets to verify the effectiveness of the method; secondly, use any video and question provided by the user to conduct qualitative analysis to verify whether the results meet expectations.

[0007] In step 1, the mathematical modeling and analysis of the video understanding task includes: modeling the video understanding process as a Markov process, where the state at each moment corresponds to a stage in the reasoning process, and the state at moment t Includes visual features of video frames and text description , expressed as: , Each round of reasoning will be based on the current state Generate the next moment answer , and then continue to look for key information through the existing answers. Indicates in status Under the condition of The probability of represents the state at time t+1, then the state transition process at the next moment is expressed as: , , in Indicates the state transition strategy; D represents the judging strategy, and It represents the feedback result after self-evaluation at time t+1. Represents all historical answers from time 0 to time t, and the process of self-supervised correction and learning is expressed as: , The state transfer process is further expressed as: , The entire video understanding process is broken down into questions and answers, self-evaluation, feedback and correction, and the status is continuously updated iteratively. The ultimate goal is to maximize the expected value C in the process of dynamic video understanding. The ultimate optimization goal of the entire thinking chain is: , It means finding the variable that maximizes the expected expectation C and .

[0008] Step 2 includes: Step 2-1, sample the video frames: uniformly sample the video input by the user at a rate of 1 frame per second (fps) and convert it into an image sequence; the frame rate of the video is usually high, such as 30fps or 60fps, which results in a large amount of image data being generated per second. If all frames are retained during video processing, huge computing and storage resources will be required, especially in processing tasks for long videos or large-scale video data sets, where the computational cost may be very high. By sampling at a rate of 1 fps, the number of frames that need to be processed per second is reduced, thereby significantly reducing the demand for computing resources while maintaining sufficient scene information. The sampling frequency of 1fps can capture the basic dynamics and scene changes of the video and avoid unnecessary redundant data; Step 2-2, generate text description: Use the existing Bootstrapping Language-Image Pre-training (BLIP) model to generate a text description (caption) for each frame of the image. The text description contains scenes, objects, and actions. The text description provides highly structured information for the subsequent reasoning process, which can effectively help understand the context and dynamic content of the video. Step 2-3, extracting and storing image features: extracting visual features from each frame of the image by guiding the visual encoder of the language image pre-training model BLIP, the visual features including information such as object recognition, scene understanding, and spatial relationships in the image; The visual features are saved in .npy format, providing fast access to basic data for subsequent reasoning and retrieval. This data storage format is designed to optimize processing speed and storage efficiency and support large-scale video analysis tasks; Step 2-4, perform agent reasoning and cognitive adaptability evaluation.

[0009] Step 2-4 includes: selecting 5 initial frames in a uniform distribution from the image sequence obtained in step 2-1, and the state at the initial moment Expressed as ,in Respectively represent the visual features and text description of the initial frame; from the perspective of event development, the dynamic changes can be modeled using the process of "start, development, climax, development, and ending". Based on the time series characteristics of events, considering the changes of events at different stages, the choice of using 5 frames for sampling can be combined with the characteristics of event development to ensure that the initial sampled frames can cover the different stages of the entire event development; Design a question-answering agent as the subject of reasoning and generating answers, wherein the question-answering agent includes a language understanding module and an answer generation module; The language understanding module uses GPT-4o to provide prior knowledge to the question-answering agent, providing it with powerful language understanding capabilities; The answer generation module is used to calculate the matching degree between the answer and the known information and generate the answer; The user's question and the corresponding text description are concatenated into prompt words and input into the question-answering agent. The question-answering agent generates preliminary answer options based on the prompt words and outputs the probability of selecting each option. represents the i-th option, Indicates that in the known The probability of selecting the i-th option under the condition is calculated as: , in , n represents the total number of options, and , Sim represents the cosine similarity function, exp represents the natural exponential function, and this process corresponds to the modeling process in step 1 , here the probability formula for generating the answer is specified, according to the current state Generate a probability distribution for each option, and then use the probability to determine the answer at the next moment ; LLM is used to represent the question-answering agent. The process of the question-answering agent generating answers through the option probability distribution is: , Design cognitive adaptability, taking cognitive adaptability as the expected expectation C proposed in step 1, and the cognitive adaptability assessment includes relative cognitive and absolute awareness ; Calculating relative awareness , used to reflect the distinction and difference between options: , Calculating absolute awareness , used to measure the certainty and sufficiency of information: , Calculate C: , Among them, The most likely , the second most likely ; Relative cognition measures the perception of the differences between answers. The greater the relative cognition, the greater the difference between the options, the clearer the cognition, and the higher the decision-making confidence. Absolute cognition measures the certainty of the overall information based on information entropy. The smaller the information entropy, the greater the absolute cognition, the clearer the cognition, and the higher the certainty of the current task. is a weight coefficient that indicates the importance of relative and absolute awareness in the overall evaluation, usually 0< <1; Cognitive adaptability ultimately brings together the evaluation results of both option discrimination and information sufficiency. Cognitive adaptability emphasizes the concept of "cognition" and reflects the comprehensive consideration of information sufficiency and option discrimination in the decision-making process. This is not only a quantification of model confidence, but also emphasizes the "adaptability" of the model when facing complex decisions. Maximizing cognitive adaptability will be the goal, corresponding to the expected expectation C in step 1 modeling; If the cognitive adaptability is greater than or equal to the preset threshold (0.6) or exceeds the maximum number of iterations (such as 5 times), the answer is output directly; otherwise, go to step 3.

[0010] Step 3 includes: Step 3-1, design a judgment agent for self-supervision feedback;

[0011] Step 3-2, perform dynamic iterative optimization.

[0012] Step 3 includes: Step 3-1 includes: the judging agent includes an answer checking module and a search interval feedback module; The answer checking module uses GPT-4-preview to provide prior knowledge for the answers generated in steps 2-4. Determine the consistency and correctness of the answer. This process corresponds to the modeling in step 1. ; First, for consistency, check the answer at the previous moment With the current answer Whether it is consistent; secondly, for correctness, check the currently known information Can the answer be inferred from the target question? (This relies on the prior knowledge of GPT); The search interval feedback module is used to predict the key frames in the known frames and record them as known key frames. Based on this, the feedback video search interval is given, and a text description of the expected search and supplementary information is provided: feedback is given based on the result of the answer check module. If it is consistent and correct, it is only necessary to take a specific number of frames Y1 before and after the current known key frame for key frame search. Here, the specific number of frames Y1 is calculated as: Y1= Y2 / (Y3-1), Among them, Y2 represents the total number of video frames, and Y3 represents the number of known frames; Special case description: When the known key frame is the first frame, only search backward; when the known key frame is the last frame, only search forward. This process is similar to humans slowly dragging the video progress bar, the purpose is to refine and supplement the current known information. If it is inconsistent or incorrect, it is necessary to expand the search range to the entire video frame interval, similar to the behavior of humans dragging the video progress bar significantly. The dual-agent verification mechanism realizes self-supervisory feedback, which can greatly reduce the cost of manual labeling.

[0013] Step 3 includes: Step 3-2 includes: according to the feedback video search interval and text description given by the judging agent in step 3-1, key frame retrieval and information supplement are performed to dynamically adjust the state , corresponding to the state transition process modeled in step 1 , so far, the formula of dynamic iterative optimization is rewritten as: ; Strategy The implementation process includes: based on the feedback video search interval and text description, the visual features of all frames in the feedback video search interval are retrieved from the .npy format file obtained in step 2-3, and the frames with the highest similarity between the visual features and the text description features in the feedback video search interval are selected as supplements, and the evaluation agent is set to give m feedback video search intervals. , the formula is: , in Indicates the mth feedback video search interval The key frames retrieved from Indicates the mth feedback video search interval The visual features of the jth image in , It is a text description feature; The visual features of the retrieved supplementary frame are integrated into the existing information to obtain updated information. At the same time, the description corresponding to the supplementary frame is also updated. Keyframes for supplementation The corresponding text description, where Indicates the mth key frame The corresponding text description, the specific update formula is: , , Where concat represents the connection operation;

[0014] After the update is completed, return to steps 2-4 to enter the next round of reasoning and iteration, regenerate the answer and evaluate the cognitive adaptability until the output conditions are met.

[0015] Step 4 includes: The EgoSchema_subset dataset is used, and the evaluation indicators include: Average number of retrieved frames: This metric measures the average number of frames required to generate the answer to each question when the method is tested on EgoSchema_subset, and is used to evaluate computational efficiency. This metric reflects the efficiency of the method in understanding and processing video content. The fewer the number of retrieved frames, the stronger the ability of the method to obtain sufficient information in a short time. Question and answer accuracy : It is used to determine whether the model output answer is the same as the labeled answer. The formula is: , TP (True Positive) indicates the number of generated answers that completely match the annotated answers, and FP (False Positive) indicates the number of generated answers that do not match the annotated answers. The higher the question-answering accuracy, the better the method is at generating correct answers. In addition, to further verify the effectiveness of the method, we also use arbitrary videos and questions provided by users for qualitative analysis. By comparing the output answers with the user's expected results, we can determine whether the performance of the method in the actual application scenario meets expectations.

[0016] The present invention also provides an electronic device, comprising a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor executes the steps of the described method.

[0017] The present invention also provides a storage medium storing a computer program or instruction. When the computer program or instruction is run on a computer, the steps of the method described are executed.

[0018] The method of the present invention can be applied to the following fields: Video Q&A system: Based on the questions raised by users, it can quickly locate and extract relevant information from long videos and generate accurate answers. For example, in the field of education, students can get explanations of specific knowledge points from teaching videos by asking questions; in the medical field, doctors can quickly retrieve key steps in surgical videos through video Q&A systems.

[0019] Video summary generation: By understanding the video content, a concise video summary is automatically generated to help users quickly understand the core information of the video. For example, in the news field, a summary of a news video can be automatically generated; in corporate training, the key content of a training video can be generated.

[0020] Video content retrieval: Locate specific clips from long videos based on user needs. For example, in film and television production, specific scenes or dialogues can be quickly located.

[0021] Intelligent monitoring and security: Automatically identify abnormal behaviors or events through real-time analysis of surveillance video content. For example, in the surveillance system of public places, video understanding technology can be used to quickly retrieve clips of abnormal events and automatically detect suspicious behaviors or safety hazards.

[0022] Personalized recommendation: Generate personalized video recommendations by understanding users’ video viewing behaviors and preferences. For example, on a video platform, relevant video clips or full videos can be recommended based on the user’s historical viewing history and questions.

[0023] The present invention has the following beneficial effects: 1) Through dynamic sampling and cognitive adaptability evaluation, redundant information processing is reduced and the LLM input length limitation is broken; 2) Combined with the dual-agent self-supervisory feedback mechanism, the accuracy of the answer is significantly improved while reducing manual labeling; 3) Iterative retrieval only supplements information when necessary, optimizing computing resource consumption; 4) It can be extended to multimodal tasks such as video question answering and summary generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flow chart of the method of the present invention.

[0025] Figure 2 It is an example diagram of the method of the present invention.

[0026] Figure 3 yes Figure 2 A clear schematic diagram of all sampled video frames in . DETAILED DESCRIPTION

[0027] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0028] like Figure 1 As shown, an embodiment of the present invention provides a dynamic iterative long video understanding method based on a large language model, including: Step 1: Mathematical modeling and analysis. We learn from the human behavior of dragging the progress bar to quickly locate the segment of interest when watching a video, analyze the logical thinking chain of humans in dynamically adjusting, finding key segments and finally understanding the video content, and model it from a mathematical perspective. This modeling method can more accurately simulate human thinking and rapid understanding mechanisms, making the machine more adaptable and efficient in understanding videos.

[0029] The video understanding process is modeled as a Markov process, where the state at each moment Corresponding to a stage in the reasoning process, the state at each moment Composed of multiple pieces of information, including visual features of video frames , text description , so the state is expressed as follows: , Each round of reasoning will be based on the current state Generate the next moment answer , and then continue to look for key information through the existing answers. Indicates in status Under the condition of The probability of represents the state at time t+1, then the state transition process at the next moment can be expressed as: , , in Indicates the state transition strategy; Considering that in the process of searching for key video clips, humans will make self-judgments and adjustments based on their own potential knowledge on whether to find the right or wrong video clips, D is used to represent the judgment strategy, and It represents the feedback after self-evaluation at time t+1. represents all historical answers from time 0 to time t. The process of self-supervised correction and learning is expressed as follows: , So the state transfer process can be further expressed as: , The entire video understanding process is broken down into questions and answers, self-evaluation, feedback and correction, and the status is continuously updated iteratively. The ultimate goal is to maximize the expected value C in the process of dynamic video understanding. Combining the above processes, the ultimate optimization goal of the entire thinking chain is: ,

[0030] The system continuously updates its status at different stages, gradually optimizing its understanding of the video content. This method simulates the cognitive reasoning chain of humans when watching videos, enabling the system to efficiently locate and parse key segments within limited resources.

[0031] Step 2: Data preprocessing, execution reasoning and cognitive adaptability evaluation.

[0032] Step 2-1, video frame sampling: The input video is uniformly sampled at a rate of 1 frame per second (fps) and converted into an image sequence. The frame rate of the video is usually high, such as 30fps or 60fps, which results in a large amount of image data per second. If all frames are retained during video processing, huge computing and storage resources will be required, especially in the processing tasks of long videos or large-scale video datasets, the computational cost may be very high. By sampling at a rate of 1 fps, the number of frames that need to be processed per second is reduced, thereby significantly reducing the demand for computing resources while maintaining sufficient scene information. The sampling frequency of 1fps can capture the basic dynamics and scene changes of the video and avoid unnecessary redundant data. In summary, the sampling method of 1 fps achieves a reasonable balance between computing cost, storage overhead, and information retention, ensuring that the system can efficiently process large-scale video data while retaining sufficient timing information to support subsequent analysis tasks; Step 2-2, text description generation: Use the existing Bootstrapping Language-Image Pre-training (BLIP) model to generate a text description (caption) for each frame of the image. The content of the text description includes scenes, objects, and actions. The text description provides highly structured information for the subsequent reasoning process, which can effectively help understand the context and dynamic content of the video. The BLIP model is trained by combining visual and language features to automatically generate natural language descriptions that match the video content, so that the machine not only relies on visual features when analyzing videos, but also uses text to enhance the understanding of scenes and events. Compared with traditional methods that only use visual features, this text enhancement method has two major advantages: 1) Enrich the semantic information of the video: The text description can extract the core events, object relationships, and timing information in the video to help the reasoning module understand the scene development more accurately, 2) Improve the utilization of multimodal information, and combine multimodal information such as vision, text, and contextual information to make the understanding of the video content more comprehensive; Step 2-3, image feature extraction and storage: The visual encoder of the BLIP model is used to extract visual features from each frame of the image. These features include but are not limited to information such as object recognition, scene understanding, and spatial relationships in the image. The extracted visual features will be saved in the efficient .npy format to provide basic data for fast access for subsequent reasoning and retrieval. This data storage format is designed to optimize processing speed and storage efficiency and support large-scale video analysis tasks. Compared with traditional image storage methods, the .npy format has the following advantages: 1) Fast access and loading: The binary storage method enables data to be directly mapped to memory, improving the efficiency of model reasoning and retrieval, 2) Storage optimization: Supports efficient array storage and compression strategies, reduces data redundancy, and takes up less disk space, 3) Strong compatibility: Deeply integrated with the NumPy ecosystem, it facilitates seamless docking with a variety of deep learning frameworks (such as PyTorch); Step 2-4, perform agent reasoning and cognitive adaptability evaluation: select 5 initial frames from the image sequence prepared in step 2-1 according to uniform distribution, and the mathematical representation of the initial state is recorded as ,in They are the visual features and text description of the initial frame respectively. From the perspective of event development, the process of "start, development, climax, development, and ending" can be used to model its dynamic changes. Based on the time series characteristics of events, considering the changes of events at different stages, the choice of using 5 frames for sampling can be combined with the characteristics of event development to ensure that the initial sampled frames can cover the different stages of the entire event development. The sampling points include important moments without being overly concentrated in a certain area. If too many frames are initialized, it will lead to excessive resource consumption and the length of the input content exceeds the limit. If too few frames are initialized, the number of iterations and the difficulty of frame search will increase, affecting the efficiency and accuracy of reasoning; In the process of reasoning and answer generation, a question-answering agent is designed as the core reasoning subject, which is responsible for understanding user questions and generating reasonable answers. The question-answering agent includes a language understanding module and an answer generation module. The language understanding module uses GPT-4o to provide the question-answering agent with prior knowledge and powerful language understanding capabilities. The answer generation module calculates the degree of match between the candidate answers and the known information (visual features and text descriptions), and generates answers based on the calculation results. The user's question and the corresponding text description are concatenated into prompt words and input into the question-answering agent. The question-answering agent generates preliminary answer options based on the prompt words, and outputs the probability of selecting each option. represents the i-th option, Indicates that in the known The probability of choosing the i-th option under the condition of , n means there are n options in total and , The calculation formula is as follows: , Where Sim represents cosine similarity, which corresponds to the modeling process in step 1. . Taking into account the impact of visual features and text descriptions on answer selection, the formula gives the same weight to the similarity of the two. This design shows that the visual information and text description of the video are equally important in answer selection, ensuring the balanced use of visual and text information. At the same time, the similarity calculation of all candidate answers is normalized to ensure that the sum of the probabilities of all options is 1. Normalization is to make the answer selection follow the probability distribution and avoid excessive bias in a certain answer. It calculates the similarity of each candidate answer and converts it into a probability value, thereby effectively helping the question-answering agent generate the answer that best fits the context.

[0033] LLM is used to represent the question-answering agent. The process of the question-answering agent generating answers through the option probability distribution is: , Design cognitive adaptability and take cognitive adaptability as the expected expectation C proposed in step 1. Cognitive adaptability assessment includes relative cognitive and absolute awareness ,remember The most likely , the second most likely , the formula and definition of cognitive adaptability are: Relative awareness: reflects the distinction and difference between options: , Absolute awareness: measures the certainty and adequacy of information: , Cognitive adaptability: , Relative cognition measures the perception of the difference between answers. The greater the relative cognition, the greater the difference between the options, the clearer the cognition, and the higher the decision confidence. Absolute cognition measures the certainty of the overall information based on information entropy. The smaller the information entropy, the greater the absolute cognition, the clearer the cognition, and the higher the certainty of the current task. α is a weight coefficient, which indicates the importance of relative cognition and absolute cognition in the overall evaluation. It is set to 0.5 here, that is, the role of relative cognition and absolute cognition in the final cognitive adaptability is equal. Cognitive adaptability ultimately brings together the evaluation results of option discrimination and information sufficiency. Cognitive adaptability emphasizes the concept of "cognition" and reflects the comprehensive consideration of information sufficiency and option discrimination in the decision-making process. This is not only a quantification of model confidence, but also emphasizes the "adaptability" of the model when facing complex decisions. Maximizing cognitive adaptability will be the goal, corresponding to the expected expectation C in step 1 modeling; If the cognitive adaptability is greater than or equal to the preset threshold (0.6) or exceeds the maximum number of iterations (such as 5 times), the answer is directly output; otherwise, it goes to step 3. In this way, the system avoids falling into a long-term iteration process in order to maximize the target expectation, which brings a lot of unnecessary calculations and ensures the efficiency of the reasoning process.

[0034] Step 3: Self-supervised feedback and dynamic iterative optimization.

[0035] Step 3-1, self-supervised feedback: Design a judge agent for self-supervised feedback, use GPT-4-preview to provide the judge agent with prior knowledge, and give the answer generated in step 2-4 Determine the consistency and correctness of the answer and give feedback on the search interval and the text description of the expected search and supplementary information, corresponding to the modeled in step 1 First, for consistency, check the last answer With the current answer Are they consistent? If they are consistent, it means that the model's understanding of the problem and reasoning direction are relatively stable, and the reasoning is highly coherent. At this time, the system will take a specific number of frames Y1 before and after the key frame in the current known frame to search for the key frame. The specific number of frames Y1 is calculated as: Y1= Y2 / (Y3-1), Further refine the current reasoning path, similar to how humans slowly drag the progress bar when watching a video, in order to obtain more specific content. The more key frames are obtained, the more detailed the search will be; secondly, for correctness, the system will use the currently known information to determine the correctness of the search. and the target question, combined with the prior knowledge provided by GPT-4-preview, to verify the current answer , whether it is consistent with the question and existing information. If the current answer is verified to be correct, the system will continue to reason on the current path and further supplement the known information; if the answer is inconsistent or wrong, the system will expand the search range and turn to the entire video frame interval for substantial adjustments, similar to when humans encounter problems in the reasoning process, they drag the video progress bar significantly to view other related clips in order to obtain more information to correct the reasoning path. The whole process adopts a dual-agent verification mechanism, that is, one agent is responsible for generating and reasoning answers, and the other agent is responsible for evaluating the correctness and consistency of the answers and giving feedback. This mutual verification between the two agents enables the model to be self-corrected after each round of reasoning, thereby improving the accuracy and confidence of reasoning. Compared with traditional manual labeling and supervised learning methods, systems that use self-supervised feedback and dual-agent verification mechanisms can significantly reduce reliance on manual labeling and reduce labor costs. In addition, the system can adaptively adjust the reasoning process based on feedback, thereby improving the accuracy and consistency of reasoning, avoiding errors caused by manual labeling deviations or deficiencies in traditional methods, and improving overall efficiency and decision-making quality; Step 3-2, dynamic iterative optimization: According to the feedback video search interval and text description given by the evaluation agent in step 3-1, key frame retrieval and information supplement are performed to dynamically adjust the state , corresponding to the state transition process modeled in step 1 , so far, the formula of dynamic iterative optimization is rewritten as follows: ; Strategy The implementation process is as follows: Based on the feedback video search interval and text description, the visual features of all frames in the feedback video search interval are retrieved from the .npy format file preprocessed and stored in step 2-3, and the frame with the highest similarity between the visual features and the text description features in the feedback video search interval is selected as a supplement. Assume that the judging agent gives m feedback video search intervals. , the formula is as follows: , in Indicates the feedback video search interval The key frames retrieved from Indicates the feedback video search interval The visual features of the jth image in , It is a text description feature; The visual features of the retrieved supplementary frame are integrated into the existing information to obtain updated information. At the same time, the corresponding description of the supplementary frame is also updated. Keyframes for supplementation The corresponding text description and specific formula are as follows: , , After the update is completed, return to steps 2-4 to enter the next round of reasoning and iteration, regenerate the answer and evaluate the cognitive adaptability until the output condition is met. Compared with directly inputting the entire video into the large model for reasoning, the iterative optimization method has significant advantages. First, directly inputting the entire video may lead to a waste of computing resources and a reduction in the speed of reasoning, because the video data volume is large and contains a lot of redundant information. The iterative method gradually refines the information processing and only updates the key frames and related descriptions in each round, reducing the amount of input data required for each reasoning, thereby improving efficiency and saving computing resources. In addition, many current large language models (LLMs) have input length limitations when processing long texts or large-scale data, and cannot process the entire video content at one time. The iterative method effectively avoids this problem by inputting relevant video clips in stages to ensure that the input data does not exceed the input length limit of the model during each round of reasoning. This method can not only focus on the key information related to the current task and improve the accuracy and precision of reasoning, but also avoid the information loss caused by the limited input length, thereby optimizing the performance of the model and the quality of decision-making.

[0036] Step 4, result evaluation. First, we use the recognized open source EgoSchema_subset dataset, which focuses on complex question-answering tasks of long videos (average length of 3 minutes), covering a variety of scenarios such as daily life, academic lectures, documentaries, etc. It also contains various types of question classifications, such as causal reasoning, temporal reasoning, object recognition, action recognition, etc., to test the performance of the model in diverse reasoning tasks. There are about 5,00 question-answer pairs in total, and all are multiple-choice questions. In order to evaluate the performance of the model, multiple evaluation indicators are used, including the average number of retrieved frames and question-answering accuracy. Average number of retrieved frames: This indicator measures the average number of frames required to retrieve to generate the answer to each question when the method is tested on EgoSchema_subset, and is used to evaluate computational efficiency. This indicator reflects the efficiency of the method in understanding and processing video content. The fewer the number of retrieved frames, the stronger the ability of the method to obtain sufficient information in a short time; Question and answer accuracy : It is used to determine whether the model output answer is the same as the labeled answer. The formula is: , Among them, TP (True Positive) indicates the number of generated answers that completely match the annotated answers, and FP (False Positive) indicates the number of generated answers that do not match the annotated answers. The higher the question-answering accuracy, the stronger the method's ability to generate correct answers. The results show that when dealing with complex question-answering questions, the method only needs to retrieve an average of 7.7 frames to achieve an accuracy of 65% on the EgoSchema_subset dataset. This result shows that the model can effectively generate accurate answers while ensuring low computational costs and fewer frame retrievals, demonstrating its efficient reasoning ability and excellent performance. In contrast, many existing methods often need to process dozens or even hundreds of frames to achieve similar accuracy, with low computational efficiency and performance. This also proves the efficiency and accuracy of this method when facing long videos and complex questions.

[0037] In addition, to further verify the effectiveness of the method, any videos and questions provided by the user can be used for qualitative analysis. By comparing the output answers with the user's expected results, it can be determined whether the performance of the method in the actual application scenario meets expectations. This qualitative analysis not only helps to verify the model's ability to process real-world video data, but also reveals the model's performance on different types of problems (such as causal reasoning, action recognition, object recognition, etc.). By comparing with the user's actual application needs, the model can be further optimized and adjusted to ensure its good adaptability and accuracy in complex environments.

[0038] like Figure 2 As shown, an example of this embodiment includes: the user gives a video and asks the question "What is the dog playing with at the beginning?" in the form of a multiple-choice question. Figure 3As shown, all video frames involved in the example of this embodiment include: frame 1, frame 13, frame 50 and frame 100, where frame 1, frame 50 and frame 100 are initial frames, and frame 13 is a key frame obtained after retrieval (marked in red). First, 100 frames are sampled at 1fps for a given video, and the BLIP model is used to extract the visual features of the video frames and generate text descriptions of the video frames. During initialization, the 1st, 50th and 100th frames are sampled as initial information, and the initial information includes visual features and text descriptions. Then, the question-answering agent performs reasoning and calculation based on this information and question options (A ice, B water, C bag), and gives answers according to the probability distribution of each option. The cognitive adaptability evaluation module calculates cognitive adaptability according to the formula. During the first evaluation, the cognitive adaptability is 0.31, which is lower than the threshold of 0.6, and the result cannot be output, and the next step is entered. Then, the judging agent intervenes and determines that the answer is not accurate enough, pointing out that the current answer does not match the "at the beginning" in the question, and relying on the powerful prior knowledge of the large language model, predicts that the answer may appear in the interval of 1 to 50 frames, and the target description is "the dog is playing with things". Based on this feedback, the question-answering agent retrieves the 13th frame (whose text description is "the dog is playing with a bag") through the similarity-based information retrieval mechanism. At this point, the first round of reasoning ends. Iterate and enter the second round of reasoning again. ( Figure 3 In the figure, the first round of reasoning path is indicated by blue arrows, and the second round of reasoning is indicated by orange arrows). First, the state of known information is updated and the retrieved key frames are added. The question-answering agent gives the probability distribution and answer again based on the current information. In this round of cognitive adaptability evaluation, the cognitive adaptability is 0.61, which exceeds the threshold of 0.6, meets the iteration termination condition, and finally outputs the answer C. In the whole process, only 4 frames of video frames are used, which greatly reduces the consumption of computing resources and improves the reasoning efficiency. Through multi-module collaboration, this method continuously optimizes the understanding of long video content and the accuracy of answers in a dynamic iterative manner, achieving efficient and accurate long video question-answering reasoning.

[0039] The present invention provides a dynamic iterative long video understanding method based on a large language model. There are many methods and ways to implement the technical solution. The above is only a preferred implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the scope of protection of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A dynamic iterative long video understanding method based on a large language model, characterized in that: The following steps are involved: Step 1: Based on the self-supervised dynamic iterative thinking chain, mathematical modeling and analysis of the video understanding task are carried out; Step 2: Preprocess the video input by the user, sample the video frames, generate text descriptions for each video frame and extract visual features of the video frame; perform preliminary reasoning through the question-answering agent, and generate preliminary answers based on the input text and video frames; use the cognitive adaptability evaluation mechanism to evaluate the answer, and output the answer if the cognitive adaptability meets the requirements, otherwise proceed to step 3; Step 3: self-supervisory information feedback. In each step of the reasoning process, a judging agent is introduced to recognize the answer. The judging agent helps to confirm whether the answer is ambiguous or needs further details by detecting the consistency and accuracy of the answer. The question-answering agent will retrieve key frames and supplement information based on the feedback results, iteratively update the known information, and return to step 2 for the next round of reasoning until the cognitive adaptability reaches the preset standard. Step 4: Evaluate the results: Use the question-answering accuracy and average number of retrieval frames as evaluation indicators to conduct quantitative analysis on open source datasets to verify the effectiveness of the method. Secondly, any video and questions provided by the user are used for qualitative analysis to verify whether the results meet expectations.

2. The method according to claim 1, characterized in that In step 1, the mathematical modeling and analysis of the video understanding task includes: modeling the video understanding process as a Markov process, where the state at each moment corresponds to a stage in the reasoning process, and the state at moment t Includes visual features of video frames and text description , expressed as: ， Each round of reasoning will be based on the current state Generate the next moment answer , and then continue to look for key information through the existing answers. Indicates in status Under the condition of The probability of represents the state at time t+1, then the state transition process at the next moment is expressed as: ， ， in Indicates the state transition strategy; D represents the judging strategy, and It represents the feedback result after self-evaluation at time t+1. Represents all historical answers from time 0 to time t, and the process of self-supervised correction and learning is expressed as: ， The state transfer process is further expressed as: ， The entire video understanding process is broken down into questions and answers, self-evaluation, feedback and correction, and the status is continuously updated iteratively. The ultimate goal is to maximize the expected value C in the process of dynamic video understanding. The ultimate optimization goal of the entire thinking chain is: ， It means finding the variable that maximizes the expected expectation C and .

3. The method according to claim 2, characterized in that Step 2 includes: Step 2-1, sampling video frames: uniformly sampling the video input by the user and converting it into an image sequence; Step 2-2, generate text description: Use the guided language image pre-training model BLIP to generate a text description for each frame of the image. The content of the text description includes scenes, objects, and actions; Step 2-3, extracting and storing image features: extracting visual features from each frame of the image by guiding the visual encoder of the language image pre-training model BLIP; the visual features are saved in .npy format; Step 2-4, perform agent reasoning and cognitive adaptability evaluation.

4. The method according to claim 3, characterized in that Step 2-4 includes: selecting an initial frame in the image sequence obtained in step 2-1 according to a uniform distribution, and the state at the initial moment Expressed as ,in Represent the visual features and text description of the initial frame respectively; Design a question-answering agent as the subject of reasoning and generating answers, wherein the question-answering agent includes a language understanding module and an answer generation module; The language understanding module uses GPT-4o to provide prior knowledge for the question-answering agent; The answer generation module is used to calculate the matching degree between the answer and the known information and generate the answer; The user's question and the corresponding text description are concatenated into prompt words and input into the question-answering agent. The question-answering agent generates preliminary answer options based on the prompt words and outputs the probability of selecting each option. represents the i-th option, Indicates that in the known The probability of selecting the i-th option under the condition is calculated as: ， in , n represents the total number of options, and , Sim represents the cosine similarity function, exp represents the natural exponential function; LLM is used to represent the question-answering agent. The process of the question-answering agent generating answers through the option probability distribution is: ， Design cognitive adaptability, taking cognitive adaptability as the expected expectation C proposed in step 1, and the cognitive adaptability assessment includes relative cognitive and absolute awareness ; Calculating relative awareness , used to reflect the distinction and difference between options: ， Calculating absolute awareness , used to measure the certainty and sufficiency of information: ， Calculate C: ， Among them, The most likely , the second most likely ; is a weight coefficient; If the cognitive adaptability is greater than or equal to the preset threshold or exceeds the maximum number of iterations, the answer is directly output; otherwise, go to step 3.

5. The method according to claim 4, characterized in that Step 3 includes: Step 3-1, design a judgment agent for self-supervision feedback; Step 3-2, perform dynamic iterative optimization.

6. The method according to claim 5, characterized in that Step 3 includes: Step 3-1 includes: the judging agent includes an answer checking module and a search interval feedback module; The answer checking module uses GPT-4-preview to provide prior knowledge for the answers generated in steps 2-4. To determine the consistency and correctness of the answer, first, for consistency, check the answer at the previous moment. With the current answer Whether it is consistent; secondly, for correctness, check the currently known information Can the answer be inferred from the target question? ; The search interval feedback module is used to predict the key frames in the known frames and record them as known key frames, give feedback video search intervals, and provide text descriptions of the expected search and supplementary information: feedback is given based on the results of the answer check module. If they are consistent and correct, it is only necessary to take the number of frames Y1 before and after the current known key frame for key frame search. The calculation formula of the number of frames Y1 is: Y1= Y2 / (Y3-1), Among them, Y2 represents the total number of video frames, and Y3 represents the number of known frames; When the known key frame is the first frame, only search backward; when the known key frame is the last frame, only search forward.

7. The method according to claim 6, characterized in that Step 3 includes: Step 3-2 includes: according to the feedback video search interval and text description given by the judging agent in step 3-1, key frame retrieval and information supplement are performed to dynamically adjust the state , corresponding to the state transition process modeled in step 1 , so far, the formula of dynamic iterative optimization is rewritten as: ， Strategy The implementation process includes: based on the feedback video search interval and text description, the visual features of all frames in the feedback video search interval are retrieved from the .npy format file obtained in step 2-3, and the frames with the highest similarity between the visual features and the text description features in the feedback video search interval are selected as supplements, and the evaluation agent is set to give m feedback video search intervals. , the formula is: ， in Indicates the mth feedback video search interval The key frames retrieved from Indicates the mth feedback video search interval The visual features of the jth image in , It is a text description feature; The visual features of the retrieved supplementary frame are integrated into the existing information to obtain updated information. At the same time, the description corresponding to the supplementary frame is also updated. Keyframes for supplementation The corresponding text description, where Indicates the mth key frame The corresponding text description, the specific update formula is: ， ， Where concat represents the connection operation; After the update is completed, return to steps 2-4 to enter the next round of reasoning and iteration, regenerate the answer and evaluate the cognitive adaptability until the output conditions are met.

8. The method according to claim 7, characterized in that Step 4 includes: The EgoSchema_subset dataset is used, and the evaluation indicators include the average number of retrieval frames and question-answering accuracy. .

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores program codes, and when the program codes are executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 8.

10. A storage medium, characterized in that: A computer program or instruction is stored, and when the computer program or instruction is run on a computer, the steps of the method according to any one of claims 1 to 8 are executed.

Images