Multi-video joint understanding system for video language large model

Abstract

A multi-video joint understanding system for a video language large model comprises a video structuring module, a graph fusion module, a visual mapper, a visual encoder and a word embedding layer, through a structured video representation and graph fusion mechanism, invalid input of redundant visual information is avoided, the model input load is reduced, and efficient data utilization is achieved; through integration of cross-video spatio-temporal information, the problem that single-video knowledge is incomplete is effectively relieved, illusion answers are reduced, the processing capacity of the model for complex tasks is improved, and the method can be applied to multiple artificial intelligence application fields such as multi-video questions and answers, multi-video content understanding, multi-video abstraction and cross-video event analysis.

Description

Technical Field

[0001] The present invention relates to a technology in the field of neural networks, specifically a multi-video joint understanding system for a large video language model. Background Art

[0002] With the rapid development of large language models, large video language models have demonstrated excellent performance in video understanding tasks. However, existing large video language models mainly rely on single video input for reasoning. Due to the incompleteness of information in the spatial and temporal dimensions of a single video, the model produces problems such as hallucinatory answers and knowledge omissions. Factors affecting the accuracy of single video understanding include but are not limited to: sparse video sampling, object occlusion, perspective switching, and bias from a single information source. In addition, the model's ability to understand complex video content is also constrained by factors such as the length of the video context and the redundancy of visual representations. Some current studies attempt to compensate for the lack of single video information through multi-video data augmentation. However, directly inputting the visual information of multiple videos into a large model will introduce a large amount of redundant and noisy information. Ultra-long input sequences will also distract the model, thereby reducing reasoning efficiency and accuracy. Summary of the Invention

[0003] In response to the problem that existing technologies have certain redundancy and noise interference in problems involving complex tasks or cross-video semantic associations, resulting in the lack of accuracy of the generated text descriptions, the present invention proposes a multi-video joint understanding system for large video language models. Through structured video representation and graph fusion mechanism, the invalid input of redundant visual information is avoided, the model input load is reduced, and efficient data utilization is achieved; through the integration of cross-video spatiotemporal information, the problem of incomplete knowledge in a single video is effectively alleviated, hallucinatory answers are reduced, and the model's processing ability for complex tasks is improved. It can be applied to various artificial intelligence application fields such as multi-video question and answer, multi-video content understanding, multi-video summarization, and cross-video event analysis.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a multi-video joint understanding system for a large video language model, comprising: a video structuring module, a graph fusion module, a visual mapper, a visual encoder, and a word embedding layer, wherein: the video structuring module extracts video key frames to obtain a detailed description text of the video, obtains a text scene graph consisting of targets and interaction relationships through text-level analysis, and detects and extracts the visual features corresponding to the targets in the video frames, thereby constructing a complete video spatiotemporal structured representation; the graph fusion module enhances the video structured representation through cross-video knowledge fusion; the visual encoder performs fine-grained feature encoding processing based on the video information of the target video to obtain an optimized video representation; the visual mapper performs cross-modal feature alignment processing based on the video representation provided by the visual encoder to obtain semantically consistent fusion features; the word embedding layer splices the graph features after fusion of related videos, the original visual features corresponding to the target video, and the question text according to a preset multi-video structured prompt word template, and then inputs the large language model to generate a final answer.

[0006] The video structuring module includes: an event detector, a video descriptor, a scene graph parser, a visual locator and a spatiotemporal associator, wherein: the event detector uses a lightweight scene detection algorithm to segment the video time sequence scene according to the video to be processed, and obtains the middle frame of each scene as the key frame; the video descriptor uses a pre-trained video language model to perform a fine-grained description of the key frame based on the detailed description text of the video to be processed, and generates a semantic text covering the target, action and environment; the scene graph parser uses the scene graph parser to extract the <subject-predicate-object> triple from the description text to obtain a text scene composed of targets and interaction relationships Figure, the visual locator uses a text-based detector to locate the spatial position of the target in the triplet in the key frame, and extracts the visual features of the target area; at the same time, it establishes cross-frame target ID tags through the video target tracking algorithm to achieve the temporal continuity representation of the same target and obtain the corresponding visual features of the target in the video frame. The spatiotemporal associator constructs the interaction edges between targets in the same frame according to the predicate relationship in the text scene graph, and establishes the temporal association edges of the targets across frames by tracking the cross-frame target ID tags, finally forming a structured representation of the spatiotemporal graph that includes target features as nodes, interaction relationships as spatial edges and temporal associations as temporal edges.

[0007] The graph fusion module includes: several graph attention networks and graph mappers based on the cross-graph attention mechanism connected in series, wherein: the graph attention network adopts a hierarchical graph attention network to learn the context-aware features of the single video graph structure based on the out-of-control structured representation; the graph mapper adopts a multi-layer perceptron to map the graph feature dimension to the same dimension as the visual and text features to achieve cross-modal feature alignment.

[0008] The cross-graph attention mechanism uses the target video graph features as queries and the related video graph features as keys and values. Cross-graph attention (CGA) calculates weights and adaptively aggregates complementary information. A learnable parameter (CE) is also introduced to distinguish target and related video features to avoid information confusion. Technical Effects

[0009] Based on the video structuring module, the present invention constructs a video spatiotemporal graph structured representation through event detection, scene graph analysis, target positioning and spatiotemporal association, comprehensively analyzes the key targets and their interactive relationships in the video, and solves the problem of incomplete information in a single video. It introduces a cross-video graph fusion module, which is based on the cross-graph attention mechanism, adaptively aggregates the complementary information of related videos, and introduces learnable parameters to distinguish targets from related video features to avoid information confusion. It realizes the aligned input of cross-graph features, video features and word feature modalities, and combines the input of multi-video structured prompt word templates into a large language model to improve the effective fusion and reasoning capabilities of multimodal information. Compared with the existing technology, the present invention realizes efficient information integration in the process of joint understanding of multiple videos, avoids the introduction of redundant information and noise, and significantly improves the ability of large models to understand complex video content; through structured video representation and graph fusion mechanism, it effectively alleviates the problems of hallucinatory answers and knowledge omissions in single video understanding, and improves the accuracy and efficiency of model reasoning of multiple videos; supports artificial intelligence application scenarios such as multi-video question and answer, multi-video summarization, and cross-video event analysis, expands the scope of application of large video language models, and significantly enhances their performance in complex tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a system structure diagram of the present invention;

[0011] Figure 2 Schematic diagram of video structuring module;

[0012] Figure 3 This is a schematic diagram of the graph fusion module;

[0013] Figure 4 Schematic diagram of the embodiment effect. DETAILED DESCRIPTION

[0014] like Figure 1As shown, this embodiment relates to a multi-video joint understanding system for a large video language model, including: a video structuring module, a graph fusion module, a visual mapper, a visual encoder, and a word embedding layer. The video structuring module extracts complex video information into an efficient spatiotemporal structured representation. The graph fusion module then effectively integrates the effective information of multiple related video structured representations, thereby obtaining more reliable and accurate video understanding results. In the figure, the text G represents the graph structure features corresponding to the related videos, the text V represents the video visual features corresponding to the main video, and the text T represents the word element features corresponding to the user question.

[0015] like Figure 2 As shown, the video structuring module includes: an event detector, a video descriptor, a scene graph parser, a visual locator and a spatiotemporal associator.

[0016] The event detector uses a lightweight scene detection algorithm to segment videos into time series, using the intermediate frames of the event as video keyframes. Through algorithmic optimization, this module can quickly and efficiently complete the segmentation of video events, ensuring that the selected keyframes are representative, providing high-quality input for subsequent video description and scene analysis.

[0017] The video description module uses a pre-trained video language model to generate detailed video content description text based on extracted keyframes, covering multiple dimensions such as objects, actions, and environments. This module further optimizes the generated text using a semantic enhancement strategy, ensuring high accuracy and expressiveness, providing high-quality semantic input for scene graph parsing.

[0018] The scene graph parser extracts <subject, interaction, object> triplets from video description text to generate a textual scene graph composed of multiple triplets. Combining semantic structure analysis with relational modeling techniques, this module efficiently extracts semantic information from the description text and generates a logically consistent and semantically clear scene graph, providing reliable semantic support for the structured representation of video content.

[0019] The visual localizer determines the spatial position of the target in keyframes based on its textual description, generates a bounding box, and extracts target-level features. This module uses multimodal alignment techniques to crop and process the target region. Combined with a video object tracking algorithm, it generates cross-frame ID tags for the target, thereby achieving a temporal and spatial continuity representation of the target across frames and videos, providing precise visual information input for constructing a spatiotemporal structured representation of the video.

[0020] The spatiotemporal associator constructs spatial edges of target nodes based on the interaction relationship between targets obtained by the scene graph parser, and then constructs temporal edges of target nodes across frames based on the target tracking ID tags obtained by the visual locator, thereby constructing a spatiotemporal structured representation of the video.

[0021] like Figure 3 As shown, the graph fusion module includes: several serially connected graph attention networks and graph mappers based on the cross-graph attention mechanism.

[0022] The graph attention network inputs the spatiotemporal structured representation obtained by the video structuring module into GAT ( P, Cucurulll G, Casanova A, et al. Graph attention networks[J].arXivpreprint arXiv:1710.10903, 2017.), fuses target node features according to the graph structure, and thus models the interaction and temporal relationships between targets.

[0023] The cross-graph attention mechanism specifically includes:

[0024] i) Take the target video X tar Compute the query, i.e. the target video structured representation Q after cross-graph attention fusion tar = X tar W Q , where: key K tar =X tar W K , value V tar =X tar W V ;

[0025] ii) Use related video X rel Compute the relevant video structured representation Q rel =X rel W Q , where: key K rel =X rel W K , and the value V rel =X rel W V ;

[0026] iii) In order to enable the model to better distinguish target videos from related videos, category embedding CE is introduced, specifically: CE tar =σ(α),CE rel =1-σ(α), where: α is a learnable parameter shared by multiple frames;

[0027] iv) Add the category embedding to the query, key, and value of the attention mechanism, specifically: in: It is a query of the cross-graph attention mechanism, and the result of splicing the target video and multiple related videos along the temporal dimension is used as the key Sum

[0028] The graph mapper uses a learnable multi-layer perceptron to map the feature dimensions of the graph features to the same dimensions as the visual and text features of the original video language model, thereby achieving cross-modal feature alignment.

[0029] The word embedding layer integrates the graph structure features output from the graph fusion module<GRAPH_TOKENS> , video visual features output from the visual mapper module<VIDEO_TOKENS> and user questions<USER_QUERY> , and obtain video understanding results similar to the following:

[0030] The target video includes visual tokens represented as<VIDEO_TOKENS> and has structural representation of<GRAPH_TOKENS> .It's important to understand the relationship between visual tokens and the structuredrepresentation.

[0031] There are N related videos, each with its own structural representation. These related videos will help complete the missing information in the target video. The structural representations are asfollows:

[0032] Target video:<GRAPH_TOKENS> .Related video 1:<GRAPH_TOKENS> ....Related video N:<GRAPH_TOKENS> .Please reconsider the relationship between the videoand its structural representations.Think through the process step by step,then answer my

[0033] question:<USER_QUERY>

[0034] After specific practical experiments, it was verified that in the test environment of the multi-video question answering task, the integrated multi-video information was first input into the large language model for end-to-end fine-tuning, and then the target video and multiple related videos were used as input to start the inference of the large language model to run the device and method of the present invention, thereby obtaining more reliable and accurate question answering results. Figure 4 As shown in the figure, for the question "What is the purpose of the machine used by this man?", the traditional single-video method only generates the answer "He pushes this machine across the field" based on the target video. The answer is vague and fails to provide the specific purpose of the machine. However, the present invention uses cross-video knowledge integration and cross-graph attention mechanism, combining information from the target video and related videos to accurately generate the answer: "This machine is used to plow the land and prepare the land for sowing." The generated experimental data shows that compared with the traditional single-video method, the present invention can significantly improve the accuracy and completeness of the answer. In addition, the experimental results can also be verified by cross-graph attention visualization, such as Figure 4 As shown below, the model is able to focus on key segments in the target video and related videos, effectively reducing redundant information interference. This demonstrates that our method has excellent performance in multi-video joint reasoning tasks and can significantly improve the accuracy and completeness of question answering.

[0035] Compared with the prior art, the improvements of the present invention are reflected in multiple aspects. By introducing video structured representation and cross-video graph fusion modules, the present invention can efficiently integrate the complementary information of the target video and related videos, solving the problems of vague answers and hallucinatory errors caused by incomplete information in single video understanding. At the same time, based on the cross-graph attention mechanism, the present invention adaptively aggregates the related information between the target video and related videos, avoids information confusion, and significantly enhances the model's ability to understand complex problems. In addition, the present invention further optimizes the expression form of the input information by realizing multi-modal alignment of graph features, video features, and word features, and combining multi-video structured prompt word templates to input a large language model, which not only reduces the interference of redundant information, but also improves the overall understanding ability of multi-video content. Experimental results show that the present invention can generate more accurate and semantically consistent answers in video question-answering tasks, especially when dealing with problems with strong cross-video correlation, showing significant advantages, providing an efficient and reliable solution for multi-video joint understanding.

[0036] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principles and purpose of the present invention. The scope of protection of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. All implementation schemes within its scope shall be subject to the constraints of the present invention.

Claims

1. A multi-video joint understanding system for a large video language model, characterized by: include: Video structuring module, graph fusion module, visual mapper, visual encoder and word embedding layer, among which: the video structuring module extracts video key frames to obtain a detailed description text of the video, and constructs a complete video spatiotemporal structured representation by parsing at the text level to obtain a text scene graph composed of targets and interaction relationships and detecting and extracting the corresponding visual features of the targets in the video frames; the graph fusion module enhances the video structured representation by cross-video knowledge fusion; the visual encoder performs fine-grained feature encoding processing based on the video information of the target video to obtain an optimized video representation; the visual mapper performs cross-modal feature alignment processing based on the video representation provided by the visual encoder to obtain semantically consistent fusion features; the word embedding layer splices the graph features after fusion of related videos, the original visual features corresponding to the target video, and the question text according to the preset multi-video structured prompt word template, and then inputs the large language model to generate the final answer.

2. The multi-video joint understanding system for a large video language model according to claim 1 is characterized in that: The video structuring module includes: an event detector, a video descriptor, a scene graph parser, a visual locator and a spatiotemporal associator, wherein: the event detector uses a lightweight scene detection algorithm to segment the video time sequence scene according to the video to be processed, and obtains the middle frame of each scene as the key frame; the video descriptor uses a pre-trained video language model to perform a fine-grained description of the key frame based on the detailed description text of the video to be processed, and generates a semantic text covering the target, action and environment; the scene graph parser uses the scene graph parser to extract the <subject-predicate-object> triple from the description text to obtain a text scene composed of targets and interaction relationships Figure, the visual locator uses a text-based detector to locate the spatial position of the target in the triplet in the key frame, and extracts the visual features of the target area; at the same time, it establishes cross-frame target ID tags through the video target tracking algorithm to achieve the temporal continuity representation of the same target and obtain the corresponding visual features of the target in the video frame. The spatiotemporal associator constructs the interaction edges between targets in the same frame according to the predicate relationship in the text scene graph, and establishes the temporal association edges of the targets across frames by tracking the cross-frame target ID tags, finally forming a structured representation of the spatiotemporal graph that includes target features as nodes, interaction relationships as spatial edges and temporal associations as temporal edges.

3. The multi-video joint understanding system for a large video language model according to claim 2 is characterized in that: The event detector performs temporal segmentation on the video based on a lightweight scene detection algorithm, and uses the intermediate frames of the event as the video key frames.

4. The multi-video joint understanding system for a large video language model according to claim 2 is characterized in that: The video description device uses a pre-trained video language model to generate detailed video content description text covering multi-dimensional information of targets, actions, and environments based on the extracted key frames. It then optimizes the generated text through a semantic enhancement strategy to ensure that the generated text has high accuracy and expressiveness, providing high-quality semantic input for scene graph parsing.

5. The multi-video joint understanding system for a large video language model according to claim 2 is characterized in that: The scene graph parser generates a text scene graph consisting of multiple triplets by extracting the triples of <subject, interaction relationship, object> from the video description text.

6. The multi-video joint understanding system for a large video language model according to claim 2 is characterized in that: The visual locator determines the spatial position of the target in the keyframe based on the text description of the target, generates the target box and extracts target-level features.

7. The multi-video joint understanding system for a large video language model according to claim 2 is characterized in that: The spatiotemporal associator constructs spatial edges of target nodes based on the interaction relationship between targets obtained by the scene graph parser, and then constructs temporal edges of target nodes across frames based on the target tracking ID tags obtained by the visual locator, thereby constructing a spatiotemporal structured representation of the video.

8. The multi-video joint understanding system for a large video language model according to claim 1 is characterized in that: The graph fusion module includes: several graph attention networks and graph mappers based on the cross-graph attention mechanism connected in series, wherein: the graph attention network adopts a hierarchical graph attention network to learn the context-aware features of the single video graph structure based on the out-of-control structured representation; the graph mapper adopts a multi-layer perceptron to map the graph feature dimension to the same dimension as the visual and text features to achieve cross-modal feature alignment.

9. The multi-video joint understanding system for a large video language model according to claim 8 is characterized in that: The graph attention network inputs the spatiotemporal structured representation obtained by the video structuring module into GAT, and fuses the target node features according to the graph structure, thereby modeling the interaction and temporal relationships between targets. The cross-graph attention mechanism specifically includes: i) Take the target video X tar Compute the query, i.e. the target video structured representation Q after cross-graph attention fusion tar =X tar W Q , where: key K tar =X tar W K , value V tar =X tar W V ; ii) Use related video X rel Compute the relevant video structured representation Q rel =X rel W Q , where: key K rel =X rel W K , and the value V rel =X rel W V ; iii) In order to enable the model to better distinguish target videos from related videos, category embedding CE is introduced, specifically: CE tar =σ(α),CE rel =1-σ(α), where: α is a learnable parameter shared by multiple frames; iv) Add the category embedding to the query, key, and value of the attention mechanism, specifically: in: It is a query of the cross-graph attention mechanism, and the result of splicing the target video and multiple related videos along the temporal dimension is used as the key Sum 10. The multi-video joint understanding system for a large video language model according to claim 8 is characterized in that: The graph mapper uses a learnable multi-layer perceptron to map the feature dimensions of the graph features to the same dimensions as the visual and text features of the original video language model, thereby achieving cross-modal feature alignment.

Images