Target-layering-oriented memory-enhanced long video understanding system and method

Abstract

The invention provides a memory-enhanced long video understanding system and method oriented to target layering. The method comprises the following steps: sensing video content and extracting a target mask through a low-level target sensing module; aggregating the space-time relationship among multiple frames through a space-time memory construction module to extract frame-level space-time visual features; after splicing, storing in a long-term space-time memory library, enhancing time interaction of space-time visual features in the long-term space-time memory library through a time migration strategy, and aligning a visual embedding space with a text embedding space through a query converter; a query converter is used for processing video frames in an autoregression mode, historical information is accumulated step by step, and then the historical information is transmitted to a large language model to generate a final video understanding result. Through a hierarchical memory mechanism, gradual reasoning can be carried out from a target level to a frame level and then to a video level, and the ability of understanding complex video contents is effectively improved, especially for long videos and scenes containing complex target interaction.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and video understanding technology, and in particular to a target-oriented hierarchical memory-enhanced long video understanding system and method. Background Technology

[0002] Large language models have achieved remarkable success in the field of natural language processing (NLP), demonstrating their superior capabilities in performing multiple tasks. To extend this success to multiple modalities, multimodal large language models (MLLMs) connect pre-trained visual monomodal and language monomodal models, achieving state-of-the-art few-shot learning. They are trained using a frozen image encoder and a large language model, employing a general pre-training strategy.

[0003] Long-term video understanding remains a significant challenge in computer vision. Previous methods have included building long-term feature libraries using 3D convolutional neural networks (CNNs), object-based motion analysis, and memory-enhanced architectures; computational efficiency has been improved by decomposing dense self-attention into cascaded segment and region selection modules. Memory mechanisms are widely used in video-intensive prediction tasks to capture historical information and maintain temporal consistency, thereby improving the accuracy and consistency of long-term predictions. Other models utilize the Atkinson-Shiffrin memory model for efficient long-term video understanding. However, these methods still suffer from insufficient granularity of visual representations and lack of long-range dependencies in practical applications, specifically as follows: 1. Using simplified frame-level visual representations results in the loss of low-level spatial details such as target position and shape, leading to modal misalignment and model illusion when aligned with linguistic cues.

[0004] 2. Due to the limitations of LLM context length and GPU memory constraints, it can only process a limited number of video frames, resulting in the loss of early context information and making it difficult to establish a global scene understanding across frames. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a target-oriented hierarchical memory-enhanced long video understanding system and method. This invention can efficiently capture spatiotemporal details in complex videos and improve the performance of long video recognition and question-answering tasks through a hierarchical memory model.

[0006] In a first aspect, the present invention provides a target-hierarchical memory-enhanced long video understanding system, comprising: The low-level target perception module is used to perceive the input video content, extract the target mask from the input video, and then stitch it with the original video frames to generate a spatial representation of the target perception. The spatiotemporal memory construction module is used to aggregate the spatiotemporal relationships between multiple frames to extract frame-level spatiotemporal visual features; Long-term spatiotemporal memory is used to stitch together and store past spatiotemporal visual features from multiple video frames. The query transformer Q-Former is used to align the visual embedding space with the text embedding space; The language decoder uses Q-Former to process video frames in an autoregressive manner, gradually accumulating historical information, and then passes it to a large language model to generate the final video understanding result.

[0007] Preferably, the low-level target perception module provides an input video. any frame Use the base model to extract the target mask. ,Right now: ; In the formula, This indicates guidance information.

[0008] Preferably, the target mask of the low-level target perception module is concatenated with the original video frames to generate a spatial representation of the target perception. Specifically, it is expressed as: ; In the formula, This indicates a splicing operation.

[0009] Preferably, the spatiotemporal memory construction module splices multiple adjacent video frames along the channel dimension to form a local multi-frame temporal representation, and then uses a projector to aggregate the local multi-frame temporal representation into each frame; ; In the formula, This represents the enhanced spatiotemporal feature table obtained by spatiotemporally fusing the spatial representation of target perception with adjacent frames; This represents the process of projecting spatiotemporal visual features onto each frame; for Adjacent frames; This indicates a splicing operation.

[0010] Preferably, the spatiotemporal memory construction module uses a pre-trained image-based model. To obtain the visual features of each frame, through a location embedding layer. Integrating temporal sequence information into frame-level features, that is: ; In the formula, This represents a frame-level visual feature representation that integrates spatiotemporal features and location coding.

[0011] Preferably, the long-term spatiotemporal memory bank stores historical information by splicing spatiotemporal visual features extracted from each frame during the memory construction process, at the current time step. The stored spatiotemporal visual features are represented as follows: ; In the formula, Indicates from frame 1 to frame 2. A long-term spatiotemporal memory database composed of all frame-level spatiotemporal features of a frame.

[0012] Preferably, a time-shifting strategy is used to enhance the temporal interaction of spatiotemporal visual features in the long-term spatiotemporal memory. This strategy efficiently captures temporal dynamics by shifting feature channels along the time dimension, and strategic channel shifting facilitates information exchange between different time steps, thus enhancing the spatiotemporal visual feature... The time offset strategy is expressed as follows: ; In the formula, Indicates the first Frame offset characteristics; In terms of channel dimension, the channel is divided into three equal parts, each occupying C / 3, and each part is shifted sequentially at each time step to enhance temporal fusion; Indicates that it comes from the previous time step The first part of the channel; Including the current time step The last third of the passage; Covering the next time step The middle third of the passage.

[0013] As a preferred method, global average pooling is used to extract global information, thereby generating a global video representation vector. ,Right now: ; The global video representation vector It captures the overall context of the video.

[0014] Preferably, the global video representation vector is used. Offset features of each frame The frames are stitched together to enhance each frame's awareness of the global context, that is: ; In the formula, Indicates the first Global context enhancement features for frames.

[0015] Preferably, random sampling is used to maintain a fixed size of the long-term spatiotemporal memory, by utilizing the global context of each sampled frame. Reduce the density of the long-term spatiotemporal memory, that is: ; In the formula, This represents the features of a fixed-size long-term spatiotemporal memory obtained after global context enhancement and random sampling; This indicates the number of sampled frames.

[0016] Preferably, the query transformer Q-Former uses a set of learned queries. Extracting temporal information from the video, through the aforementioned query... Capturing temporal features in videos and facilitating alignment between visual and textual embedding spaces, each Q-Former consists of two attention submodules: a cross-attention layer that interacts with the raw visual embeddings of the frozen visual encoder and a self-attention layer that captures interactions between input queries.

[0017] Preferably, the query transformer Q-Former invokes visual spatiotemporal visual features from the long-term spatiotemporal memory through cross-attention; it invokes semantic features from the query memory through self-attention, and after alternating fusion, outputs a condensed historical information representation, which is then passed to the large language model to generate the final video understanding result.

[0018] Secondly, the present invention provides a target-oriented hierarchical memory enhancement method for long video understanding, comprising the following steps: S1) Extract the target mask from the input video and stitch it with the original video frames to generate a spatial representation of the target for perception; S2) Aggregate the spatiotemporal relationships between multiple frames to extract frame-level spatiotemporal visual features; S3) The past spatiotemporal visual features from multiple video frames are stitched together and stored in a long-term spatiotemporal memory bank; S4) Align the visual embedding space with the text embedding space using the query transformer Q-Former; S5) Use Q-Former to process video frames in an autoregressive manner, gradually accumulate historical information, and then pass it to a large language model to generate the final video understanding result.

[0019] The beneficial effects of this invention are as follows: 1. This invention, through a hierarchical memory mechanism, can perform step-by-step reasoning from the target level to the frame level and then to the video level, effectively improving the ability to understand complex video content, especially for long videos and scenarios containing complex target interactions; 2. By introducing a spatiotemporal memory construction module and a long-term memory bank, this invention can extract spatial and temporal relationships in videos and retain rich historical context, effectively reducing the possibility of the model generating hallucinatory responses and avoiding the omission or introduction of non-existent targets. 3. This invention can capture complex details in videos, ensuring accurate detection of targets in videos and improving the accuracy of target recognition; 4. While maintaining efficient video understanding, this invention employs efficient sampling methods and memory management strategies, enabling it to effectively process long video data and maintain low memory usage even under GPU memory limitations. Attached Figure Description

[0020] Figure 1 This is a structural framework diagram of the system in Embodiment 1 of the present invention; Figure 2 This is a flowchart illustrating the method of Embodiment 2 of the present invention; Figure 3 This is a result diagram of the VQA task on MSVD in Embodiment 1 of the present invention. Detailed Implementation

[0021] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings: Example 1 like Figure 1 As shown, this embodiment provides a target-oriented hierarchical memory-enhanced long video understanding system, including: The low-level target perception module is used to perceive video content and extract target masks from the video. Then, the extracted target masks are concatenated with the original video frames to generate a spatial representation of the target perception. The low-level target perception module, for a given input video any frame The base model SAM2 is used to extract the target mask. ,Right now: ; In the formula, This indicates guidance information; , , and These represent the number of video frames, the height of each frame, the width, and the channel dimension, respectively.

[0022] This embodiment uses a low-level target perception module to perform target perception on the input video, providing detailed target-level information to facilitate the understanding of long videos.

[0023] Then target mask With the original video frames By stitching the images together, a spatial representation for target perception is generated. Specifically, it is expressed as: ; In the formula, This indicates a splicing operation.

[0024] The hierarchical structure (target → frame → video) in this embodiment has fault tolerance capabilities, enhancing robustness to low-level perception failures.

[0025] The spatiotemporal memory construction module is used to aggregate the spatiotemporal relationships between multiple frames to extract frame-level spatiotemporal visual features; In this embodiment, the spatiotemporal memory construction module enhances the understanding of video content by aggregating the spatiotemporal relationships between multiple frames, effectively capturing target, location, and other detailed frame information. The spatiotemporal memory construction module stitches together multiple adjacent video frames along the channel dimension to form a local multi-frame temporal representation, and then uses a projector to aggregate this local multi-frame temporal representation into each frame. ; In the formula, This represents the enhanced spatiotemporal feature table obtained by spatiotemporally fusing the spatial representation of target perception with adjacent frames; This represents the process of projecting spatiotemporal visual features onto each frame; for Adjacent frames; This indicates a splicing operation.

[0026] The spatiotemporal memory construction module uses a pre-trained image base model. To obtain the visual features of each frame, through a location embedding layer. Integrating temporal sequence information into frame-level features, that is: ; In the formula, This represents a frame-level visual feature representation that integrates spatiotemporal features and location coding; The number of patches per frame. The channel dimension for each frame.

[0027] In this embodiment, a 3×3 convolutional layer is used as the projector, and ViT-G / 14 from EVA-CLIP is used as the image base model. .

[0028] Long-term spatiotemporal memory is used to stitch together and store past spatiotemporal visual features from multiple video frames. In this embodiment, the long-term spatiotemporal memory bank stores historical information by splicing spatiotemporal visual features extracted from each frame during the memory construction process, at the current time step. The stored spatiotemporal visual features are represented as follows: ; In the formula, Indicates from frame 1 to frame 2. A long-term spatiotemporal memory database composed of all frame-level spatiotemporal features of a frame.

[0029] In this embodiment, a time-shifting strategy is used to enhance the temporal interaction of spatiotemporal visual features in the long-term spatiotemporal memory. This time-shifting strategy is applied to visual feature vectors with P × C dimensions. Where P represents the spatial dimension and C represents the channel dimension; the channel is divided into three equal parts, each occupying C / 3; this setting is shifted sequentially at each time step to enhance temporal fusion. The aforementioned temporal shifting strategy efficiently captures temporal dynamics by shifting feature channels along the time dimension, and promotes information exchange between different time steps through strategic channel shifting, thus enhancing spatiotemporal visual features. The time offset strategy is expressed as follows: ; In the formula, Indicates the first Frame offset characteristics; Indicates that it comes from the previous time step The first part of the channel; Including the current time step The last third of the passage; Covering the next time step The middle third of the passage.

[0030] This embodiment promotes information exchange between different time steps by channel offset, thereby enhancing the memory's ability to integrate temporal information, while maintaining efficient spatial feature extraction without significantly increasing the computational burden.

[0031] In this embodiment, capturing high-level contextual knowledge is crucial for effective video understanding; global average pooling is used to extract global information, thereby generating a global video representation vector. ,Right now: ; The global video representation vector It captures the overall context of the video.

[0032] Then by using the global video representation vector Offset features of each frame The frames are stitched together to enhance each frame's awareness of the global context, that is: ; In the formula, Indicates the first Global context enhancement features for frames.

[0033] In this embodiment, random sampling is used to maintain a fixed size of the long-term spatiotemporal memory, thereby minimizing computational overhead; by utilizing the global context of each sampling frame... This reduces the density of the long-term spacetime memory while maintaining efficiency, preserving necessary information without increasing computational costs. ; In the formula, This represents the features of a fixed-size long-term spatiotemporal memory obtained after global context enhancement and random sampling; This indicates the number of sampled frames.

[0034] In this embodiment, by adjusting the sampling rate, it is adapted to the needs of different tasks in practical applications, thereby enhancing the feasibility of practical applications.

[0035] In this embodiment, when the low-level target perception module produces an erroneous output, the long-term spatiotemporal memory acts as a compensation mechanism. By storing contextual information across multiple frames and capturing spatial and temporal dependencies, the impact of local low-level target perception module errors is mitigated. This accumulated context enables the system to correct small perception errors and maintain accurate scene understanding over time.

[0036] The query transformer Q-Former is used to align the visual embedding space with the text embedding space; In this embodiment, the query transformer Q-Former uses a set of learned queries. Extracting temporal information from the video, through the aforementioned query... Capturing temporal features in videos and facilitating alignment between visual and textual embedding spaces, each Q-Former consists of two attention submodules: a cross-attention layer that interacts with the raw visual embeddings of the frozen visual encoder and a self-attention layer that captures interactions between input queries.

[0037] The language decoder uses Q-Former to process video frames in an autoregressive manner, gradually accumulating historical information to reduce the number of input tokens; it then passes the information to a large language model to generate the final video understanding result. In this embodiment, the query transformer Q-Former invokes visual spatiotemporal visual features from the long-term spatiotemporal memory through cross-attention; it invokes semantic features from the query memory through self-attention, and outputs a condensed historical information representation after alternating fusion. Then, it passes the representation to the large language model through the projection layer to generate the final video understanding result.

[0038] This approach effectively alleviates the constraints imposed by context length and memory limitations in large language models. During training, Q-Former updates via cross-entropy loss on video-text pairs while the language model remains frozen, thus achieving efficient adaptation to the video domain.

[0039] Example 2 like Figure 2 As shown, this embodiment provides a target-oriented hierarchical memory enhancement method for long video understanding, including the following steps: S1) Extract the target mask from the input video and stitch it with the original video frames to generate a spatial representation of the target for perception; Given an input video any frame The base model SAM2 is used to extract the target mask. ,Right now: ; In the formula, This indicates guidance information; , , and These represent the number of video frames, the height of each frame, the width, and the channel dimension, respectively.

[0040] This embodiment uses a low-level target perception module to perform target perception on the input video, providing detailed target-level information to facilitate the understanding of long videos.

[0041] Then target mask With the original video frames By stitching the images together, a spatial representation for target perception is generated. Specifically, it is expressed as: ; In the formula, This indicates a splicing operation.

[0042] S2) Aggregate the spatiotemporal relationships between multiple frames to extract frame-level spatiotemporal visual features; By stitching together multiple adjacent video frames along the channel dimension, a local multi-frame temporal representation is formed. Then, a projector is used to aggregate the local multi-frame temporal representation into each frame. ; In the formula, This represents the enhanced spatiotemporal feature table obtained by spatiotemporally fusing the spatial representation of target perception with adjacent frames; This represents the process of projecting spatiotemporal visual features onto each frame; for Adjacent frames; This indicates a splicing operation.

[0043] The spatiotemporal memory construction module uses a pre-trained image base model. To obtain the visual features of each frame, through a location embedding layer. Integrating temporal sequence information into frame-level features, that is: ; In the formula, This represents a frame-level visual feature representation that integrates spatiotemporal features and location coding; The number of patches per frame. The channel dimension for each frame.

[0044] In this embodiment, a 3×3 convolutional layer is used as the projector, and ViT-G / 14 from EVA-CLIP is used as the image base model. .

[0045] S3) The past spatiotemporal visual features from multiple video frames are stitched together and stored in a long-term spatiotemporal memory bank; In this embodiment, the long-term spatiotemporal memory bank stores historical information by splicing spatiotemporal visual features extracted from each frame during the memory construction process, at the current time step. The stored spatiotemporal visual features are represented as follows: ; In the formula, Indicates from frame 1 to frame 2. A long-term spatiotemporal memory database composed of all frame-level spatiotemporal features of a frame.

[0046] In this embodiment, a time-shifting strategy is used to enhance the temporal interaction of spatiotemporal visual features in the long-term spatiotemporal memory. This time-shifting strategy is applied to visual feature vectors with P × C dimensions. Where P represents the spatial dimension and C represents the channel dimension; the channel is divided into three equal parts, each occupying C / 3; this setting is shifted sequentially at each time step to enhance temporal fusion. The aforementioned temporal shifting strategy efficiently captures temporal dynamics by shifting feature channels along the time dimension, and promotes information exchange between different time steps through strategic channel shifting, thus enhancing spatiotemporal visual features. The time offset strategy is expressed as follows: ; In the formula, Indicates the first Frame offset characteristics; Indicates that it comes from the previous time step The first part of the channel; Including the current time step The last third of the passage; Covering the next time step The middle third of the passage.

[0047] This embodiment promotes information exchange between different time steps by channel offset, thereby enhancing the memory's ability to integrate temporal information, while maintaining efficient spatial feature extraction without significantly increasing the computational burden.

[0048] In this embodiment, capturing high-level contextual knowledge is crucial for effective video understanding; global average pooling is used to extract global information, thereby generating a global video representation vector. ,Right now: ; The global video representation vector It captures the overall context of the video.

[0049] Then by using the global video representation vector Offset features of each frame The frames are stitched together to enhance each frame's awareness of the global context, that is: ; In the formula, Indicates the first Global context enhancement features for frames.

[0050] In this embodiment, random sampling is used to maintain a fixed size of the long-term spatiotemporal memory, thereby minimizing computational overhead; by utilizing the global context of each sampling frame... This reduces the density of the long-term spacetime memory while maintaining efficiency, preserving necessary information without increasing computational costs. ; In the formula, This represents the features of a fixed-size long-term spatiotemporal memory obtained after global context enhancement and random sampling; This indicates the number of sampled frames.

[0051] In this embodiment, by adjusting the sampling rate, it is adapted to the needs of different tasks in practical applications, thereby enhancing the feasibility of practical applications.

[0052] S4) Align the visual embedding space with the text embedding space using the query transformer Q-Former; In this embodiment, the query transformer Q-Former uses a set of learned queries. Extracting temporal information from the video, through the aforementioned query... Capturing temporal features in videos and facilitating alignment between visual and textual embedding spaces, each Q-Former consists of two attention submodules: a cross-attention layer that interacts with the raw visual embeddings of the frozen visual encoder and a self-attention layer that captures interactions between input queries.

[0053] S5) Use Q-Former to process video frames in an autoregressive manner, gradually accumulate historical information, and then pass it to the large language model to generate the final video understanding result; The query transformer Q-Former invokes visual spatiotemporal visual features from the long-term spatiotemporal memory through cross-attention; it invokes semantic features from the query memory through self-attention, and after alternating fusion, it outputs a condensed historical information representation, which is then passed to the large language model to generate the final video understanding result.

[0054] Example 3 This embodiment validates the performance of Embodiment 1 on two challenging tasks: long-term video recognition and video question answering. To ensure a fair comparison, the Vicuna-7B was chosen as the LLM architecture. To construct the spatiotemporal memory, a 3×3 convolutional layer was used as the projector, and the pre-trained image encoder ViT-G / 14 from EVA-CLIP was employed. This embodiment compares the performance of Embodiment 1 and existing methods in handling complex video understanding tasks. The results are shown in Tables 1 and 2.

[0055] Table 1. Comparison of results between the method in Example 1 and existing methods on long-term video LVU datasets. Table 2. Comparison of the results of Example 1 method and existing models on the MSRVTT-QA and MSVD-QA datasets for open video question answering tasks. As can be seen from Table 1, Example 1 outperforms other methods in multiple video understanding tasks; as can be seen from Table 2, Example 2 achieves state-of-the-art performance on the MSRVTT-QA and MSVD-QA datasets.

[0056] To visually and intuitively evaluate the video understanding performance of the method in Example 1, the experimental results of Example 1 in the VQA task on MSVD were visualized, as follows: Figure 3 As shown, the results indicate that without a low-level object perception module, MA-LMM is prone to illusions when performing long-video understanding in complex scenes; the objects inferred by LOP correspond to the low-level representations of the relevant answers. The low-level object perception module, used for low-level object recognition, facilitates more accurate high-level video interpretation, such as accurately identifying the action of "chopping vegetables" instead of misclassifying it as "fighting." This highlights the importance of incorporating hierarchical object segmentation into the framework of long-video understanding.

[0057] The embodiments and descriptions above are merely illustrative of the principles and preferred embodiments of the present invention. Various changes and modifications may be made to the present invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed.

Claims

1. A target-oriented hierarchical memory-augmented long video understanding system, characterized in that, The application relates to a system and method for long video understanding. The system comprises: a low-level target perception module for perceiving input video content and splicing a target mask extracted from the input video with an original video frame to generate a target-perceived spatial representation; a spatiotemporal memory construction module for aggregating spatiotemporal relationships between multiple frames to extract frame-level spatiotemporal visual features; a long-term spatiotemporal memory bank for storing past spatiotemporal visual features from multiple video frames after splicing; and enhancing the time interaction of the spatiotemporal visual features in the long-term spatiotemporal memory bank through a time offset strategy a query transformer Q-Former for aligning a visual embedding space with a text embedding space; 2. The target-oriented hierarchical memory-augmented long video understanding system according to claim 1, wherein: The low-level target perception module, for a given input video any frame Use the base model to extract the target mask. ,Right now: ； In the formula, represents the guide information.

3. The object-oriented hierarchical memory-augmented long video understanding system of claim 2, wherein: The low-level target perception module target mask is spliced with the original video frame to generate a target perception spatial representation Specifically represented as: ； In the formulae, represents a concatenation operation.

4. The object-oriented hierarchical memory-augmented long video understanding system of claim 3, wherein: a language decoder for processing video frames in a self-recurrent manner using the Q-Former, accumulating historical information step by step, and then transmitting the historical information to a large language model to generate a final video understanding result. ； In the formula, represents an enhanced spatio-temporal feature table obtained by spatio-temporally fusing the target perceived spatial representation with adjacent frames; represents a process of projecting the spatio-temporal visual feature to each frame; is adjacent frames of represents a splicing operation.

5. The object-oriented hierarchical memory-augmented long video understanding system of claim 4, wherein: The spatiotemporal memory construction module uses a pre-trained image base model. To obtain the visual features of each frame, through a location embedding layer Integrating temporal sequence information into frame-level features, that is: ； In the formula, represents a frame-level visual feature representation that fuses spatio-temporal features and position encoding.

6. The object-oriented hierarchical memory-augmented long video understanding system of claim 5, wherein: The long-term spatio-temporal memory bank stores historical information by concatenating spatio-temporal visual features extracted from each frame in the memory construction process, at the current time step The stored concatenated spatio-temporal visual features are represented as: ； In the formula, represents a long-term spatiotemporal memory bank spliced by all frame-level spatiotemporal features from the 1st frame to the frame.

7. The object-oriented hierarchical memory-augmented long video understanding system of claim 6, wherein: The time offset strategy captures time dynamics by offsetting feature channels along the time dimension, and facilitates information exchange between different time steps by channel offsetting The time offset strategy is represented as: ； wherein represents the first frame offset characteristic; is the channel dimension, the channels are divided into three equal parts, each part occupying C / 3, and each group is offset in turn at each time step, thereby enhancing temporal fusion; represents the first part of the channel from the previous time step ; includes the last third of the channel of the current time step ; covers the middle third of the channel of the next time step ; extract global information by global average pooling to generate a global video representation vector i.e.: ； The global video representation vector captures the overall context of the video; by concatenating the global video representation vector with the offset features of each frame to enhance each frame's perception of the global context, i.e.: ； In the formula, represents the global context enhancement features of the frame.

8. The object-oriented hierarchical memory-augmented long video understanding system of claim 7, wherein: The size of the fixed long-term spatio-temporal memory bank is maintained using random sampling, by exploiting the global context of each sampled frame reducing the density of the long-term spatio-temporal memory bank, i.e.: ； In the formula, represents the fixed-size long-term spatio-temporal memory bank feature obtained after global context augmentation and random sampling; represents the number of sampled frames.

9. The object-oriented hierarchical memory-augmented long video understanding system of claim 8, wherein: The query transformer Q-Former uses a set of learned queries extracting temporal information from videos, by the query capturing temporal features in videos and facilitating alignment between visual embedding space and text embedding space, each Q-Former consists of two attention sub-modules, namely cross-attention layer interacting with original visual embeddings of a frozen visual encoder and self-attention layer capturing interactions between input queries; The spatiotemporal memory construction module splices multiple adjacent video frames along the channel dimension to form a local multi-frame time sequence representation, and then uses a projector to aggregate the local multi-frame time sequence representation into each frame.

10. A target-oriented hierarchical memory-augmented long video understanding method, characterized in that, The query transformer Q-Former calls the visual spatiotemporal visual features of the long-term spatiotemporal memory bank through cross-attention, calls the semantic features of the query memory bank through self-attention, alternately fuses to output condensed historical information representation, and then transmits the historical information representation to a large language model to generate a final video understanding result. The method uses the system of any one of claims 1-9 for long video understanding, and the method comprises the following steps: S1), low-level target perception of an input video, splicing a target mask extracted from the input video with an original video frame to generate a target-perceived spatial representation; S2), aggregating spatiotemporal relationships between multiple frames to extract frame-level spatiotemporal visual features; S3), splicing past spatiotemporal visual features from multiple video frames and storing them in a long-term spatiotemporal memory bank; S4), aligning a visual embedding space with a text embedding space using a query transformer Q-Former; S5), processing video frames in a self-recurrent manner using the Q-Former, accumulating historical information step by step, and then transmitting the historical information to a large language model to generate a final video understanding result.

Images