Object-oriented hierarchical memory network device

Abstract

The application provides a kind of object-oriented hierarchical memory network device based on, comprising: low-level object perception module, space-time memory construction module and long-term space-time memory modeling module;The low-level object perception module is used to extract the object feature of input video by segmenting the base model;The space-time memory construction module is used to extract the information of object interaction and scene by analyzing the interframe space and time relationship, so as to splice the object feature with the original video frame input, generate the spatial representation of target perception;The long-term space-time memory modeling module is used to store historical information through memory bank, capture global dependence relationship at video level, splice multiple adjacent video frames output by the space-time memory construction module in channel dimension, to form local multi-frame space-time representation.The purpose of enhancing the model's ability to capture long video fine-grained space-time dynamics is achieved.

Description

Technical Field

[0001] This invention belongs to the field of video processing technology, and in particular relates to an object-oriented hierarchical memory network device. Background Technology

[0002] Object-oriented hierarchical memory-based multimodal video understanding is an important technology at the intersection of computer vision and natural language processing. Its core lies in combining object perception, spatiotemporal memory construction, and long-term memory modeling to achieve efficient understanding and analysis of fine-grained spatiotemporal dynamics in long video content. This technology has shown broad application potential in multiple fields such as video surveillance, video retrieval, video captioning generation, and cross-modal reasoning. However, existing multimodal video understanding methods face many challenges when processing long video sequences, especially in capturing fine-grained spatiotemporal relationships and complex dynamic scenes. Long video understanding not only requires models to capture fine-grained temporal dependencies in the video but also to maintain high accuracy under complex conditions such as lengthy video content, rapid target movement, and frequent scene changes. Furthermore, given the massive nature of video data, models must balance efficiency and scalability to meet the stringent requirements of real-time applications.

[0003] For long video understanding tasks, researchers have proposed various methods, including frame-level feature-based methods and object detection-based methods. Frame-level feature-based methods reduce computational burden by compressing video frame information, but lack the ability to capture low-level object details; while object detection-based methods can identify objects in the video, they often ignore global information at the long video level. Furthermore, existing methods often struggle to capture fine-grained spatiotemporal relationships when processing long videos and are computationally expensive. However, in the complex video environments of real-world applications, these methods still face the following challenges:

[0004] 1. Lack of fine-grained spatiotemporal modeling: Existing methods often struggle to capture fine-grained spatiotemporal relationships when processing long videos, resulting in poor performance in complex dynamic scenes.

[0005] 2. High computational cost: Although frame-level feature-based methods can reduce computational burden, they lack the ability to capture details of low-level objects, which limits their performance in complex scenes.

[0006] 3. Insufficient utilization of object-level information: Existing methods do not fully utilize object-level information, which can easily lead to illusions in video understanding tasks, such as missing visible objects or generating non-existent objects. Summary of the Invention

[0007] In view of the problems existing in the prior art, the present invention provides a hierarchical memory network device based on object-oriented approach, which at least partially solves the problem of difficulty in capturing fine-grained spatiotemporal dynamics in long videos.

[0008] This disclosure provides an object-oriented hierarchical memory network device, including:

[0009] Low-level object perception module, spatiotemporal memory construction module, and long-term spatiotemporal memory modeling module;

[0010] The low-level object perception module is used to extract object features from the input video through the segmentation base model;

[0011] The spatiotemporal memory construction module is used to extract information about object interaction and scene by analyzing the spatial and temporal relationships between frames, thereby splicing the object features with the input original video frames to generate a spatial representation of the target perception.

[0012] The long-term spatiotemporal memory modeling module is used to store historical information through a memory bank, capture global dependencies at the video level, and stitch together multiple adjacent video frames output by the spatiotemporal memory construction module in the channel dimension to form a local multi-frame spatiotemporal representation.

[0013] Optionally, the low-level object perception module is used to perform object perception on the input video using a segmentation base model, extract target masks, and thus extract object features.

[0014] Optionally, the spatiotemporal memory construction module generates a spatial representation of the target perception by concatenating the target mask with the input original video frame along the channel dimension.

[0015] Optionally, the long-term spatiotemporal memory construction module forms a local multi-frame temporal representation by splicing multiple adjacent video frames along the channel dimension and aggregating spatiotemporal features.

[0016] Optionally, the spatiotemporal memory construction module uses a pre-trained image model to extract frame-level visual features and incorporates temporal information through position encoding.

[0017] Optionally, the long-term spatiotemporal memory modeling module splices the extracted spatiotemporal features into a long-term spatiotemporal memory library.

[0018] Optionally, the long-term spatiotemporal memory modeling module enhances the global context awareness capability of each frame through a global memory enhancement strategy, calculated as follows:

[0019] ;

[0020] in, Indicates the first memory in the memory repository The feature vector of a frame, Total number of memory frames, This represents the global context information for each sampled frame.

[0021] Optionally, the long-term spatiotemporal memory modeling module optimizes time interaction through a time channel offset strategy;

[0022] The time channel offset strategy implements timing offset using the following formula:

[0023] ;

[0024] in, Indicates the previous time step The former One channel, Represents the current time step The end One channel, and Represents the next time step The middle One channel, This is the timing offset.

[0025] Optionally, the spatiotemporal memory construction module reduces the number of stored historical frames by randomly sampling video frames.

[0026] Optionally, the long-term spatiotemporal memory modeling module achieves temporal integration of features from multiple frames by storing historical information extracted from video frames. At the current time step, the long-term spatiotemporal memory modeling module achieves the cumulative storage of global video information by splicing the spatiotemporal features of multiple video frames.

[0027] The object-oriented hierarchical memory network device provided by this invention sets up a low-level object perception module, a spatiotemporal memory construction module, and a long-term spatiotemporal memory modeling module. The low-level object perception module uses a segmentation base model to extract detailed object features, and the spatiotemporal memory construction module captures the spatial and temporal relationships between frames. Finally, the long-term memory modeling module stores global video information, thereby enhancing the model's ability to capture fine-grained spatiotemporal dynamics in long videos.

[0028] The device provided by this invention also reduces computational overhead through a random sampling strategy, thereby improving the performance of the model in long video understanding tasks. Attached Figure Description

[0029] The above and other objects, features and advantages of this disclosure will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.

[0030] Figure 1This is a schematic diagram of the object-oriented hierarchical memory storage mechanism provided in the embodiments of this disclosure;

[0031] Figure 2 This is a block diagram of the principle of the hierarchical memory network device based on object-oriented design provided in the embodiments of this disclosure;

[0032] Figure 3 This is a visualization result of a video question-answering task on MSVD provided in the disclosed embodiments;

[0033] Figure 4 This disclosure provides a visualization of the relationship between object masks and answers in a VQA task on MSVD.

[0034] Figure 5 This is a schematic diagram illustrating the impact of different memory library lengths provided in the embodiments of this disclosure. Detailed Implementation

[0035] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0036] It should be understood that the following specific examples illustrate the implementation of this disclosure, and those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0037] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0038] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The illustrations only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0039] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0040] This embodiment proposes a multimodal video understanding framework based on object-oriented hierarchical memory. By introducing low-level object perception, spatiotemporal memory construction, and long-term memory modeling, it enhances the model's ability to capture fine-grained spatiotemporal dynamics in long videos. Specifically, it uses a segmentation-based model to extract detailed object features, captures spatial and temporal relationships between frames through a spatiotemporal memory construction module, and finally stores global video information through a long-term memory modeling module. This method not only effectively captures fine-grained spatiotemporal dynamics in videos but also reduces computational overhead through a random sampling strategy, improving the model's performance in long video understanding tasks. The long-term spatiotemporal memory modeling module stores global video information in a fixed-size memory bank, ensuring a balance between capturing global and local features. By combining low-level object perception, spatiotemporal memory construction, and long-term spatiotemporal memory modeling, this embodiment not only effectively captures fine-grained spatiotemporal dynamics in videos but also reduces computational overhead through a random sampling strategy, improving the model's performance in long video understanding tasks. Furthermore, this embodiment aligns multi-level visual embeddings with text embeddings through a query transformer and utilizes a large language model for text decoding, further enhancing the model's cross-modal understanding capabilities. Through this series of innovative designs, this embodiment achieves state-of-the-art performance on multiple long video understanding benchmark datasets, significantly improving the accuracy and efficiency of video understanding in complex dynamic scenes.

[0041] For ease of understanding, such as Figure 1 As shown, this embodiment discloses a hierarchical memory network device based on object-oriented principles.

[0042] Low-level object perception module, spatiotemporal memory construction module, and long-term spatiotemporal memory modeling module;

[0043] The low-level object perception module is used to extract object features from the input video through the segmentation base model;

[0044] The spatiotemporal memory construction module is used to extract information about object interaction and scene by analyzing the spatial and temporal relationships between frames, thereby splicing the object features with the input original video frames to generate a spatial representation of the target perception.

[0045] The long-term spatiotemporal memory modeling module is used to store historical information through a memory bank, capture global dependencies at the video level, and stitch together multiple adjacent video frames output by the spatiotemporal memory construction module in the channel dimension to form a local multi-frame spatiotemporal representation.

[0046] The low-level object perception module extracts detailed object features by segmenting the basic model; the spatiotemporal memory construction module extracts rich clues about object interactions and scenes by analyzing the spatial and temporal relationships between frames; and the long-term spatiotemporal memory modeling module stores historical information in a fixed-size memory bank to capture global dependencies at the video level.

[0047] In order to improve the understanding of long videos, this embodiment needs to capture key spatial objects in each frame as candidate targets, thereby promoting cross-modal alignment and improving the video content parsing ability.

[0048] The object-oriented hierarchical memory method for long video understanding proposed in this embodiment is a simple, efficient, and effective multi-level video understanding framework. It employs a hierarchical fine-grained analysis method to process video content to generate multi-level visual embedding representations.

[0049] At the object level, this embodiment uses the segmentation perception information provided by the basic model SAM2 to achieve low-level object perception capabilities in order to accurately capture target features.

[0050] At the frame level, the spatiotemporal memory construction module extracts key information about object interactions and scenes, and achieves feature modeling through inter-frame spatial and temporal relationship analysis.

[0051] At the video level, a long spatiotemporal memory modeling strategy is introduced, which uses fixed-size memory storage units to capture global dependencies across video frames.

[0052] A global memory enhancement mechanism is employed to expand the receptive field of each frame, while a random sampling strategy is combined to reduce temporal redundancy, ensuring improved computational efficiency without losing key information. After completing the multi-level visual embedding representation, it is further aligned with the text embedding through a query transformer, and text decoding is achieved with the help of a large language model.

[0053] To optimize the video processing workflow, video features are stored in compact long-term spatiotemporal memory units to integrate historical information. This hierarchical spatiotemporal memory mechanism can significantly reduce GPU computational overhead while improving the accuracy of long video understanding.

[0054] like Figure 2As shown, this embodiment proposes an efficient memory mechanism and deeply integrates it with LLMs. This method introduces fine-grained visual information at the object, frame, and video levels, optimizes visual-language alignment, effectively alleviates the illusion problem of large models, and improves computational efficiency and accuracy in long video understanding tasks.

[0055] This embodiment uses a general video modeling framework, which differs from the traditional method of directly compressing video content. It adopts a stepwise reasoning strategy, that is, to perceive first and then memorize, in order to retain key information.

[0056] This embodiment starts with low-level target perception, then constructs frame-level spatiotemporal memory, and finally performs efficient long-term spatiotemporal memory modeling at the video level.

[0057] The base model SAM2 includes a low-level object perception module to achieve efficient perception and processing of input video content. Specifically, given an input video, where represents the frame number, height, width, and number of channels, the SAM2 model is used to extract object masks to enhance object-level semantic perception capabilities.

[0058] ;

[0059] in, This module provides cue information to guide target detection. Its introduction enables OHM to accurately perceive and extract key information at the low-level target level, laying the foundation for subsequent frame-level and video-level memory modeling.

[0060] The spatiotemporal memory construction module enhances the ability to capture target, location, and frame-level information by aggregating spatiotemporal relationships across multiple frames. In this module, target masks are concatenated to the channel dimensions of the original video frames to construct a spatial representation for target perception.

[0061] ;

[0062] Subsequently, multiple adjacent video frames are stitched together along the channel dimension to form a local multi-frame spatiotemporal representation. Furthermore, a projection transformer is introduced to aggregate frame-level spatiotemporal features, calculated as follows:

[0063] ;

[0064] A highly efficient pre-trained image model is used to extract frame-level visual features, and temporal information is further incorporated through positional encoding to enhance spatiotemporal modeling capabilities. The final frame-level feature representation is as follows:

[0065] ;

[0066] This represents the number of blocks in each frame. This mechanism effectively enhances the spatiotemporal consistency of video and improves the model's ability to understand long videos.

[0067] The long-term spatiotemporal memory modeling module achieves temporal integration of features from multiple frames by storing historical information extracted from video frames. At the current time step, this module accumulates and stores global video information by concatenating the spatiotemporal features of multiple video frames. Its mathematical expression is as follows:

[0068] ;

[0069] To align multi-level visual features with the text embedding space, this embodiment employs a query transformer structure similar to BLIP-2. The Q-Former utilizes learnable query vectors z to extract temporal information from the video, where z represents the number of query vectors. These query vectors effectively capture temporal features in the video and facilitate alignment between the visual and text embedding spaces.

[0070] To optimize the temporal interaction capabilities of the spatiotemporal memory storage module, this embodiment proposes a Temporal Shift Strategy (TSS). By shifting feature channels along the time dimension, it enhances the ability to model temporal dynamics. Specifically, TSS operates on the feature vector, whose dimension is... ,in, Indicates the time dimension; Indicates spatial dimension, This represents the channel dimension. The TSS mechanism divides the channel into three equal parts, with each part containing a percentage of channels. The timing offset is achieved using the following formula:

[0071] ;

[0072] in, Indicates the previous time step The former One channel, Represents the current time step The end One channel, and Represents the next time step The middle This mechanism effectively promotes information exchange at different time steps and enhances the temporal information integration capability of the memory repository through a channel offset strategy, while ensuring spatial feature extraction efficiency and not significantly increasing computational complexity.

[0073] In efficient video understanding tasks, obtaining global contextual information is crucial. This invention employs a global average pooling method to extract global feature vectors from a memory repository. The mathematical expression is as follows:

[0074] ;

[0075] in, Indicates the first memory in the memory repository The feature vector of a frame, This represents the total number of memory frames.

[0076] Preferably, the global feature vector effectively captures the overall background information of the video, and enhances the ability of each frame to perceive the global context through the following operations:

[0077] ;

[0078] in, Frame-level features that enhance global contextual information.

[0079] This embodiment proposes a memory sampling strategy to reduce the number of stored historical frames. Unlike existing video compression methods, such as those based on cosine similarity to calculate the relationship between adjacent frames, this method maintains a fixed-size memory bank through random sampling, effectively reducing computational complexity. The method in this invention reduces memory bank density while utilizing the global context information of each sampled frame. This ensures that critical information is preserved without incurring additional computational overhead.

[0080] ;

[0081] in, This indicates the number of frames sampled. The proposed long spatiotemporal memory is compatible with Q-Former, thus effectively enhancing the attention mechanism for long-term modeling. By adjusting the sampling rate, this method can adapt to different task requirements, demonstrating good adaptability in practical applications and improving the model's generalization ability.

[0082] This embodiment can be seamlessly integrated into existing large-scale multimodal models to achieve efficient long video modeling capabilities.

[0083] The text decoding module utilizes Q-Former to process video frames in an autoregressive manner, gradually accumulating historical information to reduce the number of input tokens, thereby lowering the computational burden on large models. This strategy effectively alleviates the bottlenecks of large language models in terms of context window limitations and GPU memory usage.

[0084] During training, Q-Former is updated using cross-entropy loss on video-text pairs, while the large language model remains frozen, thus achieving efficient adaptive training that enables it to model video content more accurately.

[0085] The query transformer (Q-Former) aligns multi-level visual embeddings with text embeddings and uses a large language model for text decoding; the query transformer extracts temporal information from the video through cross-attention layers and self-attention layers.

[0086] Autoregressive decoding is performed using a Large Language Model (LLM) to generate natural language responses; the decoding process combines cross-entropy loss and knowledge distillation loss.

[0087] The model performs exceptionally well on multiple long video understanding benchmark datasets, particularly outperforming existing methods in fine-grained tasks such as relationship recognition and scene understanding. The model significantly improves the accuracy and efficiency of video understanding through an object-oriented hierarchical memory mechanism and a multi-level alignment mechanism.

[0088] This embodiment employs a combination of quantitative and qualitative evaluation to compare the performance of the proposed method with existing mainstream methods on three challenging tasks: long video recognition and video question answering. In the long video recognition task, experiments were conducted based on the widely used LVU dataset. This dataset covers a rich variety of real-world scenes and is highly representative. Specifically, the LVU dataset contains 30,000 video clips from approximately 3,000 movies, with each clip ranging from 1 to 3 minutes in length. To improve experimental efficiency, several challenging sub-tasks were selected to validate the effectiveness of the proposed method. For evaluation metrics, Top-1 classification accuracy was used as the primary metric to measure the model's recognition performance. In the video question answering task, three open-ended video question answering (VQA) datasets were selected for evaluation: MSRVTT-QA and MSVD-QA. Notably, MSRVTT-QA and MSVD-QA primarily contain short videos, with each video clip ranging from 10 to 15 seconds in length. Based on existing experimental protocols, Top-5 accuracy and recall are used as core evaluation metrics to comprehensively measure the model's performance on this task.

[0089] surface Comparison of state-of-the-art methods on the LVU dataset

[0090]

[0091] In terms of implementation details, this embodiment builds upon the framework of Instruct BLIP, following the approach described in [reference needed]. For low-level object perception, SAM2 is used to extract object masks from every 100 frames of video via zero-shot inference. To construct spatiotemporal memory, a 3×3 convolutional layer is used as the projector, and the pre-trained visual encoder ViT-G / 14 from EVA-CLIP is employed. To ensure fair comparison, the same backbone architecture as [reference needed] is used as the benchmark. Specifically, Q-Former weights from Instruct BLIP are used as the query mechanism, while Vicuna-7B serves as the large-scale language model. A cosine learning rate decay strategy is used for training, and all experiments were completed on four NVIDIA RTX 4090 GPUs. 100 frames were sampled from each video segment in the LVU dataset at a rate of 1 FPS, according to [reference needed].

[0092] As shown in Table 2, the performance of the device in this embodiment for long video recognition is compared with that of previous state-of-the-art methods on the LVU benchmark. The device in this embodiment performs excellently in the long video recognition task, with a significant improvement in average Top-1 accuracy of 3.8% compared to the benchmark method. Especially when trained for only one epoch, the device in this embodiment significantly outperforms the benchmark method.

[0093] surface Comparative analysis of state-of-the-art techniques for complex subtasks using the LVU dataset

[0094]

[0095] Table 3 shows the experimental results of the device in this embodiment for video question answering, compared with various methods, on the MSRVT-QA and MSVD-QA datasets. The device in this embodiment achieves state-of-the-art performance on these datasets, significantly outperforming recent LLM-based models. To save time, only one epoch was trained, and the device in this embodiment is compared with the benchmark model MA-LMM.

[0096] surface Comparison of state-of-the-art methods for video question answering tasks

[0097]

[0098] In terms of qualitative results, this embodiment, for example Figure 3As shown, the apparatus disclosed in this embodiment demonstrates significant advantages in achieving fine-grained long video understanding. MA-LMM models often exhibit hallucinations when handling tasks requiring detailed video understanding. In the first case, MA-LMM accurately identified the overall content of the video as a "drill," but failed to accurately distinguish specific objects and made incorrect assumptions, such as incorrectly identifying gender and misclassifying similar-looking objects as a gun and wood. Despite having the same number of input video frames, the apparatus disclosed in this embodiment effectively utilizes detailed object-level information, accurately distinguishing subtle differences, as in the third case where it correctly distinguished between "girl" and "lady." This highlights the importance of fine-grained low-level object information for long video understanding. Previous methods often struggle to align frame-level information with language without low-level object perception. The apparatus disclosed in this embodiment integrates multi-level information from the object level, frame level, and video level, enabling it to capture fine-grained object details and global semantics, providing a more comprehensive understanding of video content. Figure 4 As shown, the device disclosed in this embodiment uses a low-level object perception (LOP) module to infer the relationship between objects and answers in a video question-answering task. MA-LMM is prone to illusions when handling long video understanding and complex scenes, while the LOP module uses low-level object perception to perform accurate high-level video interpretation, thus correctly identifying the action of "chopping wood" instead of incorrectly identifying it as "fighting." This result highlights the importance of incorporating hierarchical object segmentation into the video understanding framework.

[0099] surface Ablation experiments on different modules in the relational task on the LVU dataset.

[0100]

[0101] In terms of ablation studies, Table 4 presents the ablation studies that evaluate the contributions of each component in the device disclosed in this embodiment. The baseline model MA-LMM scored 58.2%. In contrast, the complete network OHM of the device disclosed in this embodiment achieved 60.9%, exceeding the baseline by 2.7%, significantly demonstrating the effectiveness of the device disclosed in this embodiment. Removing the low-level object-aware module resulted in a slight performance degradation, indicating that capturing more refined object-level information is necessary.

[0102] Omitting the spatiotemporal memory construction in this embodiment further reduces performance, highlighting its role in maintaining temporal consistency. Excluding the long-term memory bank leads to performance degradation, further validating the importance of memory-related components in enhancing model recall and frame stability. Replacing the projector with frames and object masks in the low-level object perception module results in performance degradation. The impact of the number of objects inferred using SAM2 on projector performance was also investigated. As the number of encoded targets increased from 2 to 4, performance improved from 60.2% to 60.9%. These results demonstrate that each module makes a unique contribution to the overall model performance, with the complete apparatus disclosed in this embodiment achieving optimal performance.

[0103] Table 5 in this embodiment illustrates the performance of different design components in the device disclosed in this embodiment on the MSVD dataset. The device disclosed in this embodiment achieves a top-1 accuracy of 60.9%, exceeding the baseline and improving by 2.9%, highlighting the effectiveness of the device disclosed in this embodiment. Removing the global memory enhancement module, which aggregates global video-level information into each frame, leads to a decrease in accuracy, indicating that global enhancement effectively integrates video-wide information and improves overall performance. Removing the temporal shift strategy results in a 1.0% decrease in top-1 accuracy, proving that the temporal interaction enhancement in the memory bank is effective. Furthermore, disabling the memory sampling module, which selectively reduces the number of frames processed, performs video-level enhancement without significantly losing key information, leading to a decrease in accuracy. This highlights the module's role in reducing temporal redundancy and interference, while also accelerating the algorithm's execution.

[0104] surface Ablation experiments on relational tasks on the LVU dataset, with different long-term spatiotemporal memory modeling strategies.

[0105]

[0106] like Figure 5 As shown, this embodiment investigates the impact of long-term spatiotemporal memory bank length on performance. It was found that increasing the memory bank length improves Top-1 accuracy because a larger memory bank can retain more relevant historical information, providing richer context. However, performance plateaus when the memory bank length reaches 10. Compared to MA-LMM, OHM requires a shorter memory bank due to the integration of hierarchical spatiotemporal information at the object, frame, and video levels. This structure allows the model to capture fine-grained object changes and global video context within a shorter memory, ensuring comprehensive scene understanding. The results demonstrate that OHM effectively reduces temporal redundancy while maintaining performance, thus accelerating the algorithm, even when extending the video length.

[0107] This embodiment discloses an object-oriented hierarchical memory network device designed to improve the performance of long video understanding in multimodal large language models. Unlike existing methods, this device employs a hierarchical structure, capturing video details from the object and frame levels to the video level, effectively aligning visual content with language queries and reducing illusions. The device has achieved leading results in tasks such as video question answering and video caption generation, while also demonstrating excellent performance in optimizing memory usage. This device provides an efficient and scalable solution for processing complex, real-world video content, suitable for large language models. Future research will explore the applicability of this object-oriented hierarchical memory model to more video understanding tasks.

[0108] The channels are three channels in a special color mode. These three channels each carry different color information, and are combined with different intensities to present a rich color picture. For example, the common RGB color mode has three channels: red, green, and blue.

[0109] This embodiment has the following beneficial effects:

[0110] 1. Hierarchical Memory Mechanism Enhances Fine-Grained Understanding: Through hierarchical modeling at the object, frame, and video levels, progressive reasoning from local to global is achieved. By capturing precise details at the object level (such as position and shape) using a segmentation-based model, combined with frame-level spatiotemporal relationship modeling and video-level global information aggregation, fine-grained visual-language alignment capability is significantly improved, effectively alleviating the illusion problem (such as misidentification or fictitious objects) caused by neglecting low-level object details in traditional methods.

[0111] 2. Long-term spatiotemporal memory modeling optimizes resource efficiency: Global memory enhancement: Through spatiotemporal feature channel offset strategy and global average pooling, long video context is dynamically fused to expand the receptive field of a single frame and reduce information loss.

[0112] 3. Efficient memory sampling: A random sampling strategy is used to compress redundant frames and maintain a fixed memory library size. This reduces GPU memory usage (by about 30% compared to the baseline model) while retaining key information, thus overcoming the context length limitation of LLMs.

[0113] 4. Significantly Improved Multi-Task Performance: In long video recognition tasks (LVU dataset), the average Top-1 accuracy is improved by 3.8% compared to the baseline. In video question answering tasks, both Top-5 accuracy and recall reach state-of-the-art (SOTA) levels, for example, surpassing the baseline by 2.9% in just one training epoch on MSVD-QA. Qualitative experiments show that OHM can accurately distinguish easily confused objects (such as "drill" and "gun") and reduce action misjudgments (such as "chopping vegetables" and "fighting").

[0114] 5. Modular design and compatibility: Each module (low-level object perception, spatiotemporal memory construction, memory sampling, etc.) can be flexibly adapted to existing multimodal frameworks (such as BLIP-2), supporting efficiency-accuracy trade-offs for different tasks and enhancing the scalability of practical applications.

[0115] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0116] In this disclosure, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The block diagrams of devices, apparatuses, devices, and systems involved in this disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as "comprising," "including," "having," etc., are open-ended terms meaning "including but not limited to," and are used interchangeably with them. The terms "or" and "and" as used herein refer to the terms "and / or," and are used interchangeably with them unless the context clearly indicates otherwise. The term "such as" as used herein refers to the phrase "such as but not limited to," and is used interchangeably with it.

[0117] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.

[0118] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.

[0119] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.

[0120] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0121] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A hierarchical memory network device based on object-oriented programming, characterized in that, include: Low-level object perception module, spatiotemporal memory construction module, and long-term spatiotemporal memory modeling module; The low-level object perception module is used to extract object features from the input video through the segmentation base model; The spatiotemporal memory construction module is used to extract information about object interaction and scene by analyzing the spatial and temporal relationships between frames, thereby splicing the object features with the input original video frames along the channel dimension to generate a spatial representation of the target perception. The long-term spatiotemporal memory modeling module is used to store historical information through a memory bank, capture global dependencies at the video level, and stitch together multiple adjacent video frames output by the spatiotemporal memory construction module in the channel dimension to form a local multi-frame spatiotemporal representation. The long-term spatiotemporal memory modeling module enhances the global context awareness capability of each frame through a global memory enhancement strategy, calculated as follows: ； in, Indicates the first memory in the memory repository The feature vector of a frame, Total number of memory frames, Represents the global context information for each sampled frame; The long-term spatiotemporal memory modeling module optimizes time interaction through a time channel offset strategy; The time channel offset strategy implements timing offset using the following formula: ； in, Indicates the previous time step The former One channel, Represents the current time step The end One channel, and Represents the next time step The middle One channel, This is the timing offset.

2. The object-oriented hierarchical memory network device according to claim 1, characterized in that, The low-level object perception module is used to perform object perception on the input video using the segmentation base model, extract the target mask, and thus extract object features.

3. The object-oriented hierarchical memory network device according to claim 2, characterized in that, The spatiotemporal memory construction module generates a spatial representation of the target perception by splicing the target mask with the input original video frame along the channel dimension.

4. The object-oriented hierarchical memory network device according to claim 3, characterized in that, The long-term spatiotemporal memory modeling module forms a local multi-frame temporal representation by splicing multiple adjacent video frames along the channel dimension and aggregating spatiotemporal features.

5. The object-oriented hierarchical memory network device according to claim 4, characterized in that, The spatiotemporal memory construction module uses a pre-trained image model to extract frame-level visual features and incorporates temporal information through positional encoding.

6. The object-oriented hierarchical memory network device according to claim 5, characterized in that, The long-term spatiotemporal memory modeling module splices the extracted spatiotemporal features into a long-term spatiotemporal memory library.

7. The object-oriented hierarchical memory network device according to claim 1, characterized in that, The spatiotemporal memory construction module reduces the number of historical frames stored by randomly sampling video frames.

8. The object-oriented hierarchical memory network device according to claim 7, characterized in that, The long-term spatiotemporal memory modeling module integrates features from multiple frames by storing historical information extracted from video frames. At the current time step, the long-term spatiotemporal memory modeling module accumulates and stores global video information by splicing the spatiotemporal features of multiple video frames.

Images