Memory-enhanced time-sensitive model control method for super-long video understanding

Abstract

A memory-enhanced time-sensitive model control method for super-long video understanding is applied to the field of super-long video processing, and comprises the following steps: dynamically sampling an input original video frame sequence to obtain a compressed first frame sequence, the first frame sequence comprising a selected frame obtained by dynamic sampling and a corresponding timestamp; performing time dynamic storage on the first frame sequence according to the sliding window and a first-in-first-out architecture to obtain a frame representation set which is located in a memory buffer area and has time perception and an output exceeding frame representation; based on the sliding window, performing space-time weighted integration on the frame representation set and the exceeding frame representation to generate a comprehensive frame representation; and performing iterative coding according to the comprehensive frame representation and the user query to obtain a response to the user query. According to the method, the calculation burden is reduced, the time sensitivity is improved, the understanding and reasoning of a long-time video are enhanced, the overall performance and accuracy of the model are improved, and the excellent zero sample learning ability is shown.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing, in particular to a memory-enhanced time-sensitive model control method for ultra-long video understanding. BACKGROUND

[0002] In the field of ultra-long videos, a major challenge is to accurately locate key information and temporal events in these videos, especially under limited computing resources.

[0003] There are existing video large language models for long video understanding, such as MovieChat and TimeChat. MovieChat is a framework composed of a fast-updating short-term memory and a stable long-term memory, which can represent more frames. However, the memory module is only a storage component and is not used for joint training with other modules. In addition, the difference between the video distributions used in the training and inference processes results in suboptimal quality of the video features inferred by the model. And the video distributions used for training and inference are also different, resulting in poor quality of the video features inferred by the model, and the effect is also affected by the existing model. In addition, TimeChat proposes a sliding video Q-former to preserve more rich semantics for long videos. However, its method may reduce its effectiveness in understanding ultra-long videos.

[0004] In summary, the existing technology has several major drawbacks when dealing with ultra-long videos:

[0005] Lack of time sensitivity: Existing models such as MovieChat and TimeChat often fail to effectively capture and interpret temporal relationships in long videos, resulting in insufficient information integration capabilities over long periods of time.

[0006] Computational burden: Learning representations directly from long videos increases the computational burden, as more frames need to be processed, which can lead to a decrease in model performance.

[0007] Limitations of memory mechanisms: Although some models introduce memory mechanisms, these mechanisms often serve only as storage components and fail to be jointly trained with other modules, thereby limiting the overall performance of the model. SUMMARY

[0008] At least one embodiment of the present application provides a memory-enhanced time-sensitive model control method for ultra-long video understanding, which solves the problems of lack of time sensitivity, heavy computational burden, and low overall performance in the prior art when dealing with ultra-long videos.

[0009] To solve the above technical problems, the present application is implemented as follows:

[0010] In a first aspect, the embodiments of the present application provide a memory-enhanced time-sensitive model control method for ultra-long video understanding, comprising:

[0011] dynamically sampling an input original video frame sequence to obtain a compressed first frame sequence, the first frame sequence comprising selected frames obtained by dynamic sampling and corresponding timestamps;

[0012] performing time dynamic storage on the first frame sequence according to a sliding window and a first-in-first-out architecture to obtain a frame representation set with time awareness located in a memory buffer and an output beyond frame representation;

[0013] based on the sliding window, spatiotemporal weighted integration is performed on the frame representation set and the beyond frame representation to generate a comprehensive frame representation;

[0014] iterative encoding is performed according to the comprehensive frame representation and a user query to obtain a response to the user query.

[0015] Preferably, the method as described above, the dynamic sampling of the input original video frame sequence to obtain the compressed first frame sequence comprises:

[0016] segmenting the original video frame sequence into multiple segments according to a first time interval;

[0017] randomly sampling a preselected frame for each of the segments to obtain a preselected frame sequence;

[0018] texture feature extraction is performed on each of the preselected frames to obtain a texture saliency distribution;

[0019] the probability of each of the preselected frames is determined according to a preset cumulative texture distribution function and a time accumulation curve;

[0020] the preselected frame sequence is equally divided into a preset number of sequence segments;

[0021] a preselected frame is randomly selected from each of the sequence segments as the selected frame to obtain the first frame sequence.

[0022] Specifically, the method as described above, the texture feature extraction performed on each of the preselected frames to obtain a texture saliency distribution comprises:

[0023] texture features corresponding to each of the preselected frames and a texture distribution histogram are obtained through texture feature extraction;

[0024] the texture distribution histogram difference corresponding to each adjacent preselected frame is obtained;

[0025] a texture distribution histogram difference sum is obtained according to each of the texture distribution histogram differences;

[0026] The texture distribution histogram difference is normalized to obtain the texture saliency distribution.

[0027] Preferably, the method as described above, the time dynamic storage of the first frame sequence according to the sliding window and the first-in-first-out architecture obtains a set of time-aware frame representations located in the memory buffer and an output of the frame representation beyond, including:

[0028] Binding the visual mark and the timestamp corresponding to each frame in the first frame sequence;

[0029] Mapping each frame after binding to a plurality of image embedding vectors to obtain the corresponding frame representation;

[0030] Caching the frame representation according to the sliding window to obtain a set of frame representations, and outputting the frame representation in advance through the first-in-first-out architecture to obtain the frame representation beyond.

[0031] Preferably, the method as described above, the spatiotemporal weighted integration of the set of frame representations and the frame representation beyond based on the sliding window generates a comprehensive frame representation, including:

[0032] Performing temporal weighted average on the frames in each sliding window through a preset temporal weighted average algorithm to obtain a temporal representation of each sliding window;

[0033] Performing visual feature extraction on the frames in each sliding window to obtain image embedding vectors corresponding to each frame;

[0034] According to the activation function, the cosine similarity corresponding to the image embedding vectors in the same sliding window is normalized, and the frames in each sliding window are spatially weighted and summed to obtain a spatial representation of each sliding window;

[0035] Fusing the temporal representation and the spatial representation corresponding to the same sliding window to obtain the comprehensive frame representation.

[0036] Preferably, the method as described above, the iterative encoding according to the comprehensive frame representation and the user query obtains a response to the user query, including:

[0037] Iterative encoding of the user query and the comprehensive frame representation corresponding to each sliding window as a resampling input into a preset video large language model to obtain a preset number of visual outputs;

[0038] Integrating all the visual outputs to obtain a response to the user query.

[0039] Preferably, the method as described above, further comprising:

[0040] Multimodal alignment of the video large language model based on video and text input;

[0041] The video large language model is fine-tuned based on low-rank adaptation.

[0042] Secondly, this application provides a memory-enhanced time-sensitive model control device for ultra-long video understanding, comprising:

[0043] The dynamic sampling module is used to dynamically sample the input original video frame sequence to obtain a compressed first frame sequence, which includes the selected frame obtained by dynamic sampling and the corresponding timestamp.

[0044] The time dynamic caching module is used to perform time dynamic storage on the first frame sequence according to the sliding window and first-in-first-out architecture, so as to obtain a time-aware frame representation set located in the memory buffer and the output overflow frame representation.

[0045] An integration module is used to perform spatiotemporal weighted integration of the frame representation set and the extra frame representation based on the sliding window to generate a comprehensive frame representation;

[0046] The iterative encoding module is used to perform iterative encoding based on the comprehensive frame representation and the user query to obtain a response to the user query.

[0047] Thirdly, this application provides an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, it implements the steps of the memory-enhanced time-sensitive model control method for ultra-long video understanding as described above.

[0048] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the memory-enhanced time-sensitive model control method for ultra-long video understanding as described above.

[0049] Fifthly, this application provides a computer program product, including computer instructions that, when executed by a processor, implement the steps of the memory-enhanced time-sensitive model control method for ultra-long video understanding as described above.

[0050] Compared with the prior art, the memory-enhanced time-sensitive model control method for ultra-long video understanding provided in this application has at least the following technical effects:

[0051] Dynamic sampling intelligently selects representative frames and retains timestamp information, reducing computational burden and improving time sensitivity. By storing fixed-length frame representations based on a sliding window, it enhances the understanding of long-duration videos. Spatiotemporal integration based on information in the memory buffer improves the overall performance and accuracy of the model. Iterative encoding based on user queries further enhances the model's deep understanding and reasoning, and demonstrates excellent zero-shot learning capabilities. Attached Figure Description

[0052] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0053] Figure 1 This is one of the flowcharts illustrating the memory-enhanced time-sensitive model control method for ultra-long video understanding according to an embodiment of this application;

[0054] Figure 2 This is a second schematic flowchart of the memory-enhanced time-sensitive model control method for ultra-long video understanding according to an embodiment of this application;

[0055] Figure 3 This is the third flowchart illustrating the memory-enhanced time-sensitive model control method for ultra-long video understanding according to an embodiment of this application.

[0056] Figure 4 This is the fourth flowchart illustrating the memory-enhanced time-sensitive model control method for ultra-long video understanding according to an embodiment of this application.

[0057] Figure 5 This is the fifth flowchart illustrating the memory-enhanced time-sensitive model control method for ultra-long video understanding according to an embodiment of this application.

[0058] Figure 6 This is the sixth flowchart illustrating the memory-enhanced time-sensitive model control method for ultra-long video understanding according to an embodiment of this application.

[0059] Figure 7 This is a schematic flowchart of a memory-enhanced time-sensitive model control device for ultra-long video understanding, according to an embodiment of this application. Detailed Implementation

[0060] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0061] The terms “first,” “second,” etc., used in the specification and claims of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus. The terms “and / or” in the specification and claims indicate at least one of the connected objects.

[0062] The technologies described in this document are not limited to New Radio (NR) systems and Long Time Evolution (LTE) / LTE-Advanced (LTE-A) systems, but can also be used in various wireless communication systems such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal Frequency Division Multiple Access (OFDMA), Single-carrier Frequency-Division Multiple Access (SC-FDMA), and other systems. The terms "system" and "network" are often used interchangeably. CDMA systems can implement radio technologies such as CDMA2000 and Universal Terrestrial Radio Access (UTRA). UTRA includes Wideband Code Division Multiple Access (WCDMA) and other CDMA variants. TDMA systems can implement radio technologies such as the Global System for Mobile Communication (GSM). OFDMA systems can implement radio technologies such as Ultra Mobile Broadband (UMB), Evolution-UTRA (E-UTRA), IEEE 802.21 (Wi-Fi), IEEE 802.16 (WiMAX), IEEE 802.20, and Flash-OFDM. UTRA and E-UTRA are parts of the Universal Mobile Telecommunications System (UMTS). LTE and more advanced versions of LTE (such as LTE-A) are newer UMTS versions using E-UTRA. UTRA, E-UTRA, UMTS, LTE, LTE-A, and GSM are described in documents from an organization called the 3rd Generation Partnership Project (3GPP). CDMA2000 and UMB are described in documents from an organization called 3rd Generation Partnership Project 2 (3GPP2).The techniques described herein can be used with the systems and radio technologies mentioned above, as well as with other systems and radio technologies. However, the following description describes NR systems for illustrative purposes, and NR terminology is used in most of the following description, although these techniques can also be applied to applications beyond NR systems.

[0063] The following description provides examples and is not intended to limit the scope, applicability, or configuration set forth in the claims. Changes may be made to the function and arrangement of the elements discussed without departing from the spirit and scope of this disclosure. Various procedures or components may be appropriately omitted, substituted, or added to the examples. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Furthermore, features described with reference to certain examples may be combined in other examples.

[0064] Please refer to Figure 1 This application provides a memory-enhanced time-sensitive model control method for ultra-long video understanding, comprising:

[0065] Step S101: Dynamically sample the input original video frame sequence to obtain a compressed first frame sequence, the first frame sequence including the selected frame obtained by dynamic sampling and the corresponding timestamp.

[0066] Step S102: Based on the sliding window and first-in-first-out architecture, the first frame sequence is dynamically stored in time to obtain a time-aware set of frame representations located in the memory buffer and an output of the overflow frame representation.

[0067] Step S103: Based on the sliding window, perform spatiotemporal weighted integration on the frame representation set and the excess frame representation to generate a comprehensive frame representation;

[0068] Step S104: Based on the comprehensive frame representation and the user query, perform iterative encoding to obtain a response to the user query.

[0069] The video large language model provided in this embodiment, when understanding ultra-long videos, dynamically samples the original video frame sequence (more than 10,000 frames) to obtain the first frame sequence. It can intelligently select a group of representative frames, which helps reduce redundant information and optimize subsequent processing, while retaining timestamp information to ensure time sensitivity. Furthermore, a fixed-length frame representation is dynamically stored in a memory buffer using a sliding window, resulting in a time-aware frame representation set. Based on a first-in-first-out (FIFO) architecture, when the memory buffer capacity is reached, the previously stored data is output as the overflow frame representation, ensuring effective management and utilization of time information. This ensures that the buffer retains a dynamic and coherent representation of the most recent video frame. Further, based on the sliding window, the time-aware frame representation set and the overflow frame representation are integrated with temporal and spatial information to obtain a comprehensive frame representation. Based on the obtained comprehensive frame representation and the user's input query, iterative encoding is performed to obtain the information required by the user and complete the response to the user's query.

[0070] In summary, this application reduces computational burden and improves temporal sensitivity by dynamically sampling and intelligently selecting representative frames while retaining timestamp information. It enhances the understanding of long-duration videos by storing fixed-length frame representations based on sliding windows. It improves the overall performance and accuracy of the model by performing spatiotemporal integration based on information in the memory buffer. Furthermore, it enhances the model's deep understanding and reasoning by performing iterative encoding based on user queries, and demonstrates excellent zero-shot learning capabilities.

[0071] See Figure 2 Preferably, in the method described above, the step of dynamically sampling the input original video frame sequence to obtain the compressed first frame sequence includes:

[0072] Step S201: Divide the original video frame sequence into multiple segments according to the first time interval;

[0073] Step S202: For each segment, randomly sample a pre-selected frame to obtain a pre-selected frame sequence;

[0074] Step S203: Extract texture features for each of the preselected frames and obtain the texture saliency distribution;

[0075] Step S204: Determine the probability of each of the pre-selected frames based on the preset cumulative texture distribution function and time cumulative curve;

[0076] Step S205: Divide the preselected frame sequence into a preset number of sequence segments;

[0077] Step S206: Randomly select a pre-selected frame from each of the sequence segments as the selected frame to obtain the first frame sequence.

[0078] In this embodiment, for ease of understanding by those skilled in the art, an example of the dynamic sampling step is provided. The original video is considered as visual input V, representing a sequence of N red, green, blue (RGB) video frames, each frame having a size of H × W. The video V is divided into K segments of equal duration according to a first time interval, represented as follows: This allows you to extract segments of video over a given period of time. Further processing of each segment... Randomly sample a frame As a representative frame within that time period, a sequence of frames can be obtained. This involves pre-selecting a frame sequence, which can be extracted to effectively compress ultra-long videos for further analysis. Next, texture feature extraction is performed on these selected frames to obtain the corresponding texture saliency distribution, capturing the changes in texture, boundaries, and spatial variations between consecutive frames. To adaptively sample frames based on the texture distribution, temporal segmentation is further performed using the cumulative texture distribution. Specifically, a cumulative curve is constructed along the time dimension, and a hyperparameter μ is introduced to adjust the original feature distribution. This is used to control the smoothness of the sampling. Specifically, a preset cumulative texture distribution function, denoted as F(x) = P(X ≤ x), is defined for all real numbers x and represents the probability that the value of the random variable X is less than or equal to x. Based on this cumulative texture distribution function, the pre-selected frame sequence is divided into M equal sequence segments, ensuring that each sequence segment has equal probability quality. From each segment, a value is randomly selected, corresponding to the index of the frame to be sampled, thus obtaining the desired first frame sequence. This ensures a diverse representation of texture features within the sampled subset.

[0079] See Figure 3 Specifically, in the method described above, the step of extracting texture features for each of the pre-selected frames and obtaining the texture saliency distribution includes:

[0080] Step S301: By extracting texture features, obtain the texture features and texture distribution histograms corresponding to each of the preselected frames;

[0081] Step S302: Obtain the texture distribution histogram differences corresponding to each adjacent preselected frame;

[0082] Step S303: Based on the differences in the texture distribution histograms, obtain the sum of texture distribution histogram differences;

[0083] Step S304: Normalize the difference in the texture distribution histogram to obtain the texture saliency distribution.

[0084] This embodiment illustrates the above-described texture feature extraction steps, using Local Binary Pattern (LBP) features as an example of texture features. For each frame... Calculate its texture distribution histogram, using F( This process allows us to capture changes in texture, boundaries, and spatial variations between consecutive frames. Among these, adjacent frames... and The difference in texture histograms between them can be represented as: .

[0085] Furthermore, the distribution histogram differences are calculated based on the differences in the texture distribution histograms: ,in, ∈ R, representing the texture size per frame. = 0. Further use Norm pairs Normalization yields the texture saliency distribution. ,Right now .

[0086] See Figure 4 Preferably, in the method described above, the step of dynamically storing the first frame sequence in time according to a sliding window and a first-in-first-out architecture to obtain a time-aware set of frame representations located in a memory buffer and an output of overflow frame representations includes:

[0087] Step S401: Bind according to the visual marker and timestamp corresponding to each frame in the first frame sequence;

[0088] Step S402: Map each bound frame to multiple image embedding vectors to obtain the corresponding frame representation;

[0089] Step S403: Cache the frame representation according to the sliding window to obtain a set of frame representations, and output the first frame representation through the first-in-first-out architecture to obtain the excess frame representation.

[0090] In this embodiment, the time-dynamic storage step of the first frame sequence is illustrated. After dynamic sampling, the resulting first frame sequence, with a length of M and dimensions H × W, represents the long video V. Then, a frozen visual encoder is used to extract visual features from each frame. This frozen visual encoder consists of a frozen pre-trained computer vision model (e.g., Vision Transformer (ViT) based on the Transformer architecture) and a trainable multimodal model architecture (e.g., time-aware Q-former). The time-aware Q-former adds a corresponding timestamp to each frame; for example, "this frame was sampled at 4 seconds" serves as a condition for the Q-former to fuse visual and timestamp information, binding time information to visual markers using timestamp descriptions. Further, the visual encoder maps each frame to... Each image embedding vector generates a video frame representation. ,in Represents the corresponding i The i-th frame is embedded in the dimensional image set. Then, the video frame representation is stored in a memory buffer that can store fixed-length frame markers using a sliding window method. The sliding window-based storage method can be represented as:

[0091]

[0092] in, Let be the j-th memory buffer feature, C be the number of frames in each sliding window, and M be the total number of frames in the first frame sequence. Furthermore, the memory buffer employs a first-in, first-out (FIFO) architecture. When the memory buffer reaches its capacity, the previously stored frames are unloaded and output as out-of-bounds frame representations. Then, the memory buffer is reset based on the sliding window, specifically including but not limited to outputting all frame representations in the entire memory buffer as out-of-bounds frame representations, or outputting the frame representation corresponding to a single sliding window in the memory buffer as an out-of-bounds frame representation. This mechanism ensures that the memory buffer retains a dynamic and coherent representation of the most recent video frame, guaranteeing the system's ability to engage in meaningful dialogue about the video content.

[0093] See Figure 5 Preferably, in the method described above, the step of performing spatiotemporal weighted integration of the frame representation set and the excess frame representation based on the sliding window to generate a comprehensive frame representation includes:

[0094] Step S501: Using a preset time-weighted averaging algorithm, perform a time-weighted average on the frames within each sliding window to obtain the time representation of each sliding window;

[0095] Step S502: Visual features are extracted from the frames within each sliding window to obtain the image embedding vector corresponding to each frame;

[0096] Step S503: Normalize the cosine similarity of the image embedding vectors within the same sliding window according to the activation function, and perform spatial weighted summation on the frames within each sliding window to obtain the spatial representation of each sliding window;

[0097] Step S504: Merge the temporal representation and the spatial representation corresponding to the same sliding window to obtain the comprehensive frame representation.

[0098] In this embodiment, the spatiotemporal weighted integration step is illustrated, wherein, due to the pre-established memory buffer... A "time representation" was designed. At this point, a time-weighted average is applied to the frames in the memory buffer or sliding window. This comprehensive representation retains temporal information and focuses more on the most recent frames. The formula can be expressed as: ;in, , , and Designed as an index: Specifically, i is the index of the memory buffer or sliding window, counting from 1. C is the length of the memory buffer or sliding window. α is a smoothing factor, ranging from 0 to 1, and in a specific embodiment, it is 0.5.

[0099] On the other hand, "spatial representation" Cosine similarity is another key component for representing spatial information. By normalizing the cosine similarity using an activation function (e.g., softmax) and performing a weighted summation of frames in the memory buffer, it is ensured that frames with higher similarity values ​​contribute more to the final representation. Therefore, "spatial representation"... The spatial information of the frame is stored in the memory buffer. In this process, the temporal and spatial representations are collaboratively integrated. This integration produces a holistic and rich representation of the memory buffer, encapsulating temporal dynamics and spatial dependencies. With the sliding window matching the length of the memory buffer, the memory buffer can be cleared and new frames pushed into it after the merging process is complete. Furthermore, compressed representations can be cached in each memory buffer. Additionally, when processing the current memory buffer, the unified representation cached in each previous memory buffer can be accessed.

[0100] See Figure 6 Preferably, in the method described above, the step of iteratively encoding based on the comprehensive frame representation and the user query to obtain a response to the user query includes:

[0101] Step S601: The user query and the comprehensive frame representation corresponding to each sliding window are used as resampling input to a preset video large language model for iterative encoding to obtain a preset number of visual outputs;

[0102] Step S602: Integrate all the visual outputs to obtain a response to the user query.

[0103] In this embodiment, the iterative encoding steps are illustrated, where, considering that each dynamic memory buffer has been merged, a perceptron resampler is employed. It takes variable video frame features from the visual encoder and the user query as input and produces a fixed number of visual outputs, reducing the computational complexity of visual text cross-attention. When the memory buffer is updated, a compressed representation of the previous memory buffer and frame features is obtained in the new memory buffer, which can be considered as input to the perceptron resampler. This process is repeated in each updated memory buffer until the entire video sequence is processed. In this way, each updated buffer is part of the algorithm's iterative process, ensuring that the tracking adapts to dynamic changes in objects within the video.

[0104] During iterative encoding, a predefined number of potential input queries are learned and fed into deep learning model architectures such as the Transformer, with cross-conceptual attention paid to visual features. Combining video frame representations with user queries facilitates deep understanding and reasoning using large language models.

[0105] Preferably, the method described above further includes:

[0106] Multimodal alignment of the video large language model based on video and text input;

[0107] The video large language model is fine-tuned based on low-rank adaptation.

[0108] This embodiment also provides an example of training a video big language model, wherein the video big language model can connect inputs from different modes, including video tags. and text query tags All of these tags have the same embedding dimension. By maximizing the likelihood probability in the equation, the model ultimately achieves multimodal understanding. The model function can be expressed as:

[0109]

[0110] Where L is the generated sequence The length of θ is a trainable parameter.

[0111] Taking a two-stage training approach as an example, to improve computational efficiency, the pre-training checkpoints of the open-source model are reused in the first stage of training to enable the model to learn, understand, and generate text descriptions related to visual content. In the second stage of training, the instruction tuning stage, the model is fine-tuned using instruction data, allowing it to obtain appropriate responses to different instructions. That is, after the first stage of training, the model's parameters are frozen, and fine-tuning is only performed on the video and time-related parts. To better adapt the video large language model to video tasks, this embodiment also applies a low-rank adaptation (LoRA) fine-tuning method. By introducing a low-rank matrix, the number of parameters in the context is reduced, thus maintaining high efficiency and low computational requirements during fine-tuning. This lays the foundation for the development of future video analysis tools, promotes the progress of multimodal learning, and has broad application prospects.

[0112] The various methods of the embodiments of this application have been described above. Apparatus for implementing the above methods will now be provided.

[0113] See Figure 7 Another embodiment of this application provides a memory-enhanced time-sensitive model control device for ultra-long video understanding, comprising:

[0114] The dynamic sampling module 701 is used to dynamically sample the input original video frame sequence to obtain a compressed first frame sequence, the first frame sequence including the selected frame obtained by dynamic sampling and the corresponding timestamp.

[0115] The time dynamic caching module 702 is used to perform time dynamic storage on the first frame sequence according to the sliding window and first-in-first-out architecture, so as to obtain a time-aware frame representation set located in the memory buffer and the output overflow frame representation.

[0116] Integration module 703 is used to perform spatiotemporal weighted integration of the frame representation set and the excess frame representation based on the sliding window to generate a comprehensive frame representation;

[0117] The iterative encoding module 704 is used to perform iterative encoding based on the comprehensive frame representation and the user query to obtain a response to the user query.

[0118] Preferably, in the apparatus described above, the dynamic sampling module includes:

[0119] The first segmentation unit is used to segment the original video frame sequence into multiple segments according to a first time interval;

[0120] The first sampling unit is used to randomly sample a pre-selected frame for each of the segments to obtain a pre-selected frame sequence;

[0121] The first feature extraction unit is used to extract texture features for each of the preselected frames and obtain the texture saliency distribution;

[0122] The probability determination unit is used to determine the probability of each of the preselected frames based on a preset cumulative texture distribution function and a time cumulative curve.

[0123] The second segmentation unit is used to divide the preselected frame sequence into a preset number of sequence segments;

[0124] The second sampling unit is used to randomly select a pre-selected frame from each of the sequence segments as the selected frame to obtain the first frame sequence.

[0125] Specifically, in the apparatus described above, the first feature extraction unit includes:

[0126] The second processing subunit is used to extract texture features and obtain texture features and texture distribution histograms corresponding to each of the preselected frames through texture feature extraction.

[0127] The third processing subunit is used to obtain the texture distribution histogram differences corresponding to each adjacent preselected frame;

[0128] The fourth processing subunit is used to obtain the texture distribution histogram difference sum based on the differences in each texture distribution histogram;

[0129] The fifth processing subunit is used to normalize the differences in the texture distribution histogram to obtain the texture saliency distribution.

[0130] Preferably, the time-dynamic caching module of the apparatus described above includes:

[0131] The first processing module is used to bind according to the visual markers and timestamps corresponding to each frame in the first frame sequence;

[0132] The second processing module is used to map each bound frame to multiple image embedding vectors to obtain the corresponding frame representation;

[0133] The third processing module is used to cache the frame representation according to the sliding window to obtain a set of frame representations, and output the first frame representation through the first-in-first-out architecture to obtain the excess frame representation.

[0134] Preferably, in the apparatus described above, the integrated module includes:

[0135] The fourth processing module is used to perform a time-weighted average of the frames within each sliding window using a preset time-weighted average algorithm to obtain the time representation of each sliding window;

[0136] The fifth processing module is used to extract visual features from the frames within each sliding window to obtain the image embedding vector corresponding to each frame.

[0137] The sixth processing module is used to normalize the cosine similarity of the image embedding vectors within the same sliding window according to the activation function, and to perform spatial weighted summation on the frames within each sliding window to obtain the spatial representation of each sliding window;

[0138] The seventh processing module is used to fuse the temporal representation and the spatial representation corresponding to the same sliding window to obtain the comprehensive frame representation.

[0139] Preferably, in the apparatus described above, the iterative encoding module comprises:

[0140] The eighth processing module is used to take the user query and the comprehensive frame representation corresponding to each sliding window as resampling input to a preset video large language model for iterative encoding to obtain a preset number of visual outputs;

[0141] The ninth processing module is used to integrate all the visual outputs to obtain a response to the user query.

[0142] Preferably, in the apparatus described above, the iterative encoding module further includes:

[0143] The tenth processing module is used to perform multimodal alignment of the video large language model based on video and text input;

[0144] The eleventh processing module is used to fine-tune the video large language model based on low-rank adaptation.

[0145] It should be noted that the control device in this embodiment corresponds to the memory-enhanced time-sensitive model control method for ultra-long video understanding described above. The implementation methods in the above embodiments are all applicable to the embodiments of this device and can achieve the same technical effect. The device provided in this application embodiment can implement all the method steps implemented in the above method embodiments and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiments and the beneficial effects will not be described in detail.

[0146] Another embodiment of this application provides an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor. When the program is executed by the processor, it implements the steps of the memory-enhanced time-sensitive model control method for ultra-long video understanding as described above, and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0147] Another embodiment of this application provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the steps of the memory-enhanced time-sensitive model control method for ultra-long video understanding described above, and achieves the same technical effect. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0148] Another embodiment of this application provides a computer program product, including computer instructions that, when executed by a processor, implement the steps of the memory-enhanced time-sensitive model control method for ultra-long video understanding as described above, and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0149] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0150] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A memory-enhanced time-sensitive model control method for ultra-long video understanding, characterized in that, include: The input original video frame sequence is dynamically sampled to obtain a compressed first frame sequence, which includes the selected frame obtained by dynamic sampling and the corresponding timestamp. Based on the sliding window and first-in-first-out architecture, the first frame sequence is dynamically stored in time to obtain a time-aware set of frame representations located in the memory buffer and an output of the overflow frame representations. Based on the sliding window, the frame representation set and the super-frame representation are spatiotemporally weighted and integrated to generate a comprehensive frame representation; Based on the comprehensive frame representation and the user query, iterative encoding is performed to obtain the response to the user query.

2. The method according to claim 1, characterized in that, The step of dynamically sampling the input original video frame sequence to obtain the compressed first frame sequence includes: The original video frame sequence is divided into multiple segments according to the first time interval; For each of the segments, a pre-selected frame is randomly sampled to obtain a pre-selected frame sequence; Texture features are extracted for each of the preselected frames, and the texture saliency distribution is obtained; The probability of each preselected frame is determined based on the preset cumulative texture distribution function and time cumulative curve; The preselected frame sequence is divided into a preset number of sequence segments; Randomly select a pre-selected frame from each of the sequence segments as the selected frame to obtain the first frame sequence.

3. The method according to claim 2, characterized in that, The step of extracting texture features from each of the preselected frames and obtaining the texture saliency distribution includes: By extracting texture features, the texture features and texture distribution histograms corresponding to each of the preselected frames are obtained; Obtain the texture distribution histogram differences corresponding to each adjacent preselected frame; Based on the differences in the texture distribution histograms, the sum of texture distribution histogram differences is obtained; The difference in the texture distribution histogram is normalized to obtain the texture saliency distribution.

4. The method according to claim 1, characterized in that, The step of dynamically storing the first frame sequence in time according to the sliding window and first-in-first-out architecture to obtain a time-aware set of frame representations in the memory buffer and an output of the overflow frame representation includes: Binding is performed based on the visual markers and timestamps corresponding to each frame in the first frame sequence; Each bound frame is mapped to multiple image embedding vectors to obtain the corresponding frame representation; The frame representations are cached using the sliding window to obtain a set of frame representations, and the first frame representation is output using the first-in-first-out architecture to obtain an overflow frame representation.

5. The method according to claim 1, characterized in that, The step of performing spatiotemporal weighted integration of the frame representation set and the excess frame representations based on the sliding window to generate a comprehensive frame representation includes: A time-weighted average is performed on the frames within each sliding window using a preset time-weighted average algorithm to obtain the time representation of each sliding window. Visual features are extracted from the frames within each sliding window to obtain the image embedding vector corresponding to each frame. The cosine similarity of the image embedding vectors within the same sliding window is normalized according to the activation function, and the frames within each sliding window are spatially weighted and summed to obtain the spatial representation of each sliding window. The combined frame representation is obtained by fusing the temporal representation and the spatial representation corresponding to the same sliding window.

6. The method according to claim 1, characterized in that, The step of iteratively encoding based on the comprehensive frame representation and the user query to obtain a response to the user query includes: The user query and the comprehensive frame representation corresponding to each sliding window are used as resampling input to a preset video large language model for iterative encoding to obtain a preset number of visual outputs. Integrate all the visual outputs to obtain a response to the user query.

7. The method according to claim 6, characterized in that, Also includes: Multimodal alignment of the video large language model based on video and text input; The video large language model is fine-tuned based on low-rank adaptation.

8. A memory-enhanced time-sensitive model control device for ultra-long video understanding, characterized in that, include: The dynamic sampling module is used to dynamically sample the input original video frame sequence to obtain a compressed first frame sequence, which includes the selected frame obtained by dynamic sampling and the corresponding timestamp. The time dynamic caching module is used to perform time dynamic storage on the first frame sequence according to the sliding window and first-in-first-out architecture, so as to obtain a time-aware frame representation set located in the memory buffer and the output overflow frame representation. An integration module is used to perform spatiotemporal weighted integration of the frame representation set and the extra frame representation based on the sliding window to generate a comprehensive frame representation; The iterative encoding module is used to perform iterative encoding based on the comprehensive frame representation and the user query to obtain a response to the user query.

9. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the memory-enhanced time-sensitive model control method for ultra-long video understanding as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the memory-enhanced time-sensitive model control method for ultra-long video understanding as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps of the memory-enhanced time-sensitive model control method for ultra-long video understanding as described in any one of claims 1 to 7.

Images