Memory perception based weakly supervised online video temporal instance localization method and system

Abstract

The present application relates to a memory-aware based weakly supervised online video moment localization method and system, belonging to the field of artificial intelligence technology, comprising: multi-modal feature fusion on a given video and its text query to obtain unified frame-level representation at each stage; using an offline-guided online model architecture, the fused features are input into the offline and online modules in the form of the whole and frame by frame; in the offline module, a Gaussian mask is generated to reconstruct the hidden part of the query, obtaining the proposal of the action starting moment; in the online module, the long-term historical memory in the window is used for enhancement, and the attention weight in the window is dynamically generated, and the score of the current frame is calculated by weighting; the proposal obtained by the offline module is used as a pseudo label to provide supervision information for the score sequence of the online module; only the online module needs to be inferred separately, and the weakly supervised online moment localization with high performance is completed. The present application significantly improves the expansion capability and application value of the model.

Description

TECHNICAL FIELD

[0001] The present application relates to a weakly supervised online video moment positioning method and system based on memory perception, and belongs to the technical field of artificial intelligence. BACKGROUND

[0002] With the popularization of intelligent perception terminals and the penetration of digital lifestyle, the application scenarios and data volume of online videos have grown dramatically. Video moment positioning, as a key technology in the field of video analysis, plays an important role in many practical scenarios.

[0003] In the field of intelligent security, video moment positioning can monitor and screen abnormal behaviors in surveillance videos in real time, such as illegal driving and fighting, providing strong support for public safety. In industrial production, by locating the actions of workers in real time, it can be determined whether their behavior conforms to the operation specification, thereby improving production efficiency while increasing safety protection. In the field of education, by quickly locating the content of teaching videos, it promotes the effective dissemination of knowledge and the improvement of learning efficiency.

[0004] Traditional online video moment positioning tasks are often based on complete start time label information. This means that during the model training phase, annotators must mark the detailed behavior boundaries of video data. However, the process of video annotation using fully supervised learning method not only consumes time and effort, but also significantly increases the cost of algorithm development. In addition, the subjectivity of the annotation process may lead to differences in the annotation results between different annotators, which further limits the scalability of the video moment behavior positioning model and restricts its application range in practical scenarios. In the face of these contradictions, weakly supervised learning, with its compatibility with video-level labels, has become a key path to break through the bottleneck of massive low-annotation video analysis.

[0005] Through online video moment positioning under weak supervision, not only does it deepen the research of traditional video action positioning, but also has stronger scalability, which can cope with the growing scale of video data. Currently, this research mainly faces the following two challenges:

[0006] (1) In the absence of frame-level annotation, traditional sample mining strategies are difficult to apply, and an effective pseudo-supervision mechanism needs to be established. Due to the lack of fine-grained annotation signals, the model is difficult to accurately determine the semantic boundaries of video segments, and is easily disturbed by background segments or multiple events, leading to unstable training and decreased positioning performance. Therefore, it is urgent to build a mechanism that can generate high-quality pseudo-labels from video-text weak supervision signals to provide more reliable learning signals and enhance the robustness in the training process.

[0007] (2) In the condition of missing future information, the action state needs to be accurately identified, and key information needs to be enhanced and noise needs to be filtered. In the online processing process, the model can only rely on historical frame information for judgment, and it is easy to cause event misjudgment or omission due to incomplete information. Therefore, it is necessary to effectively capture long-time sequence dependence, highlight the historical segment related to the current action, and suppress the interference of irrelevant background or redundant frames, so as to realize more accurate event state identification and positioning. SUMMARY

[0008] In view of the shortcomings of the existing method, the present application provides a weakly supervised online video time positioning method based on memory perception;

[0009] The present application also provides a weakly supervised online video time positioning system based on memory perception.

[0010] The present application constructs a weakly supervised online video time positioning model based on memory perception, combines video information and text query information, and realizes positioning of the target time in the video without frame-level labeling. To this end, the present application first proposes a weakly supervised positioning architecture of offline guiding online, generates pseudo-labels in the offline stage as supervision information in the online stage, and realizes online time positioning task only using video-level labeling; secondly, the present application excavates positive and negative samples and generates corresponding Gaussian mask to reconstruct the partially covered text query as the target, and predicts the action start time proposal; finally, the present application designs an online action detection module to enhance the short-term memory of single frame by long-term historical memory in the sliding window, enriches the context representation, and uses attention weighting decision mechanism to comprehensively consider the long-term and short-term representation at the current time, obtains the final single frame time score, and realizes time positioning in the online state. This method does not need manual labeling of the start and end time of the action, not only greatly saves the labeling cost, but also makes the model have better generalization ability and application value.

[0011] The technical scheme of the present application is as follows:

[0012] A weakly supervised online video time positioning method based on memory perception, comprising:

[0013] Multi-modal feature fusion is performed on the given video and its text query to obtain unified frame-level representation at each stage;

[0014] The offline guiding online model architecture is used, and the fused features are input into the offline and online modules in the form of the whole and frame by frame;

[0015] In the offline module, the query of the covered part of the word is reconstructed by generating a Gaussian mask to obtain the action start time proposal;

[0016] In the online module, for each frame of short-term memory feature input, first use the long-term history memory within the window to enhance it, then dynamically generate its attention weight within the window, and calculate the score of the current frame by weighting;

[0017] The proposal obtained by the offline module is used as a pseudo label to provide supervision information for the score sequence of the online module;

[0018] Finally, only the online module needs to be inferred alone, and the high-performance weakly supervised online moment localization is completed.

[0019] As a further preferred scheme, multi-modal feature fusion includes:

[0020] For uncropped videos, use a pre-trained model to extract video and query features from the video;

[0021] Input the extracted query features into a long short-term memory network to convert the query text into a representation with timing characteristics;

[0022] Fuse the video features and query features through element-wise Hadamard product, and normalize to obtain the final feature representation.

[0023] As a further preferred scheme, multi-modal feature fusion includes:

[0024] S 11: For video features , the corresponding query features are ; Let the total number of video segments be , the number of words in the sentence be , the video feature of the th frame be , and the feature of the th word in the query sentence be

[0025] S 12: Process the query through a long short-term memory network to obtain a representation with timing characteristics ;

[0026] S 13: After the video features and query features are respectively mapped to the same feature space through a fully connected layer, the final fused features are calculated and normalized using Hadamard product ; is the final feature representation;

[0027] wherein and are fully connected layer weight parameters, represents Hadamard product, denotes the Frobenius norm, is the transpose of the all-one vector.

[0028] As a further preferred solution, in the offline module, the query of the masked part of the word is reconstructed by generating a Gaussian mask to obtain the proposal of the action starting moment; comprising:

[0029] The fusion features are input as a whole into the offline module, and the probability distribution of the action occurrence at each moment in the preset video is in a Gaussian distribution;

[0030] A center position and a width corresponding to the Gaussian mask are calculated for each proposal, which are used to determine the time boundary of the action;

[0031] Positive and negative samples are mined from the video, and query reconstruction is used to evaluate the quality of the proposal.

[0032] As a further preferred solution, in the offline module, the query of the masked part of the word is reconstructed by generating a Gaussian mask to obtain the proposal of the action starting moment; comprising:

[0033] S 21: In the offline proposal generation stage, the occurrence probability of the Gaussian shape distribution is modeled as the state of the event in the video, that is, the target event is considered to be represented by a Gaussian shape probability sequence with a center and a width , denoted as , wherein is the occurrence probability of the target event at the moment , and based on the above modeling manner of the target event, the positioning proposal with the time boundary is obtained accordingly;

[0034] S 22: The fusion features are averaged in the time dimension as a whole, and then a fully connected layer is used for prediction to obtain the center and the width of the Gaussian distribution;

[0035] For each proposal, the occurrence probability of the event at the moment is calculated:

[0036] ;

[0037] wherein is any moment in the time sequence that is not , , are normalized position coordinates, , is a parameter for controlling the scaling ratio of ;

[0038] S 23: Introducing diversity loss It acts on previously predicted Stack probability sequence corresponding to each proposal , in For the first A probability sequence:

[0039] ;

[0040] in, It is the Frobenius norm. Controlling the degree of overlap between proposals It is the identity matrix;

[0041] S 24: Use query reconstruction to measure the quality of localization, and transform the original query features One-third of the words are randomly replaced with specific symbols. The cross-modal semantic representation under the positive sample mask is obtained. :

[0042] ;

[0043] in, and They are respectively using The Transformer decoder and encoder are used as attention maps to replace traditional attention mechanisms; then, after passing through a fully connected layer, the probability distribution of the next word in the vocabulary with an added event mask is obtained. :

[0044] ;

[0045] in, It is a fully connected layer. This indicates the first [item] in the original query text. The features of each word, i.e., the prediction target; Indicates the query after random replacement Center front Features of each word;

[0046] Finally, cross-entropy loss is used to calculate... Right now Difference from the true distribution :

[0047] ;

[0048] in, Indicates the original query The Middle Each word feature For the original query The number of words in the text;

[0049] S 25: Achieve comparative learning through positive and negative proposal sample mining. As a positive sample The regions on the left and right sides are used as negative samples. and The entire video region serves as a difficult negative sample. Query reconstruction loss for negative samples and hard negative samples and By using respectively , and Replacement of S24 The final reconstruction loss was obtained. for:

[0050] ;

[0051] in, It refers to the cross-entropy loss between the predicted word features and the actual word features;

[0052] Furthermore, for a single video, there should be... Therefore, a contrast loss function is introduced. The following formula is used to minimize Value to determine The optimal choice among the proposals, denoted as [number] , and then make Participating in the contrast loss function The calculation is as follows:

[0053] ;

[0054] , They are respectively determined as number After the proposal, the query reconstruction loss of the negative samples and hard negative samples corresponding to the proposal; , These refer to the two negative samples generated from the left and right sides of the proposal, respectively.

[0055] Interval parameters and Controlling the separation strength between different sample categories and satisfying .

[0056] As a further preferred approach, in the online module, for each input frame's short-term memory features, they are first enhanced using long-term historical memory within a window, and then their attention weights within the window are dynamically generated, and the score for the current frame is calculated using a weighted average; including:

[0057] The fused features are input into the online module frame by frame, and a sliding window containing long-term historical information is set for each frame;

[0058] After processing long-term and short-term features, they are input into the decoder for cross-attention calculation to obtain enhanced frame features;

[0059] The enhanced features are used to predict the probability of events occurring in the current frame using a long short-term memory network;

[0060] The score at the current moment is calculated using dynamic programming with weighted averages, and the predicted scores are then calculated under different event states.

[0061] As a further preferred approach, in the online module, for each input frame's short-term memory features, they are first enhanced using long-term historical memory within a window, and then their attention weights within the window are dynamically generated, and the score for the current frame is calculated using a weighted average; including:

[0062] S 31: In the online module, feature fusion Processed frame by frame in chronological order; length set to... A sliding window represents the current frame as and including The historical information features of a frame are represented as ,Will joint action Input to the online module at any given time; additionally, in the initial stage At that time, the visible frame rate had not yet reached [the target value]. ,at this time The header is filled with a learnable embedding feature;

[0063] S 32: Used to enhance short-term information contained in the current frame; features in the window are represented as... ,Will The input is fed into a standard Transformer decoder, where, As keys and values, As a query:

[0064] ;

[0065] Among them, CrossAttn( ) is the cross-attention layer in the decoder block. and It is a projection matrix. It is the final enhanced feature obtained. The dimensions of the projected key vector and query vector;

[0066] S 33: First, regarding the enhanced features The generation of predicted probabilities is specifically accomplished by a Long Short-Term Memory (LSTM) network with linear layers. At each time step, the hidden state of the LSTM unit... and unit state All have been updated to:

[0067] ;

[0068] in, , These refer to the hidden state and the cell state of the previous time step, respectively.

[0069] Subsequently, After linear layer transformation, the probabilities of the initial, final, and intermediate states are obtained respectively:

[0070] ;

[0071] in, For weight parameters, For deviation, hour, Indicates the event is in The probability that a given moment belongs to the initial stage; hour, Indicates the event is in The probability that a given moment belongs to the termination phase; hour, Indicates the event is in The probability that a given moment belongs to the middle stage;

[0072] S 34: Subsequently, enhancement features were applied. Calculate frame context weights:

[0073] ;

[0074] in, , and All are learnable parameters; These are intermediate variables in the weight calculation process; for At each time step, the weights are multiplied by the memory probabilities within the entire window to obtain the score, as shown below:

[0075] ;

[0076] in, For the current moment, The earliest time in the current sliding window. The default window size. This is the actual window size at the current moment, because... For the normalized weight sequence in the window, It is a list of probabilities for each moment in the window to be a different event stage; and Both refer to any one in and In between, , and They are respectively , and Attention weights after time-normalization; for The moment serves as the final score for different stages of the event; , They are respectively and The probability of time as different stages of an event;

[0077] when Upon reaching the last frame, the online event prediction score sequence for the entire video is obtained. :

[0078] ;

[0079] in, Predict score sequences for online events throughout the entire video. yes The moment serves as the final score for different stages of the event;

[0080] Finally, the outer product of the initial score sequence and the ending score sequence is performed to obtain the event prediction score map. Then, the high-score regions are filtered out using the maximum suppression algorithm to obtain the final online prediction result.

[0081] ;

[0082] in, This refers to the final result obtained from online prediction, i.e., the location segment of the target query in the video; This refers to the maximization suppression algorithm.

[0083] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of a memory-aware, weakly supervised online video time-of-motion localization method.

[0084] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of a memory-aware, weakly supervised online video time-location method.

[0085] A weakly supervised online video time-of-motion localization system based on memory perception includes:

[0086] The multimodal feature fusion module is configured to: add temporal semantics to the query features, perform Hadamard product calculation between the video and the query features, and obtain cross-modal feature encoding;

[0087] The offline event mask proposal generation module is configured to: calculate Gaussian masks for positive and negative samples using global features, reconstruct queries to optimize mask generation, and construct pseudo boundaries;

[0088] The online event detection module is configured to: set up a sliding window containing long-term information, enhance the contextual expression of short-term information, and use an in-window weighting strategy as a scoring mechanism.

[0089] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0090] 1. This invention proposes a multimodal online time-of-flight localization method that does not require manual frame-level annotation and can automatically locate the target time. Through offline-guided online architecture design, it utilizes Gaussian mask reconstruction queries to generate pseudo-boundaries and combines two-stage unified multimodal fusion features. This effectively enables the online stage to use pseudo-boundaries as optimization targets, thereby achieving independent inference capabilities. It provides a solution for online time-of-flight localization under weak supervision, significantly improving the model's scalability and application value.

[0091] 2. This invention proposes an online decision-making mechanism that combines long short-term memory (LSM), significantly enhancing the expression of contextual semantic information during frame-by-frame processing. By encoding LSM within a window through cross-attention, key parts of the current features are emphasized. These enhanced features are then input into a memory probability generator, which, combined with past memories, provides an initial event state probability. Simultaneously, frame context weights are calculated on another branch, weighting the output probabilities within the window range. This mechanism, distinct from greedy strategies, comprehensively obtains the current score, effectively addressing the problem of limited global perspective in online inference and improving the accuracy of time-of-flight localization. Attached Figure Description

[0092] Figure 1This is a flowchart of a weakly supervised online video moment localization method based on an offline-online two-stage framework according to an embodiment of the present invention;

[0093] Figure 2 This is a flowchart of the online event detection module based on memory perception according to an embodiment of the present invention. Detailed Implementation

[0094] The present invention will be further defined below with reference to the accompanying drawings and embodiments, but is not limited thereto.

[0095] Example 1

[0096] Terminology Explanation:

[0097] 1. Pre-trained models: For video data, C3D was used respectively. [1] and I3D [2] The model extracts visual features from the ActivityNetCaptions and Charades-STA datasets; for query text data, GloVe is used throughout. [3] The model extracts word vector features.

[0098] 2. Long Short-Term Memory (LSTM) is a type of recurrent neural network (RNN). Typically, an LSTM unit consists of a cell, an input gate, an output gate, and a forget gate.

[0099] 3. The standard Transformer decoder is a deep learning architecture that uses an attention mechanism. It consists of an encoder and a decoder, including modules such as multi-head attention, feedforward network, residual connection and layer normalization.

[0100] 4. Traditional attention mechanism refers to the standard dot product attention used in the Transformer model. Specifically, it first calculates the dot product similarity between the query and the key, then obtains the attention weights through softmax, and finally uses the attention weights to perform a weighted sum of the values ​​to obtain the output.

[0101] like Figure 1As shown, a weakly supervised online temporal SentencE gRounding model (WISER) is constructed to achieve a system that obtains temporal location results based on an untrimmed video set and video-level text annotations. This system then intelligently locates the start and end times of target segments in the video that match the query description. The model first fuses visual and textual features to obtain cross-modal representations, which enter two stages in different data formats: In the upper-layer offline event mask proposal generation stage, the module absorbs all the overall features containing comprehensive information at once, providing an event mask proposal and start, end, and intermediate labels based on the proposal from a global perspective; In the lower-layer online event detection stage, features are input into the module in a streaming manner to simulate the online process, predicting the probability of the current moment as the start, end, and intermediate of an event frame by frame, while video frames after this moment are invisible to the module. Finally, in the offline stage, positive and negative samples are constructed and the query is reconstructed to optimize the proposal and obtain pseudo-labels, which are then used as supervisory information for predicting probability sequences in the online stage. During the inference phase, the offline module is frozen, and the online event detection is activated separately to achieve accurate retrieval of target moments in online videos under weak supervision.

[0102] A weakly supervised online video moment localization method based on memory perception includes:

[0103] Multimodal feature fusion is performed on a given video and its text query to obtain a unified frame-level representation at each stage;

[0104] Using an offline-guided online model architecture, the fused features are input into the offline and online modules respectively, in whole and frame-by-frame form.

[0105] In the offline module, the query of the masked part of the word is reconstructed by generating a Gaussian mask to obtain the proposal at the start time of the action;

[0106] In the online module, for each frame of short-term memory features in the input, the features are first enhanced using long-term historical memory within the window, and then attention weights within the window are dynamically generated to calculate the score of the current frame.

[0107] The proposals obtained from the offline module are used as pseudo-labels to provide supervision information for the score sequence of the online module;

[0108] Finally, high-performance weakly supervised online time-of-flight localization can be achieved simply by inferring the online module separately.

[0109] Example 2

[0110] The difference between the memory-based weakly supervised online video time-localization method described in Example 1 and the method described in Example 1 is as follows:

[0111] Multimodal feature fusion, including:

[0112] For uncropped videos, a pre-trained model is used to extract video and query features from the video.

[0113] The extracted query features are input into a long short-term memory network to transform the query text into a representation with temporal features;

[0114] The video features and query features are fused through element-wise Hadamard product, and the final feature representation is obtained after normalization.

[0115] Multimodal feature fusion, including:

[0116] S 11: Regarding video features The corresponding query features are ; The total number of video clips. The number of words in the sentence. For the first Video features of a frame To query the first character in the sentence Features of each word;

[0117] S 12: Query Representations with temporal characteristics are obtained by processing through a Long Short-Term Memory (LSTM) network. ;

[0118] S 13: After video features and query features are mapped to the same feature space through fully connected layers, the final fused features are obtained by calculating and normalizing the Hadamard product. ; This refers to the final feature representation;

[0119] in, and These are the weight parameters of the fully connected layer. It represents the Hadamah accumulation. Denotes the Frobenius norm. It is the transpose of a single vector.

[0120] In the offline module, the query for the masked part of the word is reconstructed by generating a Gaussian mask to obtain the proposal at the start of the action; including:

[0121] The fused features are input into the offline module as a whole, and the probability distribution of the action at each moment in the preset video is a Gaussian distribution;

[0122] For each proposal, calculate the center position and width of a Gaussian mask to determine the time boundary of the action;

[0123] We extract positive and negative samples from videos and use query reconstruction to evaluate the quality of proposals.

[0124] In the offline module, the query for the masked part of the word is reconstructed by generating a Gaussian mask to obtain the proposal at the start of the action; including:

[0125] S 21: In the offline proposal generation stage, the occurrence probability of events in the video is modeled using a Gaussian distribution, that is, it is assumed that the target event is generated by a central... and width Gaussian morphological probability sequence It means that, among them, for Given the probability of the target event occurring at time t, and based on the above modeling method for the target event, the corresponding time boundary is obtained as follows: Positioning proposal;

[0126] S 22: Integrate features The overall average over time is then used for prediction via a fully connected layer. Center and width of Gaussian distribution As shown below:

[0127] ;

[0128] ;

[0129] in, This is a representation of the fused features after averaging over the time dimension. It is the sigmoid activation function. and These are the weights and offset parameters in the fully connected layer, respectively.

[0130] For each proposal, the following calculations are obtained. The probability of an event occurring at a given time. :

[0131] ;

[0132] in, For time series not being At any time, , All are normalized position coordinates. , It is used for control The scaling parameter;

[0133] S 23: Introducing diversity loss It acts on previously predicted Stack probability sequence corresponding to each proposal , in For the first A probability sequence:

[0134] ;

[0135] in, It is the Frobenius norm. Controlling the degree of overlap between proposals The identity matrix is ​​used; the diversity loss effectively prevents redundant proposals from converging to similar time positions, while promoting comprehensive coverage of potential events throughout the video.

[0136] S 24: Use query reconstruction to measure the quality of localization, and transform the original query features One-third of the words are randomly replaced with specific symbols. The cross-modal semantic representation under the positive sample mask is obtained. :

[0137] ;

[0138] in, and They are respectively using The Transformer decoder and encoder are used as attention maps to replace traditional attention mechanisms; then, after passing through a fully connected layer, the probability distribution of the next word in the vocabulary with an added event mask is obtained. :

[0139] ;

[0140] in, It is a fully connected layer. This indicates the first [item] in the original query text. The features of each word, i.e., the prediction target; Indicates the query after random replacement Center front Features of each word;

[0141] Finally, cross-entropy loss is used to calculate... Right now Difference from the true distribution : Recorded as in Query reconstruction loss obtained under the proposal:

[0142] ;

[0143] in, Indicates the original query The Middle Each word feature For the original query The number of words in the text;

[0144] S 25: Achieve comparative learning through positive and negative proposal sample mining. As a positive sample The regions on the left and right sides are used as negative samples. and The entire video region serves as a difficult negative sample. Query reconstruction loss for negative samples and hard negative samples and By using respectively , and Replacement of S24 The final reconstruction loss was obtained. for:

[0145] ;

[0146] in, It refers to the cross-entropy loss between the predicted word features and the actual word features;

[0147] Furthermore, for a single video, there should be... Therefore, a contrast loss function is introduced. The following formula is used to minimize Value to determine The optimal choice among the proposals, denoted as [number] , and then make Participating in the contrast loss function The calculation is as follows:

[0148] ;

[0149] , They are respectively determined as number After the proposal, the query reconstruction loss of the negative samples and hard negative samples corresponding to the proposal; , These refer to the two negative samples generated from the left and right sides of the proposal, respectively.

[0150] Interval parameters and Controlling the separation strength between different sample categories and satisfying .

[0151] In the online module, for each input frame's short-term memory features, they are first enhanced using long-term historical memory within a window, then their attention weights within the window are dynamically generated, and the score for the current frame is calculated using a weighted average; including:

[0152] The fused features are input into the online module frame by frame, and a sliding window containing long-term historical information is set for each frame;

[0153] After processing long-term and short-term features, they are input into the decoder for cross-attention calculation to obtain enhanced frame features;

[0154] The enhanced features are used to predict the probability of events occurring in the current frame using a long short-term memory network;

[0155] The score at the current moment is calculated using dynamic programming with weighted averages, and the predicted scores are then calculated under different event states.

[0156] In the online module, for each input frame's short-term memory features, they are first enhanced using long-term historical memory within a window, then their attention weights within the window are dynamically generated, and the score for the current frame is calculated using a weighted average; including:

[0157] S 31: For example Figure 2 As shown, in the online module, features are fused. Processed frame by frame in chronological order; considering the lack of context information in a single frame, the length is set to... A sliding window represents the current frame as and including The historical information features of a frame are represented as ,Will joint action Input to the online module at any given time; additionally, in the initial stage At that time, the visible frame rate had not yet reached [the target value]. ,at this time The header is padded with a learnable embedding feature to keep the window length fixed.

[0158] S 32: Used to enhance short-term information contained in the current frame; features in the window are represented as... ,Will The input is fed into a standard Transformer decoder, where, As keys and values, As a query:

[0159] ;

[0160] Among them, CrossAttn( ) is the cross-attention layer in the decoder block. and It is a projection matrix. It is the final enhanced feature obtained. The dimensions of the projected key vector and query vector;

[0161] S 33: For example Figure 2 As shown, firstly, the enhanced features... The generation of predicted probabilities is specifically accomplished by a Long Short-Term Memory (LSTM) network with linear layers. At each time step, the hidden state of the LSTM unit... and unit state All have been updated to:

[0162] ;

[0163] in, , These refer to the hidden state and the cell state of the previous time step, respectively.

[0164] Subsequently, After linear layer transformation, the probabilities of the initial, final, and intermediate states are obtained respectively:

[0165] ;

[0166] in, For weight parameters, For deviation, hour, Indicates the event is in The probability that a given moment belongs to the initial stage; hour, Indicates the event is in The probability that a given moment belongs to the termination phase; hour, Indicates the event is in The probability that a given moment belongs to the middle stage;

[0167] S 34: For example Figure 2 As shown, the enhancement features are then... Calculate frame context weights:

[0168] ;

[0169] in, , and All are learnable parameters; These are intermediate variables in the weight calculation process; for At each time step, the weights are multiplied by the memory probabilities within the entire window to obtain the score, as shown below:

[0170] ;

[0171] in, For the current moment, The earliest time in the current sliding window. The default window size. This is the actual window size at the current moment, because... For the normalized weight sequence in the window, It is a list of probabilities for each moment in the window to be a different event stage; and Both refer to any one in and In between, , and They are respectively , and Attention weights after time-normalization; for The moment serves as the final score for different stages of the event; , They are respectively and The probability of time as different stages of an event;

[0172] when Upon reaching the last frame, the online event prediction score sequence for the entire video is obtained. :

[0173] ;

[0174] in, Predict score sequences for online events throughout the entire video. yes The moment serves as the final score for different stages of the event;

[0175] Finally, the outer product of the initial score sequence and the ending score sequence is performed to obtain the event prediction score map. Then, the high-score regions are filtered out using the maximum suppression algorithm to obtain the final online prediction result.

[0176] ;

[0177] in, This refers to the final result obtained from online prediction, i.e., the location segment of the target query in the video; This refers to the maximization suppression algorithm. See “Efficient non-maximum suppression,” in 18th international conference on pattern recognition (ICPR'06), vol. 3. IEEE, 2006, pp. 850–855.

[0178] Table 1 shows the comparative model of the present invention;

[0179] Table 1. Comparison Model Introduction;

[0180]

[0181] Table 2 shows the comparison results of the positioning accuracy of the present invention and the modified baseline model;

[0182] Table 2. Comparison of positioning accuracy between the present invention and the modified baseline model.

[0183]

[0184] As shown in Table 2, WISER has a significantly superior positioning accuracy compared to the modified baseline model.

[0185] Example 3

[0186] A computer device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the memory-aware, weakly supervised online video time-of-motion localization method described in Embodiment 1 or 2.

[0187] Example 4

[0188] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the memory-aware, weakly supervised online video time-localization method described in Embodiment 1 or 2.

[0189] Example 5

[0190] A weakly supervised online video time-of-motion localization system based on memory perception includes:

[0191] The multimodal feature fusion module is configured to: add temporal semantics to the query features, perform Hadamard product calculation between the video and the query features, and obtain cross-modal feature encoding;

[0192] The offline event mask proposal generation module is configured to: calculate Gaussian masks for positive and negative samples using global features, reconstruct queries to optimize mask generation, and construct pseudo boundaries;

[0193] The online event detection module is configured to: set up a sliding window containing long-term information, enhance the contextual expression of short-term information, and use an in-window weighting strategy as a scoring mechanism.

Claims

1. A weakly supervised online video time-of-motion localization method based on memory perception, characterized in that, include: Multimodal feature fusion is performed on a given video and its text query to obtain a unified frame-level representation at each stage; Using an offline-guided online model architecture, the fused features are input into the offline and online modules respectively, in whole and frame-by-frame form. In the offline module, the query of the masked part of the word is reconstructed by generating a Gaussian mask to obtain the proposal at the start time of the action; In the online module, for each frame of short-term memory features in the input, the features are first enhanced using long-term historical memory within the window, and then attention weights within the window are dynamically generated to calculate the score of the current frame. The proposals obtained from the offline module are used as pseudo-labels to provide supervision information for the score sequence of the online module; Finally, high-performance weakly supervised online time-of-flight localization can be achieved simply by inferring the online module separately. In the online module, for each input frame's short-term memory features, they are first enhanced using long-term historical memory within a window, then their attention weights within the window are dynamically generated, and the score for the current frame is calculated using a weighted average; including: The fused features are input into the online module frame by frame, and a sliding window containing long-term historical information is set for each frame; After processing long-term and short-term features, they are input into the decoder for cross-attention calculation to obtain enhanced frame features; The enhanced features are used to predict the probability of events occurring in the current frame using a long short-term memory network; The score at the current moment is calculated using dynamic programming with weighted average, and the predicted scores are calculated under different event states. In the online module, for each input frame's short-term memory features, they are first enhanced using long-term historical memory within a window, then their attention weights within the window are dynamically generated, and the score for the current frame is calculated using a weighted average; including: S31: In the online module, the fused feature F is processed frame by frame in chronological order; a sliding window with a length of n is set, and the current frame is represented as f t , and the historical information feature containing n - 1 frames is represented as F h , and [F h , f t are jointly used as the input of the online module at time t; additionally, at the initial stage when t + 1 < n and the number of visible frames has not reached n, a learnable embedding feature is padded at the head of F h ​ S32: Used to enhance short-term information contained in the current frame; the feature in the window is represented as F. t ={F h ,f t }, F t The input is fed into a standard Transformer decoder, where F t As both key and value, f t As a query: Among them, CrossAttn(f t ,F t ,F t ) is the cross-attention layer in the decoder block, W Q W K and W V It is the projection matrix, W t The final enhanced feature, d k The dimensions of the projected key vector and query vector; S33: First, enhance the feature W t The generation of predicted probabilities is specifically accomplished by a Long Short-Term Memory (LSTM) network with linear layers. At each time step, the hidden state h of the LSTM unit... t and unit state c t All have been updated to: h t ,c t =LSTM(h t-1 ,c t-1 ,W t ); Among them, h t-1 c t-1 These refer to the hidden state and the cell state of the previous time step, respectively. Subsequently, h t After linear layer transformation, the probabilities of the initial, final, and intermediate states are obtained respectively: Among them, W ξ b is the weight parameter. ξ For the deviation, when ξ = s, This represents the probability that the event belongs to the initial stage at time t; when ξ = e, This represents the probability that the event belongs to the termination phase at time t; when ξ = m, This represents the probability that the event belongs to the intermediate stage at time t; S34: Subsequently, the enhanced feature W t Calculate frame context weights: Among them, W w b w and u w All are learnable parameters; u t These are intermediate variables in the weight calculation process; for time t, the weights are multiplied by the memory probabilities within the entire window to obtain the scores, as shown below: Where t is the current time, t m The earliest time in the current sliding window, where n is the preset window size. m This is the actual window size at the current moment, because... For the normalized weight sequence in the window, It is a list of probabilities for each moment in the window to be a different event stage; t i Both t and t′ refer to any point in t. m The time interval between t and t and t i t m The attention weights after normalization at time t; Let time t be the final score for different event phases; t m The probabilities of time t as different event stages; When t reaches the last frame, the online event prediction score sequence for the entire video is obtained. in, Predict score sequences for online events throughout the entire video. It is the final score at time t as different event stages; Finally, the outer product of the initial score sequence and the ending score sequence is performed to obtain the event prediction score map. Then, the high-score regions are filtered out using the maximum suppression algorithm to obtain the final online prediction result. Among them, seg on It refers to the final result obtained from online prediction, that is, the location segment of the target query in the video; NMS(·) refers to the maximum suppression algorithm.

2. The method for weakly supervised online video time localization based on memory perception according to claim 1, characterized in that, Multimodal feature fusion, including: For uncropped videos, a pre-trained model is used to extract video and query features from the video. The extracted query features are input into a long short-term memory network to transform the query text into a representation with temporal features; The video features and query features are fused through element-wise Hadamard product, and the final feature representation is obtained after normalization.

3. The method for weakly supervised online video time localization based on memory perception according to claim 2, characterized in that, Multimodal feature fusion, including: S11: For video features V = {v0, v1, ..., v...} t …,v T-1 The corresponding query feature is Q = {q0, q1, ..., q}. n …,q N-1 }; T is the total number of video clips, N is the number of words in a sentence, v t Let q be the video feature of frame t. n To query the features of the nth word in a sentence; S12: Process the query Q through a Long Short-Term Memory network to obtain a representation f with temporal characteristics. Q =LSTM(Q); S13: After the video features and query features are mapped to the same feature space through fully connected layers, the final fused features are obtained by calculating and normalizing the Hadamard product. F refers to the final feature representation; Among them, w Q and W V These are the weight parameters of the fully connected layer, ⊙ represents the Hadamard product, ||·|| F Denotes the Frobenius norm. It is the transpose of a single vector.

4. The weakly supervised online video time-of-motion localization method based on memory perception according to claim 1, characterized in that, In the offline module, the query for the masked part of the word is reconstructed by generating a Gaussian mask to obtain the proposal at the start of the action; including: The fused features are input into the offline module as a whole, and the probability distribution of the action at each moment in the preset video is a Gaussian distribution; For each proposal, calculate the center position and width of a Gaussian mask to determine the time boundary of the action; We extract positive and negative samples from videos and use query reconstruction to evaluate the quality of proposals.

5. The weakly supervised online video time-of-motion localization method based on memory perception according to claim 4, characterized in that, In the offline module, the query for the masked part of the word is reconstructed by generating a Gaussian mask to obtain the proposal at the start of the action; including: S21: In the offline proposal generation stage, the occurrence probability of events in the video is modeled using a Gaussian morphological distribution, that is, the target event is assumed to be a Gaussian morphological probability sequence M with center c and width ω. p ={m0,…,m t …,m T-1 } represents, where m t Let be the probability of the target event occurring at time t, and based on the modeling method for the target event, the corresponding time boundary is obtained as follows: Positioning proposal; S22: Average the fused features F over the time dimension, and then predict the center and width of the K pairs of Gaussian distributions through a fully connected layer. For each proposal, the probability m of the event occurring at time t is calculated. t : Where t′ represents any time other than t in the time series. All are normalized position coordinates. θ is a parameter used to control the scaling ratio of ω; S23: Introducing Diversity Loss The stacking probability sequence corresponding to the previously predicted K proposals is applied. in For the k-th probability sequence: Among them, ||·|| F Let λ be the Frobenius norm, λ∈[0,1] controlling the degree of overlap between proposals, and I be the identity matrix. S24: Use query reconstruction to measure the quality of localization by randomly replacing 1 / 3 of the words in the original query feature Q with specific symbols. Obtain the cross-modal semantic representation G under the positive sample mask. p : Where D(·) and E(·) are respectively represented by m p The Transformer decoder and encoder are used as attention maps to replace traditional attention mechanisms; then, after passing through a fully connected layer, the probability distribution of the next word in the vocabulary with an added event mask is obtained. Where FC(·) is a fully connected layer, This represents the feature of the (i+1)th word in the original query text, i.e., the prediction target; Indicates the query after random replacement Features of the first i words; Finally, cross-entropy loss is used to calculate P. p Right now Difference from the true distribution Where, q i+1 Let N represent the feature of the (i+1)th word in the original query Q, and N be the number of words in the original query Q. S25: Comparative learning is achieved through positive and negative proposal sample mining, M p As a positive sample, M p The regions on the left and right sides are used as negative samples. and The entire video region is used as a difficult negative sample M h Query reconstruction loss for negative samples and hard negative samples and By using respectively and M h Replacement of S24's M p The final reconstruction loss L is obtained. rec for: in, It refers to the cross-entropy loss between the predicted word features and the actual word features; Furthermore, for a single video, there should be... Therefore, a contrast loss function is introduced. Using the method shown in the following formula, with the minimum The value is used to determine the optimal choice among K proposals, and its number is denoted as k. * Then make k * Participating in the contrast loss function The calculation is as follows: They are respectively determined to be numbered k * After the proposal, the query reconstruction loss of the negative samples and hard negative samples corresponding to the proposal; n1 and n2 refer to the two negative samples generated from the left and right sides of the proposal, respectively. The interval parameters β1 and β2 control the separation strength between different sample categories, and satisfy β1 < β2.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the weakly supervised online video time-location method based on memory perception as described in any one of claims 1-5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the weakly supervised online video time-location method based on memory perception as described in any one of claims 1-5.

8. A memory-aware, weakly supervised online video time-of-motion localization system, used to implement the memory-aware, weakly supervised online video time-of-motion localization method according to any one of claims 1-5, characterized in that, include: The multimodal feature fusion module is configured to: add temporal semantics to the query features, perform Hadamard product calculation between the video and the query features, and obtain cross-modal feature encoding; The offline event mask proposal generation module is configured to: calculate Gaussian masks for positive and negative samples using global features, reconstruct queries to optimize mask generation, and construct pseudo boundaries; The online event detection module is configured to: set up a sliding window containing long-term information, enhance the contextual expression of short-term information, and use an in-window weighting strategy as a scoring mechanism.

Images