Weak supervision online video moment positioning method and system based on memory perception

Abstract

The invention relates to a weak supervision online video moment positioning method and system based on memory perception, and belongs to the technical field of artificial intelligence, and the method comprises the steps: carrying out the multi-modal feature fusion of a given video and a text query thereof, and obtaining the unified frame level representation at each stage; inputting the fused features into an offline module and an online module in an integral and frame-by-frame manner by using an offline guide online model architecture; in the off-line module, generating a Gaussian mask to reconstruct query of a covered part of words, and obtaining a proposal of an action starting moment; in the on-line module, the long-term historical memory in the window is used for enhancing the score, the attention weight of the score in the window is dynamically generated, and the score of the current frame is calculated in a weighted mode; taking the proposal obtained by the offline module as a pseudo tag, and providing supervision information for the score sequence of the online module; and high-performance weak supervision on-line moment positioning can be completed only by independently deducing the on-line module. The expansion capability and the application value of the model are remarkably improved.

Description

Technical Field

[0001] This invention relates to a weakly supervised online video time-of-motion localization method and system based on memory perception, belonging to the field of artificial intelligence technology. Background Technology

[0002] With the widespread adoption of smart sensing terminals and the penetration of digital lifestyles, the application scenarios and data volume of online video are growing rapidly. Video time-lapse localization, as a key technology in the field of video analytics, plays a crucial role in numerous real-world scenarios.

[0003] In the field of intelligent security, real-time video positioning can monitor and screen abnormal behaviors in surveillance videos, such as illegal driving and fighting, providing strong support for ensuring public safety. In industrial production, by locating workers' actions in real time, it is possible to determine whether their behavior conforms to operating procedures, thereby improving production efficiency while increasing safety. In the field of education, by quickly locating teaching video content, it promotes the effective dissemination of knowledge and improves learning efficiency.

[0004] Traditional online video moment localization tasks often rely on complete start-time label information. This means that during model training, annotators must meticulously label the behavioral boundaries of the video data. However, using fully supervised learning methods for video annotation is not only time-consuming and labor-intensive but also significantly increases algorithm development costs. Furthermore, the subjectivity of the annotation process can lead to discrepancies in annotation results among different annotators. This uncertainty further limits the scalability of video moment localization models and restricts their application in real-world scenarios. Faced with these challenges, weakly supervised learning, with its compatibility with video-level labels, has become a key path to overcome the bottleneck of analyzing massive amounts of poorly labeled videos.

[0005] By employing online video moment localization under weak supervision, this research not only deepens the understanding of traditional video motion localization but also demonstrates greater scalability, enabling it to handle the ever-increasing scale of video data. Currently, this research faces two main challenges:

[0006] (1) In the absence of frame-level annotations, traditional sample mining strategies are difficult to apply, and an effective pseudo-supervision mechanism needs to be established. Due to the lack of fine-grained annotation signals, the model has difficulty in accurately judging the semantic boundaries of video segments and is easily affected by background segments or multiple events, resulting in unstable training and decreased localization performance. Therefore, it is urgent to build a mechanism that can generate high-quality pseudo-labels from weak video-text supervision signals to provide more reliable learning signals and enhance robustness during training.

[0007] (2) In the absence of future information, it is necessary to accurately identify the action state while taking into account both key information enhancement and noise filtering. During online processing, the model can only rely on historical frame information for judgment, which can easily lead to false detection or missed detection of events due to incomplete information. Therefore, it is necessary to effectively capture long-term temporal dependencies, highlight historical segments related to the current action, and suppress interference from irrelevant background or redundant frames to achieve more accurate event state identification and localization. Summary of the Invention

[0008] To address the shortcomings of existing methods, this invention proposes a weakly supervised online video moment localization method based on memory perception. This invention also proposes a weakly supervised online video time-of-motion positioning system based on memory perception.

[0009] This invention constructs a memory-aware, weakly supervised online video moment localization model. Combining video information and text query information, it achieves target moment localization in videos without frame-level annotation. To this end, this invention first proposes an offline-guided online weakly supervised localization architecture. By generating pseudo-labels offline, it serves as supervision information in the online phase, enabling online moment localization using only video-level annotation. Second, by mining positive and negative samples and generating corresponding Gaussian morphological masks, this invention aims to reconstruct partially masked text queries and predict action start moment proposals. Finally, this invention designs an online action detection module that enhances the short-term memory of a single frame with long-term historical memory within a sliding window, enriching contextual representation. It also utilizes an attention-weighted decision mechanism to comprehensively consider both long-term and short-term representations at the current moment, obtaining the final single-frame moment score, thereby achieving moment localization in the online state. This method eliminates the need for manual annotation of action start and end moments, significantly saving annotation costs and giving the model better generalization ability and application value.

[0010] The technical solution of the present invention is as follows: A weakly supervised online video moment localization method based on memory perception includes: Multimodal feature fusion is performed on a given video and its text query to obtain a unified frame-level representation at each stage; Using an offline-guided online model architecture, the fused features are input into the offline and online modules respectively, in whole and frame-by-frame form. In the offline module, the query of the masked part of the word is reconstructed by generating a Gaussian mask to obtain the proposal at the start time of the action; In the online module, for each frame of short-term memory features in the input, the features are first enhanced using long-term historical memory within the window, and then attention weights within the window are dynamically generated to calculate the score of the current frame. The proposals obtained from the offline module are used as pseudo-labels to provide supervision information for the score sequence of the online module; Finally, high-performance weakly supervised online time-of-flight localization can be achieved simply by inferring the online module separately.

[0011] As a further preferred option, multimodal feature fusion includes: For uncropped videos, a pre-trained model is used to extract video and query features from the video. The extracted query features are input into a long short-term memory network to transform the query text into a representation with temporal features; The video features and query features are fused through element-wise Hadamard product, and the final feature representation is obtained after normalization.

[0012] As a further preferred option, multimodal feature fusion includes: S 11: Regarding video features The corresponding query features are ; The total number of video clips. The number of words in the sentence. For the first Video features of a frame To query the first character in the sentence Features of each word; S 12: Query Representations with temporal characteristics are obtained by processing through a Long Short-Term Memory (LSTM) network. ; S 13: After video features and query features are mapped to the same feature space through fully connected layers, the final fused features are obtained by calculating and normalizing the Hadamard product. ; This refers to the final feature representation; in, and These are the weight parameters of the fully connected layer. It represents the Hadamah accumulation. Denotes the Frobenius norm. It is the transpose of a vector that is all one.

[0013] As a further preferred solution, in the offline module, the query for the masked part of the word is reconstructed by generating a Gaussian mask to obtain the proposal at the start time of the action; including: The fused features are input into the offline module as a whole, and the probability distribution of the action at each moment in the preset video is a Gaussian distribution; For each proposal, calculate the center position and width of a Gaussian mask to determine the time boundary of the action; We extract positive and negative samples from videos and use query reconstruction to evaluate the quality of proposals.

[0014] As a further preferred solution, in the offline module, the query for the masked part of the word is reconstructed by generating a Gaussian mask to obtain the proposal at the start time of the action; including: S 21: In the offline proposal generation stage, the occurrence probability of events in the video is modeled using a Gaussian distribution, that is, it is assumed that the target event is generated by a central... and width Gaussian morphological probability sequence It means that, among them, for Given the probability of the target event occurring at time t, and based on the above modeling method for the target event, the corresponding time boundary is obtained as follows: Positioning proposal; S 22: Integrating features The overall average over time is then used for prediction via a fully connected layer. For the center and width of the Gaussian distribution ; For each proposal, the following calculations are obtained. The probability of an event occurring at a given time. : ; in, For time series not being At any time, , All are normalized position coordinates. , It is used for control The scaling parameter; S 23: Introducing diversity loss It acts on previously predicted Stack probability sequence corresponding to each proposal , in For the first A probability sequence: ; in, It is the Frobenius norm. Controlling the degree of overlap between proposals It is the identity matrix; S24: Use query reconstruction to measure the quality of localization, and transform the original query features One-third of the words are randomly replaced with specific symbols. The cross-modal semantic representation under the positive sample mask is obtained. : ; in, and They are respectively using The Transformer decoder and encoder are used as attention maps to replace traditional attention mechanisms; then, after passing through a fully connected layer, the probability distribution of the next word in the vocabulary with an added event mask is obtained. : ; in, It is a fully connected layer. This indicates the first [item] in the original query text. The features of each word, i.e., the prediction target; Indicates the query after random replacement Center front Features of each word; Finally, cross-entropy loss is used to calculate... Right now Difference from the true distribution : ; in, Indicates the original query The Middle Each word feature For the original query The number of words in the text; S 25: Achieve comparative learning through positive and negative proposal sample mining. As a positive sample The regions on the left and right sides are used as negative samples. and The entire video region serves as a difficult negative sample. Query reconstruction loss for negative samples and hard negative samples and By using respectively , and Replacement of S24 The final reconstruction loss was obtained. for: ; in, It refers to the cross-entropy loss between the predicted word features and the actual word features; Furthermore, for a single video, there should be... Therefore, a contrast loss function is introduced. The following formula is used to minimize Value to determine The optimal choice among the proposals, denoted as [number] , and then make Participating in the contrast loss function The calculation is as follows: ; , They are respectively determined as number After the proposal, the query reconstruction loss of the negative samples and hard negative samples corresponding to the proposal; , These refer to the two negative samples generated from the left and right sides of the proposal, respectively. Interval parameters and Controlling the separation strength between different sample categories and satisfying .

[0015] As a further preferred approach, in the online module, for each input frame's short-term memory features, they are first enhanced using long-term historical memory within a window, and then their attention weights within the window are dynamically generated, and the score for the current frame is calculated using a weighted average; including: The fused features are input into the online module frame by frame, and a sliding window containing long-term historical information is set for each frame; After processing long-term and short-term features, they are input into the decoder for cross-attention calculation to obtain enhanced frame features; The enhanced features are used to predict the probability of events occurring in the current frame using a long short-term memory network; The score at the current moment is calculated using dynamic programming with weighted averages, and the predicted scores are then calculated under different event states.

[0016] As a further preferred approach, in the online module, for each input frame's short-term memory features, they are first enhanced using long-term historical memory within a window, and then their attention weights within the window are dynamically generated, and the score for the current frame is calculated using a weighted average; including: S 31: In the online module, feature fusion Processed frame by frame in chronological order; length set to... A sliding window represents the current frame as and including The historical information features of a frame are represented as ,Will joint action Input to the online module at any given time; additionally, in the initial stage At that time, the visible frame rate had not yet reached [the target value]. ,at this time The header is filled with a learnable embedding feature; S 32: Used to enhance short-term information contained in the current frame; features in the window are represented as... ,Will The input is fed into a standard Transformer decoder, where, As keys and values, As a query: ; Among them, CrossAttn( ) is the cross-attention layer in the decoder block. and It is a projection matrix. It is the final enhanced feature obtained. The dimensions of the projected key vector and query vector; S 33: First, regarding the enhanced features The generation of predicted probabilities is specifically accomplished by a Long Short-Term Memory (LSTM) network with linear layers. At each time step, the hidden state of the LSTM unit... and unit state All have been updated to: ; in, , These refer to the hidden state and the cell state of the previous time step, respectively. Subsequently, After linear layer transformation, the probabilities of the initial, final, and intermediate states are obtained respectively: ; in, For weight parameters, For deviation, hour, Indicates the event is in The probability that a given moment belongs to the initial stage; hour, Indicates the event is in The probability that a given moment belongs to the termination phase; hour, Indicates the event is in The probability that a given moment belongs to the middle stage; S 34: Subsequently, enhancement features were applied. Calculate frame context weights: ; in, , and All are learnable parameters; These are intermediate variables in the weight calculation process; for At each time step, the weights are multiplied by the memory probabilities within the entire window to obtain the score, as shown below: ; in, For the current moment, The earliest time in the current sliding window. The default window size. This is the actual window size at the current moment, because... For the normalized weight sequence in the window, It is a list of probabilities for each moment in the window to be a different event stage; and Both refer to any one in and In between, , and They are respectively , and Attention weights after time-normalization; for The moment serves as the final score for different stages of the event; , They are respectively and The probability of time as different stages of an event; when Upon reaching the last frame, the online event prediction score sequence for the entire video is obtained. : ; in, Predict score sequences for online events throughout the entire video. yes The moment serves as the final score for different stages of the event; Finally, the outer product of the initial score sequence and the ending score sequence is performed to obtain the event prediction score map. Then, the high-score regions are filtered out using the maximum suppression algorithm to obtain the final online prediction result. ; in, This refers to the final result obtained from online prediction, i.e., the location segment of the target query in the video; This refers to the maximization suppression algorithm.

[0017] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of a memory-aware, weakly supervised online video time-of-motion localization method.

[0018] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of a memory-aware, weakly supervised online video time-location method.

[0019] A weakly supervised online video time-of-motion localization system based on memory perception includes: The multimodal feature fusion module is configured to: add temporal semantics to the query features, perform Hadamard product calculation between the video and the query features, and obtain cross-modal feature encoding; The offline event mask proposal generation module is configured to: calculate Gaussian masks for positive and negative samples using global features, reconstruct queries to optimize mask generation, and construct pseudo boundaries; The online event detection module is configured to: set up a sliding window containing long-term information, enhance the contextual expression of short-term information, and use an in-window weighting strategy as a scoring mechanism.

[0020] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention proposes a multimodal online time-of-flight localization method that does not require manual frame-level annotation and can automatically locate the target time. Through offline-guided online architecture design, it utilizes Gaussian mask reconstruction queries to generate pseudo-boundaries and combines two-stage unified multimodal fusion features. This effectively enables the online stage to use pseudo-boundaries as optimization targets, thereby achieving independent inference capabilities. It provides a solution for online time-of-flight localization under weak supervision, significantly improving the model's scalability and application value.

[0021] 2. This invention proposes an online decision-making mechanism that combines long short-term memory (LSM), significantly enhancing the expression of contextual semantic information during frame-by-frame processing. By encoding LSM within a window through cross-attention, key parts of the current features are emphasized. These enhanced features are then input into a memory probability generator, which, combined with past memories, provides an initial event state probability. Simultaneously, frame context weights are calculated on another branch, weighting the output probabilities within the window range. This mechanism, distinct from greedy strategies, comprehensively obtains the current score, effectively addressing the problem of limited global perspective in online inference and improving the accuracy of time-of-flight localization. Attached Figure Description

[0022] Figure 1 This is a flowchart of a weakly supervised online video moment localization method based on an offline-online two-stage framework according to an embodiment of the present invention. Figure 2 This is a flowchart of the online event detection module based on memory perception according to an embodiment of the present invention. Detailed Implementation

[0023] The present invention will be further defined below with reference to the accompanying drawings and embodiments, but is not limited thereto.

[0024] Example 1 Terminology Explanation: 1. Pre-trained models: For video data, C3D was used respectively. [1] and I3D [2] The model extracts visual features from the ActivityNetCaptions and Charades-STA datasets; for query text data, GloVe is used throughout. [3] The model extracts word vector features.

[0025] 2. Long Short-Term Memory (LSTM) network is a type of recurrent neural network (RNN). Typically, an LSTM unit consists of a cell, an input gate, an output gate, and a forget gate.

[0026] 3. The standard Transformer decoder is a deep learning architecture that uses an attention mechanism. It consists of an encoder and a decoder, including modules such as multi-head attention, feedforward network, residual connection and layer normalization.

[0027] 4. Traditional attention mechanism refers to the standard dot product attention used in the Transformer model. Specifically, it first calculates the dot product similarity between the query and the key, then obtains the attention weights through softmax, and finally uses the attention weights to perform a weighted sum of the values ​​to obtain the output.

[0028] like Figure 1As shown, a weakly supervised online temporal SentencE gRounding model (WISER) is constructed to achieve a system that obtains temporal location results based on an untrimmed video set and video-level text annotations. This system then intelligently locates the start and end times of target segments in the video that match the query description. The model first fuses visual and textual features to obtain cross-modal representations, which enter two stages in different data formats: In the upper-layer offline event mask proposal generation stage, the module absorbs all the overall features containing comprehensive information at once, providing an event mask proposal and start, end, and intermediate labels based on the proposal from a global perspective; In the lower-layer online event detection stage, features are input into the module in a streaming manner to simulate the online process, predicting the probability of the current moment as the start, end, and intermediate of an event frame by frame, while video frames after this moment are invisible to the module. Finally, in the offline stage, positive and negative samples are constructed and the query is reconstructed to optimize the proposal and obtain pseudo-labels, which are then used as supervisory information for predicting probability sequences in the online stage. During the inference phase, the offline module is frozen, and the online event detection is activated separately to achieve accurate retrieval of target moments in online videos under weak supervision.

[0029] A weakly supervised online video moment localization method based on memory perception includes: Multimodal feature fusion is performed on a given video and its text query to obtain a unified frame-level representation at each stage; Using an offline-guided online model architecture, the fused features are input into the offline and online modules respectively, in whole and frame-by-frame form. In the offline module, the query of the masked part of the word is reconstructed by generating a Gaussian mask to obtain the proposal at the start time of the action; In the online module, for each frame of short-term memory features in the input, the features are first enhanced using long-term historical memory within the window, and then attention weights within the window are dynamically generated to calculate the score of the current frame. The proposals obtained from the offline module are used as pseudo-labels to provide supervision information for the score sequence of the online module; Finally, high-performance weakly supervised online time-of-flight localization can be achieved simply by inferring the online module separately.

[0030] Example 2 The difference between the memory-based weakly supervised online video time-localization method described in Example 1 and the method described in Example 1 is as follows: Multimodal feature fusion, including: For uncropped videos, a pre-trained model is used to extract video and query features from the video. The extracted query features are input into a long short-term memory network to transform the query text into a representation with temporal features; The video features and query features are fused through element-wise Hadamard product, and the final feature representation is obtained after normalization.

[0031] Multimodal feature fusion, including: S 11: Regarding video features The corresponding query features are ; The total number of video clips. The number of words in the sentence. For the first Video features of a frame To query the first character in the sentence Features of each word; S 12: Query Representations with temporal characteristics are obtained by processing through a Long Short-Term Memory (LSTM) network. ; S 13: After video features and query features are mapped to the same feature space through fully connected layers, the final fused features are obtained by calculating and normalizing the Hadamard product. ; This refers to the final feature representation; in, and These are the weight parameters of the fully connected layer. It represents the Hadamah accumulation. Denotes the Frobenius norm. It is the transpose of a vector that is all one.

[0032] In the offline module, the query for the masked part of the word is reconstructed by generating a Gaussian mask to obtain the proposal at the start of the action; including: The fused features are input into the offline module as a whole, and the probability distribution of the action at each moment in the preset video is a Gaussian distribution; For each proposal, calculate the center position and width of a Gaussian mask to determine the time boundary of the action; We extract positive and negative samples from videos and use query reconstruction to evaluate the quality of proposals.

[0033] In the offline module, the query for the masked part of the word is reconstructed by generating a Gaussian mask to obtain the proposal at the start of the action; including: S21: In the offline proposal generation stage, the occurrence probability of events in the video is modeled using a Gaussian distribution, that is, it is assumed that the target event is generated by a central... and width Gaussian morphological probability sequence It means that, among them, for Given the probability of the target event occurring at time t, and based on the above modeling method for the target event, the corresponding time boundary is obtained as follows: Positioning proposal; S 22: Integrating features The overall average over time is then used for prediction via a fully connected layer. For the center and width of the Gaussian distribution As shown below: ; ; in, This is a representation of the fused features after averaging over the time dimension. It is the sigmoid activation function. and These are the weights and offset parameters in the fully connected layer, respectively.

[0034] For each proposal, the following calculations are obtained. The probability of an event occurring at a given time. : ; in, For time series not being At any time, , All are normalized position coordinates. , It is used for control The scaling parameter; S 23: Introducing diversity loss It acts on previously predicted Stack probability sequence corresponding to each proposal , in For the first A probability sequence: ; in, It is the Frobenius norm. Controlling the degree of overlap between proposals The identity matrix is ​​used; the diversity loss effectively prevents redundant proposals from converging to similar time positions, while promoting comprehensive coverage of potential events throughout the video. S 24: Use query reconstruction to measure the quality of localization, and transform the original query features One-third of the words are randomly replaced with specific symbols. The cross-modal semantic representation under the positive sample mask is obtained. : ; in, and They are respectively using The Transformer decoder and encoder are used as attention maps to replace traditional attention mechanisms; then, after passing through a fully connected layer, the probability distribution of the next word in the vocabulary with an added event mask is obtained. : ; in, It is a fully connected layer. This indicates the first [item] in the original query text. The features of each word, i.e., the prediction target; Indicates the query after random replacement Center front Features of each word; Finally, cross-entropy loss is used to calculate... Right now Difference from the true distribution : Recorded as in Query reconstruction loss obtained under the proposal: ; in, Indicates the original query The Middle Each word feature For the original query The number of words in the text; S 25: Achieve comparative learning through positive and negative proposal sample mining. As a positive sample The regions on the left and right sides are used as negative samples. and The entire video region serves as a difficult negative sample. Query reconstruction loss for negative samples and hard negative samples and By using respectively , and Replacement of S24 The final reconstruction loss was obtained. for: ; in, It refers to the cross-entropy loss between the predicted word features and the actual word features; Furthermore, for a single video, there should be... Therefore, a contrast loss function is introduced. The following formula is used to minimize Value to determine The optimal choice among the proposals, denoted as [number] , and then make Participating in the contrast loss function The calculation is as follows: ; , They are respectively determined as number After the proposal, the query reconstruction loss of the negative samples and hard negative samples corresponding to the proposal; , These refer to the two negative samples generated from the left and right sides of the proposal, respectively. Interval parameters and Controlling the separation strength between different sample categories and satisfying .

[0035] In the online module, for each input frame's short-term memory features, they are first enhanced using long-term historical memory within a window, then their attention weights within the window are dynamically generated, and the score for the current frame is calculated using a weighted average; including: The fused features are input into the online module frame by frame, and a sliding window containing long-term historical information is set for each frame; After processing long-term and short-term features, they are input into the decoder for cross-attention calculation to obtain enhanced frame features; The enhanced features are used to predict the probability of events occurring in the current frame using a long short-term memory network; The score at the current moment is calculated using dynamic programming with weighted averages, and the predicted scores are then calculated under different event states.

[0036] In the online module, for each input frame's short-term memory features, they are first enhanced using long-term historical memory within a window, then their attention weights within the window are dynamically generated, and the score for the current frame is calculated using a weighted average; including: S 31: For example Figure 2 As shown, in the online module, features are fused. Processed frame by frame in chronological order; considering the lack of context information in a single frame, the length is set to... A sliding window represents the current frame as and including The historical information features of a frame are represented as ,Will joint action Input to the online module at any given time; additionally, in the initial stage At that time, the visible frame rate had not yet reached [the target value]. ,at this time The header is padded with a learnable embedding feature to keep the window length fixed.

[0037] S 32: Used to enhance short-term information contained in the current frame; features in the window are represented as... ,Will The input is fed into a standard Transformer decoder, where, As keys and values, As a query: ; Among them, CrossAttn( ) is the cross-attention layer in the decoder block. and It is a projection matrix. It is the final enhanced feature obtained. The dimensions of the projected key vector and query vector; S 33: For example Figure 2 As shown, firstly, the enhanced features... The generation of predicted probabilities is specifically accomplished by a Long Short-Term Memory (LSTM) network with linear layers. At each time step, the hidden state of the LSTM unit... and unit state All have been updated to: ; in, , These refer to the hidden state and the cell state of the previous time step, respectively. Subsequently, After linear layer transformation, the probabilities of the initial, final, and intermediate states are obtained respectively: ; in, For weight parameters, For deviation, hour, Indicates the event is in The probability that a given moment belongs to the initial stage; hour, Indicates the event is in The probability that a given moment belongs to the termination phase; hour, Indicates the event is in The probability that a given moment belongs to the middle stage; S 34: For example Figure 2 As shown, the enhancement features are then... Calculate frame context weights: ; in, , and All are learnable parameters; These are intermediate variables in the weight calculation process; for At each time step, the weights are multiplied by the memory probabilities within the entire window to obtain the score, as shown below: ; in, For the current moment, The earliest time in the current sliding window. The default window size. This is the actual window size at the current moment, because... For the normalized weight sequence in the window, It is a list of probabilities for each moment in the window to be a different event stage; and Both refer to any one in and In between, , and They are respectively , and Attention weights after time-normalization; for The moment serves as the final score for different stages of the event; , They are respectively and The probability of time as different stages of an event; when Upon reaching the last frame, the online event prediction score sequence for the entire video is obtained. : ; in, Predict score sequences for online events throughout the entire video. yes The moment serves as the final score for different stages of the event; Finally, the outer product of the initial score sequence and the ending score sequence is performed to obtain the event prediction score map. Then, the high-score regions are filtered out using the maximum suppression algorithm to obtain the final online prediction result. ; in, This refers to the final result obtained from online prediction, i.e., the location segment of the target query in the video; This refers to the maximization suppression algorithm. See “Efficient non-maximum suppression,” in 18th international conference on pattern recognition (ICPR'06), vol. 3. IEEE, 2006, pp. 850–855.

[0038] Table 1 shows the comparative model of the present invention; Table 1. Comparison Model Introduction;

[0039] Table 2 shows the comparison results of the positioning accuracy of the present invention and the modified baseline model; Table 2. Comparison of positioning accuracy between the present invention and the modified baseline model.

[0040]

[0041] As shown in Table 2, WISER has a significantly superior positioning accuracy compared to the modified baseline model.

[0042] Example 3 A computer device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the memory-aware, weakly supervised online video time-of-motion localization method described in Embodiment 1 or 2.

[0043] Example 4 A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the memory-aware, weakly supervised online video time-localization method described in Embodiment 1 or 2.

[0044] Example 5 A weakly supervised online video time-of-motion localization system based on memory perception includes: The multimodal feature fusion module is configured to: add temporal semantics to the query features, perform Hadamard product calculation between the video and the query features, and obtain cross-modal feature encoding; The offline event mask proposal generation module is configured to: calculate Gaussian masks for positive and negative samples using global features, reconstruct queries to optimize mask generation, and construct pseudo boundaries; The online event detection module is configured to: set up a sliding window containing long-term information, enhance the contextual expression of short-term information, and use an in-window weighting strategy as a scoring mechanism.

Claims

1. A weakly supervised online video time-of-motion localization method based on memory perception, characterized in that, include: Multimodal feature fusion is performed on a given video and its text query to obtain a unified frame-level representation at each stage; Using an offline-guided online model architecture, the fused features are input into the offline and online modules respectively, in whole and frame-by-frame form. In the offline module, the query of the masked part of the word is reconstructed by generating a Gaussian mask to obtain the proposal at the start time of the action; In the online module, for each frame of short-term memory features in the input, the features are first enhanced using long-term historical memory within the window, and then attention weights within the window are dynamically generated to calculate the score of the current frame. The proposals obtained from the offline module are used as pseudo-labels to provide supervision information for the score sequence of the online module; Finally, high-performance weakly supervised online time-of-flight localization can be achieved simply by inferring the online module separately.

2. The method for weakly supervised online video time localization based on memory perception according to claim 1, characterized in that, Multimodal feature fusion, including: For uncropped videos, a pre-trained model is used to extract video and query features from the video. The extracted query features are input into a long short-term memory network to transform the query text into a representation with temporal features; The video features and query features are fused through element-wise Hadamard product, and the final feature representation is obtained after normalization.

3. The method for weakly supervised online video time localization based on memory perception according to claim 2, characterized in that, Multimodal feature fusion, including: S 11: Regarding video features The corresponding query features are ; The total number of video clips. The number of words in the sentence. For the first Video features of a frame To query the first character in the sentence Features of each word; S 12: Query Representations with temporal characteristics are obtained by processing through a Long Short-Term Memory (LSTM) network. ; S 13: After video features and query features are mapped to the same feature space through fully connected layers, the final fused features are obtained by calculating and normalizing the Hadamard product. ; This refers to the final feature representation; in, and These are the weight parameters of the fully connected layer. It represents the Hadamah accumulation. Denotes the Frobenius norm. It is the transpose of a vector that is all one.

4. The weakly supervised online video time-of-motion localization method based on memory perception according to claim 1, characterized in that, In the offline module, the query for the masked part of the word is reconstructed by generating a Gaussian mask to obtain the proposal at the start of the action; including: The fused features are input into the offline module as a whole, and the probability distribution of the action at each moment in the preset video is a Gaussian distribution; For each proposal, calculate the center position and width of a Gaussian mask to determine the time boundary of the action; We extract positive and negative samples from videos and use query reconstruction to evaluate the quality of proposals.

5. The weakly supervised online video time-of-motion localization method based on memory perception according to claim 4, characterized in that, In the offline module, the query for the masked part of the word is reconstructed by generating a Gaussian mask to obtain the proposal at the start of the action; including: S 21: In the offline proposal generation stage, the occurrence probability of events in the video is modeled using a Gaussian distribution, that is, it is assumed that the target event is generated by a central... and width Gaussian morphological probability sequence It means that, among them, for The probability of the target event occurring at time t is given, and based on the modeling method for the target event, the corresponding time boundary is obtained as follows: Positioning proposal; S 22: Integrating features The overall average over time is then used for prediction via a fully connected layer. For the center and width of the Gaussian distribution ; For each proposal, the following calculations are obtained. The probability of an event occurring at a given time. : ； in, For time series not being At any time, , All are normalized position coordinates. , It is used for control The scaling parameter; S 23: Introducing diversity loss It acts on previously predicted Stack probability sequence corresponding to each proposal , in For the first A probability sequence: ； in, It is the Frobenius norm. Controlling the degree of overlap between proposals It is the identity matrix; S 24: Use query reconstruction to measure the quality of localization, and transform the original query features One-third of the words are randomly replaced with specific symbols. The cross-modal semantic representation under the positive sample mask is obtained. : ； in, and They are respectively using The Transformer decoder and encoder are used as attention maps to replace traditional attention mechanisms; then, after passing through a fully connected layer, the probability distribution of the next word in the vocabulary with an added event mask is obtained. : ； in, It is a fully connected layer. This indicates the first [item] in the original query text. The features of each word, i.e., the prediction target; Indicates the query after random replacement Center front Features of each word; Finally, cross-entropy loss is used to calculate... Right now Difference from the true distribution : ； in, Indicates the original query The Middle Each word feature For the original query The number of words in the text; S 25: Achieve comparative learning through positive and negative proposal sample mining. As a positive sample The regions on the left and right sides are used as negative samples. and The entire video region serves as a difficult negative sample. Query reconstruction loss for negative samples and hard negative samples and By using respectively , and Replacement of S24 The final reconstruction loss was obtained. for: ； in, It refers to the cross-entropy loss between the predicted word features and the actual word features; Furthermore, for a single video, there should be... Therefore, a contrast loss function is introduced. The following formula is used to minimize Value to determine The optimal choice among the proposals, denoted as [number] , and then make Participating in the contrast loss function The calculation is as follows: ； , They are respectively determined as number After the proposal, the query reconstruction loss of the negative samples and hard negative samples corresponding to the proposal; , These refer to the two negative samples generated from the left and right sides of the proposal, respectively. Interval parameters and Controlling the separation strength between different sample categories and satisfying .

6. The weakly supervised online video time-of-motion localization method based on memory perception according to claim 1, characterized in that, In the online module, for each input frame's short-term memory features, they are first enhanced using long-term historical memory within a window, then their attention weights within the window are dynamically generated, and the score for the current frame is calculated using a weighted average; including: The fused features are input into the online module frame by frame, and a sliding window containing long-term historical information is set for each frame; After processing long-term and short-term features, they are input into the decoder for cross-attention calculation to obtain enhanced frame features; The enhanced features are used to predict the probability of events occurring in the current frame using a long short-term memory network; The score at the current moment is calculated using dynamic programming with weighted averages, and the predicted scores are then calculated under different event states.

7. The weakly supervised online video time-of-motion localization method based on memory perception according to claim 6, characterized in that, In the online module, for each input frame's short-term memory features, they are first enhanced using long-term historical memory within a window, then their attention weights within the window are dynamically generated, and the score for the current frame is calculated using a weighted average; including: S 31: In the online module, feature fusion Processed frame by frame in chronological order; length set to... A sliding window represents the current frame as and including The historical information features of a frame are represented as ,Will joint action Input to the online module at any given time; additionally, in the initial stage At that time, the visible frame rate had not yet reached [the target value]. ,at this time The header is filled with a learnable embedding feature; S 32: Used to enhance short-term information contained in the current frame; features in the window are represented as... ,Will The input is fed into a standard Transformer decoder, where, As keys and values, As a query: ； Among them, CrossAttn( ) is the cross-attention layer in the decoder block. and It is a projection matrix. It is the final enhanced feature obtained. The dimensions of the projected key vector and query vector; S 33: First, regarding the enhanced features The generation of predicted probabilities is specifically accomplished by a Long Short-Term Memory (LSTM) network with linear layers. At each time step, the hidden state of the LSTM unit... and unit state All have been updated to: ； in, , These refer to the hidden state and the cell state of the previous time step, respectively. Subsequently, After linear layer transformation, the probabilities of the initial, final, and intermediate states are obtained respectively: ； in, For weight parameters, For deviation, hour, Indicates the event is in The probability that a given moment belongs to the initial stage; hour, Indicates the event is in The probability that a given moment belongs to the termination phase; hour, Indicates the event is in The probability that a given moment belongs to the middle stage; S 34: Subsequently, enhancement features were applied. Calculate frame context weights: ； in, , and All are learnable parameters; These are intermediate variables in the weight calculation process; for At each time step, the weights are multiplied by the memory probabilities within the entire window to obtain the score, as shown below: ； in, For the current moment, The earliest time in the current sliding window. The default window size. This is the actual window size at the current moment, because... For the normalized weight sequence in the window, It is a list of probabilities for each moment in the window to be a different event stage; and Both refer to any one in and In between, , and They are respectively , and Attention weights after time-normalization; for The moment serves as the final score for different stages of the event; , They are respectively and The probability of time as different stages of an event; when Upon reaching the last frame, the online event prediction score sequence for the entire video is obtained. : ； in, Predict score sequences for online events throughout the entire video. yes The moment serves as the final score for different stages of the event; Finally, the outer product of the initial score sequence and the ending score sequence is performed to obtain the event prediction score map. Then, the high-score regions are filtered out using the maximum suppression algorithm to obtain the final online prediction result. ； in, This refers to the final result obtained from online prediction, i.e., the location segment of the target query in the video; This refers to the maximization suppression algorithm.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the weakly supervised online video time-location method based on memory perception as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the weakly supervised online video time-localization method based on memory perception as described in any one of claims 1-7.

10. A weakly supervised online video time-of-motion localization system based on memory perception, characterized in that, include: The multimodal feature fusion module is configured to: add temporal semantics to the query features, perform Hadamard product calculation between the video and the query features, and obtain cross-modal feature encoding; The offline event mask proposal generation module is configured to: calculate Gaussian masks for positive and negative samples using global features, reconstruct queries to optimize mask generation, and construct pseudo boundaries; The online event detection module is configured to: set up a sliding window containing long-term information, enhance the contextual expression of short-term information, and use an in-window weighting strategy as a scoring mechanism.

Images