Weak supervision video time sequence positioning method and device based on statement correlation time sequence noise sampling calibration and storage medium

Abstract

The invention provides a weak supervision video time sequence positioning method and device based on statement correlation time sequence noise sampling calibration and a storage medium, and the method comprises the steps: carrying out the feature extraction of an original video and statement query, and obtaining a video feature and statement query feature tensor; correlation modeling and region refinement are carried out on the multi-modal features through a statement correlation time sequence perception module, and region perception enhanced video features are obtained; through gradient guide noise sampling calibration, generating an optimal candidate fragment and a gradient induction fragment based on enhanced features, constructing a comparison calibration unit and calculating noise comparison loss; and the reconstruction loss is queried and calculated through the optimal candidate fragment reconstruction mask statement, and model parameters are iteratively optimized in combination with noise contrast loss. According to the invention, discrete segmentation and local refining coding are carried out on continuous videos to improve the event semantic distinction degree, and a difficult negative sample is generated through gradient-induced disturbance to carry out comparison calibration to improve the video time sequence positioning precision of the model under a weak supervision condition.

Description

Technical Field

[0001] This invention belongs to the field of video analysis and processing, especially the field of video temporal localization, and relates to a weakly supervised video temporal localization method, device and storage medium based on sentence association temporal noise sampling calibration. Background Technology

[0002] With the booming development of short video platforms and Vlog culture, personal video creation has transformed from a marginalized entertainment activity into a mainstream content production model. A large number of individual creators face the prominent problem of inefficient material management in their daily work. Taking travel video bloggers as an example, they typically generate 3-5 hours or even longer of continuous raw footage during a single day of shooting. This footage covers diverse scenes, people, and events, but the final cut released to the audience often only retains 5-10 minutes of core content. This significant disparity in length between raw footage and the final cut directly makes material selection and positioning a key bottleneck restricting creative efficiency. Video temporal positioning technology offers a solution to this technical challenge. This technology constructs a semantic mapping relationship between natural language descriptions and video temporal segments, enabling creators to express the semantic features of target segments in natural language. The system then uses semantic understanding and temporal positioning algorithms to accurately identify and output the temporal boundaries of corresponding segments in the unedited raw footage, automatically labeling the start and end times of video segments related to the natural language descriptions. However, accurately labeling the time periods in a video using query statements is costly, limiting the application scale of video temporal positioning. Therefore, the temporal localization of weakly supervised videos has become a key research focus.

[0003] Currently, weakly supervised video temporal localization faces two main problems: 1) low semantic discriminability and 2) insufficient supervision constraints. Low semantic discriminability refers to the fact that the visual features extracted by the model from the entire video are often a mixture of features from multiple events, leading to semantic entanglement in the feature representation. Insufficient supervision constraints mean that although the weak supervision signals themselves contain rich semantic information, existing methods fail to utilize this information to constrain the model during training, making it difficult to force the model to learn the semantic differences between events. To address the low semantic discriminability problem, existing methods attempt to fuse visual, textual, and motion features, but the fusion process is often coarse-grained, lacking refined encoding for specific event locations. To address the insufficient supervision constraints problem, existing methods randomly mask the query and use proposed fragments to reconstruct the masked query to evaluate the proposal quality. However, existing methods utilize weak supervision signals superficially during training, resulting in weak supervision signals.

[0004] The shortcomings of the aforementioned video temporal localization methods are mainly reflected in the following two aspects: 1) The coarse-grained multimodal feature fusion approach fails to refine the encoding of local events, resulting in a lack of refined modeling ability for local semantics. This makes it difficult to capture fine-grained temporal patterns of short-term events and susceptible to interference from long-term events and background noise, ultimately leading to low semantic discriminativeness; 2) Supervision is achieved through methods such as random masks, resulting in a shallow utilization of weak supervision signals. The lack of a refined adjustment mechanism for candidate segments and the failure to form temporal structure constraints lead to inaccurate event boundary localization. Based on these considerations, there is an urgent need to design a weakly supervised video temporal localization method that can improve event discriminative ability from the perspective of local semantic modeling and strengthen supervision signal constraints through gradient-induced perturbation mechanisms. This would improve localization accuracy in multi-event scenarios and reduce data annotation overhead. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing methods by providing a weakly supervised video temporal localization method based on sentence association temporal noise sampling calibration. This method can improve the semantic discriminativeness of event representation through local semantic modeling and make full use of weakly supervised signals through gradient-induced perturbation, effectively alleviating the problem of multi-event confusion.

[0006] In a first aspect, the present invention provides a weakly supervised video temporal localization method based on sentence association temporal noise sampling calibration, the method comprising: S1: Preprocess the first video to obtain a set of first video frame sequences, input the set of first video frame sequences into a three-dimensional convolutional neural network to extract the corresponding first video feature tensor, preprocess the input first statement query to obtain several texts, and embed the several texts into the model to obtain the corresponding first statement query feature tensor. S2: Through the sentence association time sequence awareness module, the video feature tensor and sentence query feature tensor are relevance modeled and the region is refined to obtain region-aware enhanced video features; among them, the relevance modeling includes cross attention and cosine similarity calculation, and the region refinement includes smooth modal relevance curve and foreground region mask generation. S3: Through the gradient-guided noise sampling calibration module, the best candidate segments and gradient-induced video segments are generated based on the region-aware enhanced video features. A contrast calibration unit is constructed and the noise contrast loss is calculated. The construction of the contrast calibration unit includes the mapping of features of the best candidate segments and gradient-induced video segments in the same dimension, and the calculation of the average pooled global features, matching scores, and noise contrast loss of the best candidate segments and gradient-induced video segments. S4: Reconstruct the masked sentence using the best candidate segments and calculate the reconstruction loss. Optimize the video temporal localization model, which consists of the sentence association temporal awareness module and the gradient-guided noise sampling calibration module, using the stochastic gradient descent algorithm, reconstruction loss, and noise contrast loss.

[0007] In a second aspect, the present invention also provides an apparatus comprising one or more processors and a memory, wherein the memory contains instructions executable by the one or more processors to cause the one or more processors to perform a weakly supervised video temporal localization method based on statement-related temporal noise sampling calibration according to embodiments of the present invention.

[0008] Thirdly, the present invention also provides a storage medium storing a computer program that enables a computer to execute a weakly supervised video temporal localization method based on statement-related temporal noise sampling calibration according to various embodiments of the present invention. Attached Figure Description

[0009] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 The flowchart illustrates a weakly supervised video temporal localization method based on sentence association temporal noise sampling calibration, as provided in an embodiment of the present invention.

[0010] Figure 2 This is a schematic diagram of the structure of a device provided in an embodiment of the present invention. Detailed Implementation

[0011] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0012] It should be noted that the weakly supervised video temporal localization method based on statement-related temporal noise sampling calibration provided in this application does not limit its execution subject. Optionally, the execution subject of the weakly supervised video temporal localization method based on statement-related temporal noise sampling calibration provided in this application can be a weakly supervised video temporal localization system based on statement-related temporal noise sampling calibration. Specifically, the execution subject of the weakly supervised video temporal localization method based on statement-related temporal noise sampling calibration provided in this application can be a weakly supervised video temporal localization unit in a weakly supervised video temporal localization system based on statement-related temporal noise sampling calibration.

[0013] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0014] Figure 1 This is a flowchart illustrating a weakly supervised video temporal localization method based on sentence association temporal noise sampling calibration, as provided in an embodiment of the present invention. Figure 1 As shown, in this invention, a weakly supervised video temporal localization method based on sentence association temporal noise sampling calibration is provided. The method includes: S1: Preprocess the first video to obtain a set of first video frame sequences, input the set of first video frame sequences into a three-dimensional convolutional neural network to extract the corresponding first video feature tensor, preprocess the input first statement query to obtain several texts, and embed the several texts into the model to obtain the corresponding first statement query feature tensor. S2: Through the sentence association time sequence awareness module, the video feature tensor and sentence query feature tensor are relevance modeled and the region is refined to obtain region-aware enhanced video features; among them, the relevance modeling includes cross attention and cosine similarity calculation, and the region refinement includes smooth modal relevance curve and foreground region mask generation. S3: Through the gradient-guided noise sampling calibration module, the best candidate segments and gradient-induced video segments are generated based on the region-aware enhanced video features. A contrast calibration unit is constructed and the noise contrast loss is calculated. The construction of the contrast calibration unit includes the mapping of features of the best candidate segments and gradient-induced video segments in the same dimension, and the calculation of the average pooled global features, matching scores, and noise contrast loss of the best candidate segments and gradient-induced video segments. S4: Reconstruct the masked sentence using the best candidate segments and calculate the reconstruction loss. Optimize the video temporal localization model, which consists of the sentence association temporal awareness module and the gradient-guided noise sampling calibration module, using the stochastic gradient descent algorithm, reconstruction loss, and noise contrast loss.

[0015] Step S1 specifically includes the following steps: In step S11, at a rate of per second Frame sampling rate ( Typically, a value of 16 is used to sample the input video, resulting in a frame count of [number missing]. video frame sequence collection ,in, Represents the set of video frame sequences. Frame video frame, Represents the real number field. and These represent the height and width of the video frame, respectively, and 3 represents the RGB three channels; In step S12, the video frame sequence set is... The input is fed into a 3D convolutional neural network to obtain the video frame feature sequence. ,in, The number of video frames in the set of video frame sequences. The feature dimension (usually 1024); The three-dimensional convolutional neural network used in steps (1-2) is the mainstream I3D or C3D in this field. The video frame feature sequence obtained by the three-dimensional convolutional neural network is used for subsequent video temporal localization. In step S13, the input statement is queried. Perform word segmentation, where, Indicates the first One word, It refers to the number of words; In step S14, a pre-trained word vector model is used to obtain the features of each word, thus obtaining the query feature sequence. ,in, For word element number, The feature dimension (usually 300); In step S15, average pooling is used to... Calculate global statement query characteristics The subscript "global" indicates the global scope. The first in the query characteristics of the statement Features of each word element; The pre-trained word vector model used in step S14 above is GloVe, which is mainstream in this field. The sentence query feature sequence obtained by this pre-trained word vector model is used for subsequent video temporal localization.

[0016] Step S2 specifically includes the following steps: The sentence association time sequence awareness module consists of a cross-modal semantic association modeling submodule and a time sequence region feature enhancement module. Its function is to segment the continuous video timeline into discrete, semantically relatively complete events based on semantic relevance, so as to enhance the model's ability to perceive highly correlated events. A cross-modal semantic association modeling submodule is constructed, which consists of a cross-attention layer and a cosine similarity calculation unit; The cross-attention layer performs the following steps: The input to the cross-attention layer is video frame features. query features ; In step S201, the and The input is fed into three fully connected layers, and the query matrix is ​​calculated for each layer. Key matrix Sum matrix ,in, , , For the learnable matrix of the cross-attention layer, This represents the number of channels for the query feature, key feature, and value feature, taken as... ; In step S202, the cross-modal attention score of video frame features and statement query features is calculated. The subscript "attn" indicates attention. This represents the exponential normalization (Softmax) function. Indicates the inner product. Indicates along the feature dimension Average pooling; The cosine similarity calculation unit performs the following steps: The input to the cosine similarity calculation unit is video frame features. and global statement query features ; In step S203, the video frame features are first... and global statement query features The input is fed into two fully connected layers and mapped to the same dimension. , The superscript "proj" indicates projection. , The feature projection matrix is ​​a learnable feature. Represents the common feature dimension; In step S204, the cosine similarity between the video frame projection features and the global statement query projection features is calculated. In this context, the subscript "cos" indicates cosine similarity. Indicates the first Cosine similarity between frame-to-frame projection features and global statement query projection features. Indicates the first Frame projection features of video frames Represents the 2-norm of a vector; In step S205, the visual text relevance curve is calculated, and the calculation formula is as follows: ,in, and represent the hyperparameters balancing the importance of cross-attention scores and cosine similarity, respectively. ; The strategy of weighted fusion of attention score and cosine similarity aims to leverage the fine-grained information capture capability of cross attention and the global semantic alignment capability of cosine similarity to separate each event in a complex video into clear peak signals, thereby improving the learning efficiency of weak supervision. A region feature enhancement submodule is constructed, which consists of a correlation smoothing and region search unit and a feature refinement unit; The input to the peak region mask generation unit is The output is a video frame mask. Since the generated modal correlation curve is based on the fusion of cross-attention score and cosine similarity, its frame-by-frame calculation method is prone to generating non-smooth noise signals. In order to suppress noise interference and obtain a smoother curve, the correlation smoothing and region search unit performs the following steps: In step S206, The input is smoothed using a Gaussian smoother to obtain a smoothed modal correlation curve. The subscript "smooth" indicates smoothness. It is the first The smoothness correlation score of video frames. It is a Gaussian kernel. It is the index of the Gaussian kernel. It is the half-width of the Gaussian kernel; In step S207, after obtaining the smooth modal correlation curve, peak detection is first performed to obtain the peak value. A set of video frame indexes ,in, Represents a local domain window (usually with a value of 10); In step S208, a symmetric padding method is used when processing boundary correlation scores, for each Define its search threshold ,in, It is the search factor (usually with a value of 0.5), from Start searching for the left boundary: , Similarly, the right boundary is obtained: , This forms the initial foreground region: ; Furthermore, from a global perspective, considering that local peak detection is prone to falling into the local extremum trap, a global filtering mechanism was designed to eliminate such invalid noise. This mechanism forcibly filters out regions that, although possessing peak characteristics, have insufficient overall signal strength, thereby obtaining the final foreground region. ,in, This represents the global filtering threshold (usually set to 0.5), and finally... Constructing a foreground region mask ,in, ; The feature refinement unit performs the following operations: The input to the feature refinement unit is the foreground region mask. The video frame feature sequence obtained in step S12 and the query feature sequence of the statement obtained in step S13 The output is region-aware enhanced video features. The function of this unit is to use the statement query as a guide to locally encode the video region features selected by the mask in order to enhance the key details related to the query. In step S209, the region mask is... Along Expanding the dimensions yields a feature sequence of video frames. A mask matrix with a consistent shape By using the mask matrix Applied to video frame feature sequences to focus on video features in peak regions. In this context, the superscript "fg" indicates foreground features. Indicates element-wise multiplication; In step S210, respectively and The input is fed into three fully connected layers, and the query matrix is ​​calculated for each layer. Key matrix Sum matrix ,in, , , For the learnable matrix of the cross-attention layer, This represents the number of channels for the query feature, key feature, and value feature, taken as... ; In step S211, the query context features of the statement are calculated. ; In step S212, Restore to ; In step S213, the query context features and video frame features are added together via residual join to obtain region-aware enhanced video features: ,in, It is a learnable projection matrix; In the above steps, a region mask is generated by Gaussian smoothing and peak detection mechanism to segment the continuous video stream into discrete semantic events, thereby achieving accurate localization of key regions. At the same time, the mask-guided cross-attention mechanism is used to refine local features, filter out background noise and enhance semantic alignment, thereby improving the model's ability to perceive and express fine-grained temporal structures in complex scenes.

[0017] Step S3 specifically includes the following steps: The gradient-guided noise sampling calibration module consists of a candidate video segment generation unit, a gradient-guided noise sampling submodule, and a comparison calibration unit; the input to the gradient-guided noise sampling calibration module is the region-aware enhanced video features generated in step S2. and the query feature tensor generated in step S1 The gradient-guided noise sampling calibration module aims to address the lack of supervision by generating a constrained gradient perturbation-hard negative sample and utilizing its semantic difference from the original candidate fragment to form a strong supervisory constraint signal for the model. Construct a candidate video segment generation unit, which is used to perceive and enhance video features from the region. Location and query features of the statement The most relevant time sequence segment; In step S301, the unit is in Add a learnable category label to it. and with The inputs are combined and interacted with in a Transformer to obtain multimodal features: ,in, The dimension representing multimodal features and These represent the Transformer encoder and decoder, respectively. at this time, The last sub-feature It contains all video frames and statement query information; in step S302, this unit will... The input is fed into a fully connected layer for prediction. Gaussian mask parameters: ,in, This represents a multilayer perceptron. and These represent the center and width of the video proposal, respectively. Typically, 8 is chosen, and each Gaussian mask is used at the th... The formula for a frame is expressed as: ; In step S303, the following is obtained: After proposing the nth candidate, the multimodal features are weighted and pooled using a Gaussian mask to obtain the nth... The aggregated feature tensor of candidate segments The superscript "pool" indicates pooling aggregation; In step S304, Features of global statement queries Projected onto the same feature space ,in, It is a learnable feature projection matrix; In step S305, the calculation of the first... Score of each candidate proposal Finally, the candidate segment with the highest score is selected. As the best candidate, its corresponding original video features are: As positive samples, among them This represents the features of the best candidate video segment. Indicates the actual duration of the video; Construct a gradient-guided noise sampling submodule, with the best candidate video segment features as input. This module aims to generate a hard negative sample in the feature space, that is, a feature that is highly similar to the original fragment in non-critical regions but semantically inverted in critical semantic regions. In step S306, the gradient attribution method is used to locate the location. The most sensitive feature dimension for query statements is used to calculate the matching score. Features gradient In this context, the subscript "attr" indicates the degree of contribution. This represents max pooling over the time dimension; In step S307, the absolute value of the gradient is taken as the importance weight. The subscript "imp" indicates importance. In step S308, a threshold is set. (Typically a value of 0.6) Generate a binary region mask. ,in , The eigenvector represents the first eigenvector. One dimension, Indicates the first The importance weights of each dimension In This indicates the region within the corresponding video segment that is semantically strongly related to the query statement. In step S309, with Initial state Where the subscript "GI" indicates gradient induction, and the process is as follows: The round of iterative updates aims to minimize its match with the query statement; Each iteration performs the following steps: In the During rounds of iteration: Calculate current features Match score with the query statement and its gradient ;in, To query the score in the nth round of calculation right The gradient of ▽ is used to find the sign of the gradient. To stabilize the generation process, a momentum update strategy is introduced to calculate the first moment. and second moment ; Calculate the gradient after bias correction ; exist Reverse update features within the constrained region ,in, This represents the noise injection coefficient (usually taken as 0.1). express The binary region mask matrix obtained after broadcasting along the temporal dimension. and This represents the momentum hyperparameter. Indicates the smoothing term. Represents element-wise product; go through After round of iterations This is the final gradient-induced video segment feature. .

[0018] The above steps utilize gradient attribution to locate key semantic regions and generate difficult negative samples that are similar to non-key regions but have reversed key semantics through momentum iteration strategy. The highly confusing features constructed by gradient-induced perturbation mechanism force the model to focus on fine-grained semantic differences rather than background information, thereby enhancing the model's ability to finely distinguish the consistency between video content and text semantics. Construct a comparison alignment unit, with the best candidate video segment features output from step S305 as input. Gradient-induced video segment features output in step S309 and the global statement query features obtained in step S15 ; In step S310, , and Mapped to the same dimension , , , ; In step S311, average pooling is used to calculate... and Global features: , ; In step S312, calculate respectively and and Matching score , ; In step S313, the noise contrast loss is calculated. ,in The boundary threshold is designed to force the matching scores of positive and negative samples to maintain the minimum distinguishing margin in the feature space, thereby enhancing the discriminative power of the model's feature representation; the final output is the temporal interval of the best candidate video segment.

[0019] The above steps calculate the matching score through unified semantic mapping and construct a ranking loss function based on boundary thresholds, which forcibly widens the feature distance between the best candidate segment and the gradient-induced segment. This mechanism establishes a clear semantic boundary between positive and negative samples, eliminates the interference of similar events by maximizing the discrimination interval, and improves the discriminativeness and location reliability of the model in the feature space.

[0020] Step S4 specifically includes the following steps: A video temporal localization model is constructed, consisting of the sentence association temporal awareness module in S2 and the gradient-guided noise sampling and calibration module in S3. This model is used to extract frames from the training video at a fixed frame rate and input them into the video temporal localization model to obtain video proposals (best video candidate segment features) related to the sentence query. The corresponding video, (obtained from step S305) and noise contrast loss (Obtained from step S313, for use in step S45 below); The statement association timing awareness module performs the following steps: In step S41, the query reconstruction loss is calculated, and the mask ratio is first set. (Usually the value is taken as) ), query the statement in step S13 Perform a random masking operation, replacing the word at the selected position with the special marker [MASK], to obtain the masked query statement. The superscript "masked" indicates that the query is masked, and the masked query feature sequence is obtained by extracting features using a pre-trained word vector model. ; In step S42, the features of the best video candidate segments selected in step S305 are... Querying feature sequences with mask statements The input is fed into the multimodal reconstruction decoder; the multimodal reconstruction decoder calculates the fused feature sequence through a cross-attention mechanism. The specific calculation process is as follows: Define three learnable linear projection matrices. , and The superscript "Re" indicates a refactoring. It is the attention hidden layer dimension, which uses masked statements to query feature sequences and generate a query vector matrix. Key vector matrix Sum value vector matrix Next, the multimodal fusion features of the query statement and video clips are calculated. Finally, in order to integrate the original query information and enhance feature representation capabilities, the aggregated context features are... Features of querying original mask statements Residual connections and layer normalization are performed, and a nonlinear transformation is applied through a feedforward neural network to obtain the final fused feature sequence: ,in, Indicates the length of the sequence queried by the statement; The gradient-guided noise sampling calibration module performs the following steps: In step S43, after obtaining the fused feature sequence, the masked first feature sequence is then applied to the query statement. Extract the corresponding fusion feature vector from each position. The algorithm maps to the vocabulary space through a fully connected layer and calculates the probability distribution of that position in each word of the vocabulary. ,in, and Let these represent the learnable mapping weight matrix and the bias, respectively. This represents the size of the vocabulary used to train the word vector model; In step S44, the reconstruction loss of the reconstructed word probability distribution and the original query at the mask position is calculated. ,in, This represents the set of indices of the masked words. express The number of elements in the middle, This indicates the index of the masked word in the query statement. Indicates the location The actual words in the original query statement. Indicates the position of the mask. Predicted words not actually exist The probability, Represents the natural logarithm operation; In step S45, the total loss of the video temporal localization model consists of the reconstruction loss obtained in step S44 and the noise contrast loss obtained in step S313. The composition, and the specific calculation formula are as follows: ;in, The weight hyperparameters for the noise contrast loss are used to balance the importance of semantic reconstruction and gradient perturbation calibration in model training.

[0021] In step S46, based on the total loss obtained in step S45, the video temporal localization model consisting of the sentence association temporal awareness module and gradient-guided noise sampling calibration is optimized using the stochastic gradient descent algorithm; steps S1 to S4 are executed sequentially for new video sequences and sentence queries until the model converges.

[0022] In steps S41 to S46 above, the performance evaluation metric for the trained model is “Recall@n, IoU=m” (Recall at n for IoU threshold m). This metric represents the proportion of the number of candidate video segments predicted by the model where at least one segment has an intersection-union ratio (IoU) greater than the threshold m with the real time segment. It comprehensively reflects the accuracy and recall capability of the model's localization.

[0023] In step S5, steps S1 to S4 are executed sequentially on other video feature tensors to obtain the temporal intervals of all statement queries in the corresponding videos until the model converges. After step S5, the following is also included: S6: Input the second video feature tensor and the second statement query into the converged video temporal localization model to obtain the start and end times of the video segments in the second video that are related to the second statement query.

[0024] Step S6 specifically includes the following steps: In step S61, the new video sequence and sentence query are preprocessed according to step S1 to obtain the video frame feature sequence set and sentence query features, which are then input into the trained video temporal localization model. During the inference process, the gradient-guided noise sampling submodule and the comparison and alignment unit are removed, and only the sentence association temporal perception module and the candidate video segment generation unit are retained for forward computation. In step S62, the input video frame feature sequence is refined using the statement association time sequence awareness module to obtain region-aware enhanced video features. The data is input into the candidate video segment generation unit to generate... Gaussian mask parameters It also calculates the matching score between the features of each candidate video segment and the global query features. Select the candidate video segment index with the highest matching score. As a prediction result, based on its corresponding center and width Determine the start time of the video segment. and end time ,in, This indicates the total duration of the original video. and This refers to the temporal boundaries of the final output video segments related to the query statement.

[0025] The above embodiments propose a weakly supervised video temporal localization method based on sentence association temporal noise sampling calibration. This method has the following features: 1) It uses a sentence association temporal perception module to discretize and refine continuous video features semantically. By fusing cross-attention and cosine similarity, it constructs a modal correlation curve and combines a peak region masking mechanism to focus on key regions. The model can segment complex video streams into semantically complete independent events, thereby enhancing the perception ability of local fine-grained temporal patterns; 2) It uses gradient-guided noise sampling calibration to strengthen the weakly supervised signal through a gradient-induced perturbation mechanism. It generates "difficult negative samples" that are highly similar in non-key regions but reversed in key semantic regions through gradient attribution. It also uses a contrast calibration unit to forcibly increase the feature distance between the best candidate segment and the gradient perturbation segment, ensuring that the model can accurately distinguish between real events and highly confusing background noise.

[0026] The above embodiments address the issues of low semantic discriminativeness and insufficient supervision constraints in weakly supervised video temporal localization, offering the following advantages: 1) By using a sentence association temporal awareness module, the continuous video timeline is discretized and locally refined, enabling the model to capture fine-grained temporal patterns of short-term events, rather than just coarse-grained multimodal fusion, thereby improving the semantic discriminativeness of event representation; 2) By generating gradient-perturbation-difficult negative samples through gradient-guided noise sampling calibration and performing comparative calibration, the model can form strong supervision constraint signals even in the absence of precise boundary annotations, improving localization accuracy in multi-event scenarios. This provides a weakly supervised video temporal localization solution to the problem of high temporal boundary annotation costs in actual video creation and material selection.

[0027] Figure 2 This is a schematic diagram of the structure of a device provided in an embodiment of the present invention.

[0028] like Figure 2As shown, in another aspect, this application also provides a device 200, including one or more central processing units (CPUs) 201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 202 or a program loaded from a storage portion 208 into a random access memory (RAM) 203. The RAM 203 also stores various programs and data required for the operation of the device 200. The CPU 201, ROM 202, and RAM 203 are interconnected via a bus 204. An input / output (I / O) interface 205 is also connected to the bus 204.

[0029] The following components are connected to I / O interface 205: an input section 206 including a keyboard, mouse, etc.; an output section 207 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 208 including a hard disk, etc.; and a communication section 209 including a network interface card such as a LAN card, modem, etc. The communication section 209 performs communication processing via a network such as the Internet. Drive 210 is also connected to I / O interface 205 as needed. Removable media 211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 210 as needed so that computer programs read from them can be installed into storage section 208 as needed.

[0030] In particular, according to embodiments of this disclosure, the methods described in any of the above embodiments can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program containing program code for performing the methods. In such embodiments, the computer program can be downloaded and installed from a network via communication section 209, and / or installed from removable medium 211.

[0031] In another aspect, this application also provides a computer-readable storage medium, which may be a computer-readable storage medium included in the apparatus of the above embodiments; or it may be a standalone computer-readable storage medium not assembled into a device. The computer-readable storage medium stores one or more programs that are used by one or more processors to perform the methods described in this application.

[0032] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or operation, or by a combination of dedicated hardware and computer instructions.

[0033] The units or modules described in the embodiments of this application can be implemented in software or hardware. The described units or modules can also be located in a processor; for example, each unit can be a software program located in a computer or mobile smart device, or a separately configured hardware device. The names of these units or modules do not, in some cases, constitute a limitation on the unit or module itself.

[0034] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the concept of this application. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A weakly supervised video temporal localization method based on sentence association temporal noise sampling calibration, characterized in that, include: S1: Preprocess the first video to obtain a set of first video frame sequences, input the set of first video frame sequences into a three-dimensional convolutional neural network to extract the corresponding first video feature tensor, preprocess the input first statement query to obtain several texts, and input the several texts into the word embedding model to obtain the corresponding first statement query feature tensor. S2: Through the sentence association time sequence awareness module, the video feature tensor and the sentence query feature tensor are relevance modeled and the region is refined to obtain region awareness enhanced video features; wherein, the relevance modeling includes cross attention and cosine similarity calculation, and the region refinement includes smooth modal relevance curve and foreground region mask generation; S3: Through the gradient-guided noise sampling calibration module, the best candidate segment and gradient-induced video segment are generated based on the region-aware enhanced video features, a contrast calibration unit is constructed, and the noise contrast loss is calculated; wherein, the construction of the contrast calibration unit includes the mapping of the best candidate segment and the gradient-induced video segment to features in the same dimension, and the calculation of the average pooled global features, matching score, and noise contrast loss of the best candidate segment and the gradient-induced video segment. S4: Reconstruct the masked statement using the best candidate fragment and calculate the reconstruction loss. Optimize the video temporal localization model composed of the statement association temporal awareness module and the gradient-guided noise sampling calibration module using the stochastic gradient descent algorithm, the reconstruction loss, and the noise contrast loss.

2. The method according to claim 1, characterized in that, Step S1 includes: S11: at a rate of per second The frame sampling rate is used to sample the first video, resulting in a number of frames. video frame sequence collection ,in, Represents the set of video frame sequences. Frame video frame, Represents the real number field. and These represent the height and width of the video frame, respectively, and 3 represents the RGB three channels; S12: Set the video frame sequence The input is fed into a 3D convolutional neural network to obtain the video frame feature sequence. ;in, The number of video frames in the set of video frame sequences. For feature dimensions; S13: Query the input statement Perform word segmentation, where, This represents the nth word. It refers to the number of words; S14: Use a pre-trained word vector model to obtain the features of each word and get the query feature sequence. ,in, For word element number, For feature dimensions; S15: Utilizing average pooling Calculate global statement query characteristics The subscript "global" indicates the global scope. The first in the query characteristics of the statement Features of each word element.

3. The method according to claim 2, characterized in that, The statement association time sequence awareness module consists of a cross-modal semantic association modeling submodule and a time sequence region feature enhancement module. The cross-modal semantic association modeling submodule consists of a cross-attention layer and a cosine similarity calculation unit. The region feature enhancement submodule consists of a correlation smoothing and region search unit and a feature refinement unit. Step S2 includes: The cross-attention layer performs the following steps: S201: The video frame feature sequence and the query feature sequence of the statement The input is fed into three fully connected layers, and the query matrix is ​​calculated for each layer. Key matrix Sum matrix ;in, , , For the learnable matrix of the cross-attention layer, This represents the number of channels for the query feature, key feature, and value feature. ; S202: Calculate the cross-modal attention score for video frame features and statement query features: ； The subscript "attn" indicates attention. Represents the exponential normalization function, Indicates the inner product. Indicates along the feature dimension Average pooling; The cosine similarity calculation unit performs the following steps: S203: The video frame feature sequence and global statement query features The input is fed into two fully connected layers and mapped to the same dimension. , The superscript "proj" indicates projection. , The feature projection matrix is ​​a learnable feature. Represents the common feature dimension; S204: Calculate the cosine similarity between the projection features of the video frame and the projection features of the global statement query. The subscript "cos" indicates cosine similarity. Indicates the first Cosine similarity between frame-to-frame projection features and global statement query projection features. This represents the projection feature of the t-th video frame. Represents the 2-norm of a vector; S205: Calculate the visual text relevance curve: ;in, and represent the hyperparameters balancing the importance of cross-attention scores and cosine similarity, respectively. ; The correlation smoothing and region search unit performs the following steps: S206: Will The input is smoothed using a Gaussian smoother to obtain a smoothed modal correlation curve: ； The subscript "smooth" indicates smoothness. It is the smooth correlation score of the t-th video frame. It is a Gaussian kernel. It is the index of the Gaussian kernel. It is the half-width of the Gaussian kernel; S207: After obtaining the modal correlation curve, peak detection is performed to obtain the peak value. A set of video frame indexes: ,in, Represents a local area window; S208: For each Define search threshold ;in, It is a search factor; from Start searching for the left boundary: Similarly, the right boundary is obtained: This forms the initial foreground region: ； The final foreground region is obtained using the following global filtering formula: ;in, This represents the global filtering threshold, and finally based on... Constructing a foreground region mask ,in, ; The feature refinement unit performs the following steps: S209: Move the region mask a along... The dimensions are expanded to obtain the feature sequence of the video frame. A mask matrix with a consistent shape By using the mask matrix Applied to the video frame feature sequence To focus on peak region video features ; where the superscript "fg" indicates foreground features, Indicates element-wise multiplication; S210: Will and The input is fed into three fully connected layers, and the query matrix is ​​calculated for each layer. Key matrix Sum matrix ,in, , , For the learnable matrix of the cross-attention layer, This represents the number of channels for the query feature, key feature, and value feature, taken as... ; S211: Query context features of calculation statements: ; S212: Will Restore to ; S213: By adding the query context features of the statement to the video frame feature sequence through residual connection, the region-aware enhanced video features are obtained: ,in, It is a learnable projection matrix.

4. The method according to claim 3, characterized in that, The gradient-guided noise sampling and calibration module consists of a candidate video segment generation unit, a gradient-guided noise sampling submodule, and a comparison calibration unit; step S3 includes: The candidate video segment generation unit performs the following steps: S301: In Add a learnable category label to it. and with The inputs are combined and interacted with in a Transformer to obtain multimodal features: ;in, The dimension representing multimodal features and These represent the Transformer encoder and decoder, respectively. S302: Will The last sub-feature The input is fed into a fully connected layer for prediction. Gaussian mask parameters: ;in, Includes all video frames and query information. This represents a multilayer perceptron. and Representing the center and width of the video proposal, respectively, each Gaussian mask at the th... The formula for a frame is expressed as: ； S303: Obtain After proposing the nth candidate, the multimodal features are weighted and pooled using a Gaussian mask to obtain the nth... The aggregated feature tensor of candidate segments The superscript "pool" indicates pooling aggregation. S304: Will Features of global statement queries Projected onto the same feature space: ,in, It is a learnable feature projection matrix; S305: Calculate the... Score of each candidate proposal Finally, the candidate segment with the highest score is selected. As the best candidate, the corresponding best candidate video segment features As a positive sample, among which... Indicates the actual duration of the video; The gradient-guided noise sampling submodule performs the following steps: S306: Localization using gradient attribution methods The most sensitive feature dimension for query statements is used to calculate the matching score. Features gradient The subscript "attr" indicates the degree of contribution. This represents max pooling over the time dimension; S307: Take the absolute value of the gradient as the importance weight. The subscript "imp" indicates importance. S308: Set threshold Generate a binary region mask , ;in, The eigenvector represents the first eigenvector. One dimension, Indicates the first The importance weights of each dimension The 1 in the text indicates a region in the corresponding video segment that is semantically strongly related to the query statement. S309: with Initial state The algorithm performs N rounds of iterative updates to minimize its matching degree with the query statement, where the subscript "GI" indicates gradient induction. Each iteration performs the following steps: In the During rounds of iteration: Calculate current features Match score with the query statement and its gradient ;in, To query the score in the nth round of calculation right The gradient of ▽ is used to find the sign of the gradient. Calculate the first moment ; Calculate the second moment ; Calculate the gradient after bias correction ; exist Reverse update features within the constrained region ,in, Indicates the noise injection coefficient. express The binary region mask matrix obtained after broadcasting along the temporal dimension. and This represents the momentum hyperparameter. Indicates the smoothing term. Represents element-wise product; After N iterations, we obtain... , For the final gradient-induced video segment features ; The comparison and alignment unit performs the following steps: S310: Will , and Mapped to the same dimension D, , , ; S311: Calculation using average pooling method and Global features: ， ； S312: Calculate separately , and Match score: ， ； S313: Calculate noise contrast loss ,in This represents the boundary threshold.

5. The method according to claim 4, characterized in that, The video temporal localization model consists of a sentence association temporal awareness module and a gradient-guided noise sampling and calibration module. Step S4 includes: The statement-associated timing awareness module performs the following steps: S41: Set the mask ratio For query statements Perform a random masking operation, replacing the word at the selected position with the special marker [MASK], to obtain the masked query statement. And use a pre-trained word vector model to extract features to obtain the masked statement query feature sequence. The superscript "masked" indicates that the text was masked. S42: The features of the best video candidate segments are... Query feature sequence with the mask statement The input is fed into the multimodal reconstruction decoder, which then performs the following steps through a cross-attention mechanism: Define three learnable linear projection matrices. , and The superscript "Re" indicates a refactoring. It is the dimension of the attention hidden layer; Generate a query vector matrix based on the feature sequence retrieved from the mask statement. Key vector matrix Sum value vector matrix ; Calculate the multimodal fusion features of the query statement and video clips: ； Aggregated context features Features of querying original mask statements Residual connections and layer normalization are performed, and a nonlinear transformation is applied through a feedforward neural network to obtain the final fused feature sequence. ； in, Indicates the length of the sequence queried by the statement; The gradient-guided noise sampling calibration module performs the following steps: S43: For the masked first statement query Extract the corresponding fusion feature vector from each position. The algorithm maps to the vocabulary space through a fully connected layer and calculates the probability distribution of that position in each word of the vocabulary. ;in, and Let these represent the learnable mapping weight matrix and the bias, respectively. This represents the size of the vocabulary used to train the word vector model; S44: Calculate the reconstruction loss of the reconstructed word probability distribution and the original query at the mask position: ;in, This represents the set of indices of the masked words. express The number of elements in the middle, This indicates the index of the masked word in the query statement. Indicates the location The actual words in the original query statement. Indicates the position of the mask. Predicted words not actually exist The probability, Represents the natural logarithm operation; S45: Based on reconstruction loss Loss compared to noise Calculate total loss : ;in, The weighting hyperparameters for noise contrast loss; S46: Based on total loss The stochastic gradient descent algorithm is used to optimize the video temporal localization model.

6. The method according to claim 5, characterized in that, After step S4, the method further includes: S5: For other video feature tensors, sequentially execute steps S1 to S4 to obtain the temporal intervals of all query statements in the corresponding videos until the model converges.

7. The method according to claim 6, characterized in that, After step S5, the method further includes: S6: Input the second video feature tensor and the second statement query of the second video into the converged video temporal localization model to obtain the start time and end time of the video segment in the second video that is related to the second statement query.

8. The method according to claim 7, characterized in that, Step S6 includes: S61: Preprocess the second video feature tensor and the second statement query to obtain the second video frame sequence set and the second statement query features, and input the second video frame sequence set and the second statement query features into the converged video temporal localization model; S62: The second video frame sequence set is refined through the statement association time-series awareness module to obtain second region-aware enhanced video features. These second region-aware enhanced video features are then input into the candidate video segment generation unit to generate... The system uses Gaussian mask parameters and calculates the matching score between the features of each candidate video segment and the global query features. The index of the candidate video segment with the highest matching score is selected as the prediction result. The start and end times of the video segment are determined based on the center and width corresponding to the index of the candidate video segment with the highest score.

9. A weakly supervised video temporal localization device based on sentence association temporal noise sampling calibration, characterized in that, The device includes: One or more processors; Memory, used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors perform the method as described in any one of claims 1-8.

10. A storage medium storing a computer program, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-8.

Images