A gaussian-guided cross-modal learning method and system for video question answering positioning

Abstract

The application discloses a video question and answer positioning method and system based on Gaussian guidance cross-modal learning. The method comprises the following steps: performing feature extraction on a video question and answer public data set to obtain video frame features and question features; inputting the video frame features into a time transformer to obtain global timing features and attention weights; constructing a time weight distribution by using the global timing features and the question features, weighting the video frame features, and obtaining weighted visual features; constructing a time backtracking mechanism by using the attention weights and the time weight distribution to obtain time backtracking features; performing feature fusion on the time backtracking features and the question features to obtain multi-modal fusion features and a predicted answer; introducing a cross-time causal contrast loss for learning to obtain a trained model; and inputting a to-be-processed video and a question into the trained model to output a predicted answer of the question and a video positioning result. The application can improve the semantic relevance of positioning, reduce the operation complexity of the model, and improve the positioning performance of the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, natural language processing, and video question answering, and particularly relates to a video question answering positioning method and system based on Gaussian guided cross-modal learning. BACKGROUND

[0002] In recent years, video question answering has received extensive attention as one of the core tasks in the field of multi-modal research. The goal of video question answering models is to analyze video content and question text to infer and generate correct answers. This task combines the technical advantages of computer vision and natural language processing, and has broad application prospects in human-computer interaction, intelligent transportation, and educational assistance. With the rapid development of deep learning, video question answering models have significantly improved in terms of answer accuracy, but their performance in real-world scenarios still has key deficiencies.

[0003] Most video question answering models use an end-to-end training framework, integrating video features, text features, and answer prediction processes into a single model. Although this approach improves the efficiency of the question answering task, since the model directly outputs answers, its reasoning process is often a "black box", making it difficult to explain the results, which limits its usability in real-world scenarios. Therefore, the video question answering positioning task has emerged, which not only requires the model to generate correct answers, but also must locate the time segment or visual content related to the question in the video.

[0004] Currently, there are few models that can complete video question answering positioning. Existing methods include post-hoc attention analysis and multi-level similarity modeling, but simple text similarity or frame-level visual information cannot solve the time modeling and causal logic in videos, and the complex model architecture designed from scratch also cannot efficiently complete the positioning of the time segment while maintaining the original question answering ability.

[0005] One of the current prior arts is a method of introducing a Gaussian mask module for weakly supervised video localization proposed in the paper "Can I Trust Your Answer? Visually Grounded Video Question Answering". This method uses a temporal Transformer to extract the time sequence features of the video, models a Gaussian distribution based on the output thereof and generates a time weight, so as to weight the video frames, highlight the time segments related to the question, and combine the weighted visual features and question features for multimodal fusion to predict the answer. However, the disadvantage of this method is that the localization effect of this method depends on the output features of the temporal Transformer, and only a single time period weight distribution can be generated, which cannot handle complex causal problems that depend on the reasoning results of multiple time segments, and only visual features are used for localization without combining question features for guidance, resulting in relatively weak semantic relevance.

[0006] The second current prior art is a method based on a gated state space multimodal Transformer proposed in the paper "Encoding and Controlling Global Semantics for Long-form Video Question Answering". This method introduces a state space layer in the multimodal Transformer, uses a series of state matrices to perform convolution operations on the video frame features to generate a global semantic representation of the time sequence, and dynamically adjusts the influence of the global semantics on the visual representation through a gating mechanism to alleviate the information loss problem caused by frame and local feature selection. In addition, this method proposes a cross-modal combination consistency C3 target to optimize the alignment of visual and language features. However, the disadvantage of this method is that the addition of the state space layer and the gating mechanism increases the complexity of the model, has a higher demand for computing resources, and limits the efficiency in practical applications, and the state space layer enhances the global semantics, which is not equivalent to the improvement of causal reasoning ability, but only enhances the feature expression of each frame without optimizing the temporal dependency relationship. SUMMARY

[0007] The purpose of the present application is to overcome the shortcomings of the existing methods, and a Gaussian guided cross-modal learning video question answering localization method and system are proposed. The main problem solved by the present application is how to improve the semantic relevance of time segment localization, reduce the computational complexity of the model, and improve the localization performance and practical application efficiency of the model.

[0008] In order to solve the above problems, the present application proposes a Gaussian guided cross-modal learning video question answering localization method, which comprises:

[0009] Feature extraction is performed on video data, question text data and answer text data in a video question answering public data set to obtain video frame features f(v) and question features f(q);

[0010] The video frame features f(v) are input into a time Transformer to obtain global timing features and attention weights A ij ;

[0011] The global timing features and the question features f(q) are used to obtain a center point μ and a range σ of a dynamic Gaussian distribution, and a normalized time weight distribution P(t) is constructed to weight the video frame features f(v) to obtain weighted visual features

[0012] The attention weights A ij and the time weight distribution P(t) are used to construct a time backtracking mechanism to obtain time backtracking features

[0013] The time backtracking features and the question features f(q) are fused to obtain multi-modal fusion features f cls and predicted answers

[0014] The video frame features f(v), the time weight distribution P(t) and the time backtracking features A cross-time causal contrast loss is introduced for learning to obtain a trained model;

[0015] A user inputs a to-be-processed video and a question into the trained model, and outputs a predicted answer of the question and a video positioning result corresponding to the answer.

[0016] Preferably, the feature extraction specifically comprises:

[0017] The video data is first sampled according to a fixed frame rate, and the total number of final sampling frames is denoted as a fixed number T. Then, the video data is uniformly sampled in a fixed number to obtain T video frames. The size of the video frames is adjusted to a uniform size, and the pixel values of the video frames are normalized to obtain preprocessed sampling video data.

[0018] The preprocessed sampling video data is input into a pre-trained CLIP model visual encoder for feature extraction to obtain video frame features f(v), and the question text data is input into a pre-trained CLIP model text encoder for encoding to obtain question features f(q).

[0019] Preferably, the video frame feature f(v) is input into a time Transformer to obtain a global timing feature and an attention matrix A ij , specifically:

[0020] After inputting the video frame feature f(v) and embedding the position information, the time-dependent relationship between frames is modeled using an improved self-attention mechanism, followed by residual connection and normalization, and then passing through a feedforward neural network to obtain a global timing feature and an attention weight A ij ;

[0021] The video frame feature f(v) is composed of a plurality of time frame features The time frame feature includes a query vector Q t , a key vector K t , and a value vector V t ;

[0022] The calculation formulas of the query vector Q t and the key vector K t are as follows:

[0023]

[0024] where Cov1D represents a one-dimensional convolution operation, and the convolution kernel size k represents information covering adjacent k-1 time frames centered on frame t;

[0025] The video frame feature f(v) is mapped to the vector space of the query vector Q t and the key vector K t using convolution to obtain the value vector V t ;

[0026] The attention weight A ij represents the local dependence degree of frame i on frame j, and its calculation formula is:

[0027]

[0028] where d is a scaling factor of the feature dimension, used to stabilize the attention value, Q i is the query vector of frame i, is the transpose of the key vector of frame j.

[0029] Preferably, the global timing feature and the question feature f(q) are used to obtain the center point μ and the range σ of the dynamic Gaussian distribution, and a normalized time weight distribution P(t) is constructed to weight the video frame feature f(v) to obtain a weighted visual feature Specifically,

[0030] The calculation formulae of the center point μ and the range σ of the dynamic Gaussian distribution are respectively:

[0031]

[0032] wherein, and are weight matrices of the global timing feature , and are weight matrices of the question feature f(q), and are randomly initialized in the model initialization stage and are updated in the gradient descent process, the question feature f(q) is a correction term, and w μ and w σ control the correction strength;

[0033] The center point μ and the range σ of the dynamic Gaussian distribution are used to construct the time weight distribution p(t), and the specific calculation formula is:

[0034]

[0035] wherein T is the total number of sampling frames;

[0036] The time weight distribution p(t) is normalized to obtain the normalized time weight distribution P(t), and the specific calculation formula is:

[0037]

[0038] The video frame feature f(v) is weighted by the normalized time weight distribution P(t) to obtain a weighted visual feature

[0039] Preferably, the time backtracking mechanism is constructed by using the attention weight A ij and the time weight distribution P(t) to obtain a time backtracking feature Specifically,

[0040] The attention weight A ij and the value vector V t are dot multiplied to generate a causal feature

[0041] The causal feature is weighted by the time weight distribution P(t) to obtain a time backtracking feature

[0042] Preferably, the time backtracking feature and the question feature f(q) are fused to obtain a multi-modal fusion feature f cls and the predicted answer Specifically,

[0043] The time backtracking feature and the question feature f(q) are spliced to obtain a fusion feature f cls through a full connection layer cls Input a classification layer to generate a probability distribution Softmax(W cls f cls +b cls of the answer through an activation function, wherein W cls and b cls are parameters of the classification layer, and the candidate answer with the maximum probability is selected as the predicted answer of the video question answering task

[0044] Preferably, the video frame feature f(v), the time weight distribution P(t) and the time backtracking feature are input, and a cross-time causal contrast loss is introduced for learning to obtain a trained model, specifically:

[0045] Positive and negative samples for contrast learning are constructed, that is, the time frame feature in the video frame feature f(v) is weighted by dot product using the time weight distribution P(t) to obtain a positive sample and a negative sample is randomly selected from a frame with a lower time weight distribution P(t)

[0046] The positive sample and the negative sample are used to construct a cross-time causal contrast loss function The specific formula is as follows:

[0047]

[0048] Wherein, sim represents the cosine similarity between feature vectors, t is the index of the positive sample, and t' is the index of the negative sample.

[0049] Correspondingly, the application also provides a video question answering positioning system based on Gaussian guided cross-modal learning, comprising:

[0050] An initialization unit is configured to extract features from video data, question text data and answer text data in a video question answering public data set to obtain a video frame feature f(v) and a question feature f(q);

[0051] A model construction and training unit is configured to input the video frame feature f(v) into a time Transformer to obtain a global time sequence feature and an attention weight A ij ; utilize the global time sequence feature and the question feature f(q) to obtain a center point μ and a range σ of a dynamic Gaussian distribution, and construct a normalized time weight distribution P(t) to weight the video frame feature f(v) to obtain a weighted visual feature utilize the attention weight A ij and the time weight distribution P(t) to construct a time backtracking mechanism to obtain a time backtracking feature fuse the time backtracking feature and the question feature f(q) to obtain a multi-modal fusion feature f cls and a predicted answer input the video frame feature f(v), the time weight distribution P(t) and the time backtracking feature introduce a cross-time causal contrast loss for learning to obtain a trained model;

[0052] A model application unit is configured to input a to-be-processed video and a question into the trained model to output a predicted answer of the question and a video positioning result corresponding to the answer.

[0053] The present application has the following beneficial effects:

[0054] The dynamic Gaussian distribution time positioning method, the time backtracking mechanism and the cross-time causal contrast loss provided by the present application have high modularity and plug-and-play characteristics, and can be easily integrated into existing video question and answer models or multi-modal tasks to provide time positioning enhancement and causal reasoning capability improvement for these models. In addition, the dynamic Gaussian distribution, the time backtracking mechanism and the causal contrast loss in the present application only rely on simple linear mapping and weighting operation of time features during calculation, and the generation and normalization process of the Gaussian distribution has low overhead, which can adapt to the computing power of most devices. Compared with end-to-end models that require complex parameter learning, the module calculation complexity of the present application is significantly reduced, which is suitable for real-time application scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 is a flowchart of a Gaussian-guided cross-modal learning video question and answer positioning method according to an embodiment of the present application;

[0056] Figure 2 is a schematic diagram of a time Transformer architecture according to an embodiment of the present application;

[0057] Figure 3is a structural diagram of a video question and answer positioning system of a Gaussian guidance cross-modal learning according to an embodiment of the present application. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0059] Figure 1 is a flowchart of a video question and answer positioning method of a Gaussian guidance cross-modal learning according to an embodiment of the present application, as shown in the figure, the method comprises: Figure 1

[0060] S1, feature extraction is performed on video data, question text data and answer text data in a video question and answer public data set, to obtain video frame features f(v) and question features f(q);

[0061] S2, the video frame features f(v) are input into a time Transformer, to obtain global timing features and attention weights A ij ;

[0062] S3, the global timing features and the question features f(q) are used to obtain center points μ and ranges σ of dynamic Gaussian distribution, and a normalized time weight distribution P(t) is constructed to weight the video frame features f(v), to obtain weighted visual features

[0063] S4, a time backtracking mechanism is constructed by using the attention weights A ij and the time weight distribution P(t), to obtain time backtracking features

[0064] S5, the time backtracking features and the question features f(q) are fused to obtain multi-modal fusion features f cls and predicted answers

[0065] S6, the video frame features f(v), the time weight distribution P(v) and the time backtracking features are input, cross-time causal contrast loss is introduced for learning, to obtain a trained model;

[0066] ​S7, the user inputs the video to be processed and the question into the trained model, and outputs the predicted answer to the question and the video positioning result corresponding to the answer.

[0067] Step S1 is specifically as follows:

[0068] The video question answering public data set is universal, such as NExT-GQA, MultiHop-EgoQA, AGQA-Decomp, etc.

[0069] S1-1, the video data is first sampled at a fixed frame rate, in this embodiment, the frame rate is 6fs, so that the frame rates of different input videos are the same, and the total number of frames after sampling is fixed as T, then uniform sampling is performed with a fixed number to obtain T frames of video frames, the size of the video frames is adjusted to a uniform size, in this embodiment, the size is 224x224, and the pixel values of the video frames are normalized to obtain preprocessed sampled video data;

[0070] S1-2, the preprocessed sampled video data is input into a pre-trained CLIP model visual encoder for feature extraction, the visual encoder of the CLIP model adopts a Vision Transformer (ViT) architecture, each frame of image is divided into fixed size image patches, and embedding representation is generated through linear mapping, combined with position embedding, feature modeling is performed through multiple layers of Transformer to obtain video frame feature f(v), the question text data is input into a pre-trained CLIP model text encoder for encoding, the text encoder of the CLIP model adopts a Transformer architecture, the question q is divided into subwords through a word segmenter to generate embedding representation, combined with position encoding, the relationship between words is modeled through multiple layers of Transformer to obtain question feature f(q), which represents the semantic information of the question and can be aligned with the visual feature in the shared embedding space.

[0071] Step S2, the architecture of the time Transformer is as shown in Figure 2 , and the steps are specifically as follows:

[0072] After inputting the video frame feature f(v) and embedding position information, the improved self-attention mechanism is used to model the time dependence between frames, then residual connection and normalization are performed, and the global time sequence feature and attention weight A ij are obtained through a feedforward neural network.

[0073] Among them, the video frame feature f(v) is composed of a plurality of time frame features , and the time frame feature includes a query vector Q t, key vector K t , value vector V t , query vector Q t , key vector K t , value vector V t Not only from itself, but also through the convolution operation of the integration of its local context features;

[0074] The query vector Q t and key vector K t The formula is respectively:

[0075]

[0076] Wherein, Cov1D represents one-dimensional convolution operation, and the convolution kernel size k represents the information covering the adjacent k-1 time frames centered on frame t;

[0077] The video frame feature f(v) is mapped to the vector space of the query vector Q t and key vector K t , and the value vector V t is obtained;

[0078] The attention weight A ij represents the local dependence degree of frame i to frame j, and the formula is:

[0079]

[0080] Wherein, d is the scaling factor of feature dimension, used to stabilize the attention value, Q i is the query vector of frame i, is the transpose of the key vector of frame j.

[0081] Step S3, as follows:

[0082] S3-1, the formula of the center point μ and the range σ of dynamic Gaussian distribution is respectively:

[0083]

[0084] Wherein, according to the definition of Gaussian distribution, μ represents the video time frame most related to the question, and σ determines the expansion width of the distribution, which represents the coverage range of the related frames; And Respectively, the weight matrix of the global time sequence feature , and Respectively, the weight matrix of the question feature f(q) , and Respectively, the weight matrix of the question feature f(q) The problem feature f(q) is a correction term for improving semantic correlation while ensuring the dominant role of video features, w μ and w σ Control the correction strength;

[0085] S3-2, the center point μ and the range σ of the dynamic Gaussian distribution are used to construct the time weight distribution p(t), and the specific calculation formula is:

[0086]

[0087] Where T is the total number of frames sampled;

[0088] The closer the time frame to the center point μ, the greater the weight and the higher the correlation;

[0089] S3-3, normalize the time weight distribution p(t) for subsequent weighted operation, and obtain the normalized time weight distribution P(t), and the specific calculation formula is:

[0090]

[0091] S3-4, the normalized time weight distribution P(t) is used to weight the video frame feature f(v), and the weighted visual feature

[0092] Step S4, specifically as follows:

[0093] Through the attention weight A ij Model the explicit dependency relationship between time frames, and combine the time weight distribution P(t) located by the Gaussian distribution, so that the model can capture the causal chain before and after the event in the video, and solve the complex problem that requires causal reasoning for video question and answer positioning;

[0094] S4-1, the attention weight A ij and the value vector V t are dot multiplied to generate causal features

[0095] S4-2, the time weight distribution P(t) is used to weight the causal features Integrate the semantics of the key time frame and its causal dependency relationship to obtain the time backtracking feature

[0096] Step S5, specifically as follows:

[0097] The time backtracking feature and the problem feature f(q) are spliced, and the fusion feature f is obtained through a fully connected layer clsThe fusion feature f cls The input classification layer generates a probability distribution Softmax(W cls f cls +b cls ) of the answer through an activation function, and the candidate answer with the maximum probability is selected as the predicted answer of the video question answering task cls and b cls are parameters of the classification layer

[0098] Step S6, specifically as follows:

[0099] S6-1, constructing positive and negative samples for contrast learning, that is, using the time weight distribution P(t) to perform dot product weighting on the time frame feature f(v) in the video frame feature f(v) to obtain a positive sample representing a time segment highly related to the question semantics, and randomly selecting a negative sample from a frame with a low time weight distribution P(t) representing a time segment unrelated to the question, and the random sampling strategy of the negative sample ensures the diversity of the contrast learning and strengthens the ability of the model to distinguish time segments;

[0100] S6-2, constructing a cross-time causal contrast loss function using the positive sample and the negative sample The specific formula is as follows:

[0101]

[0102] Where sim represents the cosine similarity between feature vectors, t is the positive sample index, and t ′ is the negative sample index.

[0103] Step S7, specifically as follows:

[0104] The predicted answer adopts the accuracy rate index, the frame index of the video positioning result is a range [mu-lambda*sigma, mu+lambda*sigma] calculated using the center point mu and the range sigma of the dynamic Gaussian distribution, and the video positioning result adopts the IoU index.

[0105] Correspondingly, the application also provides a Gaussian-guided cross-modal learning video question answering positioning system, as shown in Figure 3 , comprising:

[0106] An initialization unit 1 is configured to extract features from video data, question text data and answer text data in a video question answering public data set to obtain video frame features f(v) and question features f(q);

[0107] ​Specifically, the video data is first sampled at a fixed frame rate, and the total number of final samples is fixed as T. Then, the video data is uniformly sampled at a fixed number to obtain T video frames. The size of the video frames is adjusted to a uniform size, and the pixel values of the video frames are normalized to obtain preprocessed sampled video data.

[0108] The preprocessed sampled video data is input into a pre-trained CLIP model visual encoder for feature extraction to obtain video frame features f(v). The question text data is input into a pre-trained CLIP model text encoder for encoding to obtain question features f(q).

[0109] A model construction and training unit 2 is configured to input the video frame features f(v) into a time Transformer to obtain global temporal features and attention weights A ij . The global temporal features and the question features f(q) are used to obtain the center point μ and the range σ of a dynamic Gaussian distribution, and a normalized time weight distribution P(t) is constructed to weight the video frame features f(v) to obtain weighted visual features The attention weights A ij and the time weight distribution P(t) are used to construct a time backtracking mechanism to obtain time backtracking features The time backtracking features and the question features f(q) are fused to obtain multi-modal fusion features f cls and a predicted answer The video frame features f(v), the time weight distribution P(t), and the time backtracking features are input into a cross-time causal contrastive loss to learn and obtain a trained model.

[0110] Specifically, after embedding the position information, the video frame features f(v) are input into an improved self-attention mechanism to model the time dependency between frames. Then, residual connection and normalization are performed, and a feedforward neural network is used to obtain global temporal features and attention weights A ij .

[0111] The video frame features f(v) are composed of a plurality of time frame features Each time frame feature includes a query vector Q t , a key vector K t , and a value vector V t .

[0112] The query vector Q t and the key vector K tThe calculation formulas of the center point μ and the range σ of the dynamic Gaussian distribution are respectively:

[0113]

[0114] wherein Cov1D represents a one-dimensional convolution operation, and k represents a convolution kernel size covering information of adjacent k-1 time frames centered on frame t;

[0115] The video frame feature f(v) is mapped to the query vector Q t and the key vector K t using convolution to obtain the value vector V t ;

[0116] The attention weight A ij represents the local dependence degree of frame i to frame j, and the calculation formula is:

[0117]

[0118] wherein d is a scaling factor of the feature dimension, used for stabilizing the attention value, Q i is the query vector of frame i, is the transpose of the key vector of frame j.

[0119] The calculation formulas of the center point μ and the range σ of the dynamic Gaussian distribution are respectively:

[0120]

[0121] wherein and are weight matrices of the global time sequence feature and are weight matrices of the question feature f(q), and are the question feature f(q), and are randomly initialized in the model initialization stage and are updated in the gradient descent process, and the question feature f(q) is a correction term, w μ and w σ control the correction strength;

[0122] The center point μ and the range σ of the dynamic Gaussian distribution are used to construct the time weight distribution p(t), and the specific calculation formula is:

[0123]

[0124] wherein T is the total number of sampling frames;

[0125] The time weight distribution p(t) is normalized to obtain the normalized time weight distribution P(t), and the specific calculation formula is:

[0126]

[0127] The video frame features f(v) are weighted using the normalized time weight distribution P(t) to obtain the weighted visual features.

[0128] The attention weight A ij and the value vector V t Perform a dot product to generate causal features.

[0129] Using the time weight distribution P(v) to analyze the causal features By weighting, the time backtracking features are obtained.

[0130] The time backtracking feature The feature f(q) is concatenated with the problem feature f(q) and then passed through a fully connected layer to obtain the fused feature f. cls , will fuse features f cls Input to the classification layer, and generate the probability distribution of the answer using the activation function Softmax(W). cls f cls +b cls ), where W cls and b cls These are the parameters of the classification layer; the candidate answer with the highest probability is selected as the predicted answer for the video question answering task.

[0131] Construct positive and negative samples for contrastive learning, that is, use the temporal weight distribution P(t) to analyze the temporal frame features f(v) in the video frame features. Perform dot product weighting to obtain positive samples Negative samples are randomly selected from frames with lower time weight distribution P(t).

[0132] Using the positive samples and the negative sample Constructing a cross-time causal contrastive loss function The specific formula is as follows:

[0133]

[0134] Where sim represents the cosine similarity between feature vectors, t is the positive sample index, and t′ is the negative sample index.

[0135] Model application unit 3 is used by the user to input the video to be processed and the question into the trained model, and outputs the predicted answer to the question and the corresponding video location result.

[0136] Therefore, the application proposes a video question and answer positioning method of Gaussian guidance cross-modal learning. The dynamic Gaussian distribution time positioning method, time backtracking mechanism and cross-time causal contrast loss proposed by the application have high modularity and plug-and-play characteristics, and can be easily integrated into existing video question and answer models or multi-modal tasks, providing time positioning enhancement and causal reasoning capability improvement for these models. In addition, the dynamic Gaussian distribution, time backtracking mechanism and causal contrast loss in the application only rely on simple linear mapping and weighting operation of time features during calculation, and the generation and normalization process of Gaussian distribution has low overhead, which can adapt to the computing power of most devices. Compared with end-to-end models that require complex parameter learning, the module calculation complexity of the method is significantly reduced, which is suitable for real-time application scenarios.

[0137] The above describes in detail the video question and answer positioning method and system provided by the embodiment of the application. The principle and implementation mode of the application are described by applying specific examples. The above embodiment description is only used to help understand the method of the application and its core idea. Meanwhile, for those skilled in the art, according to the idea of the application, the specific implementation mode and application range will be changed. In summary, the content of the specification should not be understood as a limitation of the application.

Claims

1. A video question and answer positioning method of Gaussian guidance cross-modal learning, characterized in that, The method comprises: Feature extraction is performed on the video data, question text data and answer text data in the video question answering public data set to obtain video frame features f(v) and question features f(q); the video frame features f(v) are composed of a plurality of time frame features The time frame features include a query vector Q t , a key vector K t , and a value vector V t ; input the video frame feature f(v) into a time Transformer to obtain a global timing feature and an attention weight A ij ; the attention weight A ij represents the degree of local dependence of frame i on frame j; Utilizing the global timing feature and the question feature f(q), the center point μ and the range σ of the dynamic Gaussian distribution are obtained, and the calculation formula is as follows: wherein, and are weight matrices of the global timing feature , and are weight matrices of the question feature f(q), and are randomly initialized in the model initialization stage and are updated in the gradient descent process, the question feature f(q) is a correction term, and w μ and w σ control the correction strength; The time weight distribution p(t) is constructed by using the center point mu and the range sigma of the dynamic Gaussian distribution, and the specific calculation formula is: Wherein, T is the total number of frames of sampling; constructing a normalized temporal weight distribution P(t) to weight the video frame features f(v) to obtain weighted visual features constructing a time backtracking mechanism using the attention weight A ij and the time weight distribution P(t) to obtain a time backtracking feature performing dot product on the attention weight A ij and the value vector V t to generate a causal feature weighting the causal feature using the time weight distribution P(t) to obtain a time backtracking feature ​ performing feature fusion on the time backtracking feature and the question feature f(q) to obtain a multi-modal fusion feature f cls and predicting the answer inputting the video frame feature f(v), the time weight distribution P(t) and the time backtracking feature Introducing cross-time causal contrast loss for learning to obtain a trained model, which includes constructing positive and negative samples for contrast learning: using the time weight distribution P(t) to the time frame feature f(v) in the video frame feature Performing dot product weighting to obtain positive samples And randomly selecting negative samples from the frames with lower time weight distribution P(t) The user inputs the video to be processed and the question into the trained model, and outputs the predicted answer of the question and the video positioning result corresponding to the answer.

2. The Gaussian-guided cross-modal learning method for video question answering positioning according to claim 1, wherein, The feature extraction is specifically: First, the video data is sampled according to a fixed frame rate, and the total number of final sampling frames is denoted as a fixed number T. Then, the video frames are uniformly sampled in a fixed number to obtain T video frames. The size of the video frames is adjusted to a uniform size, and the pixel values of the video frames are normalized to obtain the preprocessed sampling video data. The preprocessed sampling video data is input into the pre-trained CLIP model visual encoder for feature extraction to obtain the video frame feature f(v). The question text data is input into the pre-trained CLIP model text encoder for encoding to obtain the question feature f(q).

3. The video question-answering localization method based on Gaussian-guided cross-modal learning as described in claim 1, characterized in that... The video frame feature f(v) is input into the time Transformer to obtain a global timing feature and an attention matrix A ij , specifically: The input video frame feature f(v) is embedded with position information, and a modified self-attention mechanism is used to model the time dependence between frames, followed by residual connection and normalization, and then through a feedforward neural network to obtain the global timing feature and attention weight A ij ; The query vector Q t and the key vector K t The calculation formula is respectively: Wherein, Cov1D represents one-dimensional convolution operation, and the convolution kernel size k represents information covering the adjacent k-1 time frames centered on frame t. mapping the video frame features f(v) to a query vector Q using a convolution t and a key vector K t to a vector space, resulting in the value vector V t ; The attention weight A ij represents the degree of local dependence of frame i on frame j, and its calculation formula is: where d is a scaling factor of the feature dimension, used to stabilize attention values, Q i is the query vector for frame i, is the transpose of the key vector for frame j.

4. The Gaussian-guided cross-modal learning method for video question answering positioning according to claim 1, wherein, The normalized temporal weight distribution P(t) is constructed, and the video frame features f(v) are weighted to obtain weighted visual features Specifically: The time weight distribution p(t) is normalized to obtain the normalized time weight distribution P(t), and the specific calculation formula is: weighting the video frame features f(v) with the normalized temporal weight distribution P(t) to obtain weighted visual features 5. The Gaussian-guided cross-modal learning method for video question answering positioning of claim 1, wherein, The time backtracking feature The multi-modal fusion feature f is obtained by feature fusion of the question feature f(q) and the time backtracking feature cls And the predicted answer Specifically: The time backtracking feature The feature f(q) is concatenated with the problem feature f(q) and then passed through a fully connected layer to obtain the fused feature f. cls , will fuse features f cls Input to the classification layer, and generate the probability distribution of the answer using the activation function Softmax(W). cls f cls +b cls ), where W cls and b cls These are the parameters of the classification layer; the candidate answer with the highest probability is selected as the predicted answer for the video question answering task.

6. The Gaussian-guided cross-modal learning method for video question answering positioning of claim 1, wherein, The cross-time causal contrast loss is introduced for learning, and the specific process is as follows: Utilizing the positive samples and the negative samples Constructing a cross-time causal contrast loss function The specific formula is as follows: Wherein, sim represents the cosine similarity between feature vectors, t is the positive sample index, and t' is the negative sample index.

7. A Gaussian-guided cross-modal learning video question answering localization system, characterized in that, The system comprises: An initialization unit is configured to perform feature extraction on video data, question text data and answer text data in a video question and answer public data set to obtain video frame features f(v) and question features f(q); the video frame features f(v) are composed of a plurality of time frame features The time frame features include a query vector Q t , a key vector K t and a value vector V t ; A model construction and training unit configured to input the video frame feature f(v) into a time transformer to obtain a global timing feature and an attention weight A ij ; the attention weight A ij represents the degree of local dependence of frame i on frame j; using the global timing feature and the question feature f(q), the center point μ and the range σ of the dynamic Gaussian distribution are obtained, and the calculation formulas are as follows: wherein, and are weight matrices of the global timing feature , and are weight matrices of the question feature f(q), and are initialized randomly in the model initialization stage and are updated in the gradient descent process, the question feature f(q) is a correction term, w μ and w σ control the correction strength; The time weight distribution p(t) is constructed by using the center point mu and the range sigma of the dynamic Gaussian distribution, and the specific calculation formula is: Wherein, T is the total number of frames of sampling; constructing a normalized temporal weight distribution P(t) to weight the video frame features f(v) to obtain weighted visual features using the attention weight A ij and the temporal weight distribution P(t) to construct a time backtracking mechanism to obtain time backtracking features performing dot product on the attention weight A ij and the value vector V t to generate causal features using the temporal weight distribution P(t) to weight the causal features to obtain time backtracking features performing feature fusion on the time backtracking features and the question features f(q) to obtain multi-modal fusion features f cls and predicted answers inputting the video frame features f(v), the temporal weight distribution P(t) and the time backtracking features introducing a cross-time causal contrastive loss for learning to obtain a trained model, wherein constructing positive and negative samples for contrastive learning includes using the temporal weight distribution P(t) to perform dot product weighting on the time frame features in the video frame features f(v) to obtain positive samples and randomly selecting negative samples from frames with lower weights in the temporal weight distribution P(t) and randomly selecting negative samples from frames with lower weights in the temporal weight distribution P(t) a model application unit for a user to input a to-be-processed video and a question into the trained model to output a predicted answer to the question and a video positioning result corresponding to the answer.

8. The Gaussian-guided cross-modal learning system for video question answering localization of claim 7, wherein, The initialization unit is specifically: First, the video data is sampled according to a fixed frame rate, and the total number of final sampling frames is denoted as a fixed number T. Then, the video frames are uniformly sampled in a fixed number to obtain T video frames. The size of the video frames is adjusted to a uniform size, and the pixel values of the video frames are normalized to obtain the preprocessed sampling video data. The preprocessed sampling video data is input into the pre-trained CLIP model visual encoder for feature extraction to obtain the video frame feature f(v). The question text data is input into the pre-trained CLIP model text encoder for encoding to obtain the question feature f(q).

9. The Gaussian-guided cross-modal learning system for video question answering localization of claim 7, wherein, The model construction and training unit is specifically: The input video frame feature f(v) is embedded with position information, and a modified self-attention mechanism is used to model the time dependence between frames, followed by residual connection and normalization, and then through a feedforward neural network to obtain the global timing feature and attention weight A ij ; The query vector Q t and the key vector K t The calculation formula is respectively: Wherein, Cov1D represents one-dimensional convolution operation, and the convolution kernel size k represents information covering the adjacent k-1 time frames centered on frame t. mapping the video frame features f(v) to a query vector Q using a convolution t and a key vector K t to a vector space, resulting in the value vector V t ; The attention weight A ij represents the degree of local dependence of frame i on frame j, and its calculation formula is: where d is a scaling factor of the feature dimension, used to stabilize attention values, Q i is the query vector of frame i, is the transpose of the key vector of frame j; The time weight distribution p(t) is normalized to obtain the normalized time weight distribution P(t), and the specific calculation formula is: weighting the video frame features f(v) with the normalized temporal weight distribution P(t) to obtain weighted visual features The time backtracking feature The feature f(q) is concatenated with the problem feature f(q) and then passed through a fully connected layer to obtain the fused feature f. cls , will fuse features f cls Input to the classification layer, and generate the probability distribution of the answer using the activation function Softmax(W). cls f cls +b cls ), where W cls and b cls These are the parameters of the classification layer; the candidate answer with the highest probability is selected as the predicted answer for the video question answering task. Utilizing the positive samples and the negative samples Constructing a cross-time causal contrast loss function The specific formula is as follows: Wherein, sim represents the cosine similarity between feature vectors, t is the positive sample index, and t' is the negative sample index.

Images