A video and text cross-modal retrieval method and system based on static and dynamic feature fusion

Abstract

The present invention relates to a video and text cross-modal retrieval method for static and dynamic feature fusion, the method comprising: constructing a set of nouns, prepositions and verbs for a text; converting nouns and prepositions into text static feature representations Tsf and converting verbs into text dynamic feature representations Tdf through a text encoder; encoding a video through a video encoder to obtain a video static feature representation Vsf; calculating and normalizing the inter-frame difference sequence of adjacent frames of a video, and encoding through a video encoder to obtain a video dynamic feature representation Vdf; calculating the static similarity of Tsf and Vsf, calculating the dynamic similarity of Tdf and Vdf, and performing weighted averaging on the two to obtain a final similarity score; for a given plurality of texts and videos, selecting k text-video pairs with the highest similarity scores as retrieval results. The method and system are conducive to improving the accuracy and efficiency of video and text cross-modal retrieval, and can effectively retrieve videos related to text descriptions from a large amount of video data.

Description

Technical Field

[0001] The present invention relates to the technical field of information retrieval, and in particular to a video and text cross-modal retrieval method and system for fusing static and dynamic features. Background Art

[0002] With the rapid development of the Internet, the amount of video and text data generated has shown explosive growth, especially on multimodal content platforms such as social media, video sharing platforms and news websites. User-generated content (UGC) is mainly composed of pictures, videos and text. How to achieve effective retrieval and matching between data of different modalities has always been a major challenge in the field of information processing. In some cross-modal retrieval scenarios, such as finding related videos through text, or finding corresponding text information through video, the technical requirements are more stringent.

[0003] Traditional video and text cross-modal retrieval methods usually rely on a single feature extraction method, which is difficult to fully capture the complex relationship between video and text. In addition, there is a large semantic gap between different modalities, and the significant difference in information expression between video and text further increases the difficulty of cross-modal retrieval. Especially when faced with complex video content and flexible and changeable text expressions, the retrieval effect of existing methods is often not ideal, and there is still much room for improvement in robustness and accuracy. Summary of the invention

[0004] The purpose of the present invention is to provide a video and text cross-modal retrieval method and system with static and dynamic feature fusion, which is conducive to improving the accuracy and efficiency of video and text cross-modal retrieval, and can effectively retrieve videos related to text descriptions from a large amount of video data.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is: a video and text cross-modal retrieval method with static and dynamic feature fusion, comprising the following steps:

[0006] Step S1: for a text T, construct a noun set N, a preposition set P and a verb set Vb;

[0007] Step S2, jointly training the video encoder and the text encoder based on contrastive learning;

[0008] Step S3: convert nouns and prepositions into text static feature representations Tsf, and convert verbs into text dynamic feature representations Tdf through a text encoder;

[0009] Step S4: Encode the video V through a video encoder to obtain a video static feature representation Vsf;

[0010] Step S5, calculating the inter-frame difference sequence D of adjacent frames of the video, and performing normalization processing, and then encoding the normalized inter-frame difference sequence D' through a video encoder to obtain a video dynamic feature representation Vdf;

[0011] Step S6: Calculate the static similarity Sim between the text static feature representation and the video static feature representation static ; Calculate the dynamic similarity Sim between the text dynamic feature representation and the video dynamic feature representation dynamic ; Then, for the static similarity Sim static and dynamic similarity Sim dynamic Perform weighted average to get the final similarity score Sim final ;

[0012] Step S7: For a given plurality of texts and videos, the similarity scores between each text and video are calculated respectively according to the method of steps S1-S6, and then the k text-video pairs with the highest similarity scores are selected as the retrieval results.

[0013] Furthermore, the step S1 specifically includes the following steps:

[0014] Step S11: Input text ,in, w i Indicates the first i words, n is the total number of words in the text;

[0015] Step S12: assign a part-of-speech tag to each word in the text through a part-of-speech tagging function POS(x);

[0016] The part-of-speech tagging function POS(x) w i Mapped to its corresponding part-of-speech tag:

[0017]

[0018] in, l i Words w i The part-of-speech tag;

[0019] Step S13, extract all words whose part-of-speech tags are nouns, prepositions and verbs respectively, and construct a noun set N, a preposition set P and a verb set Vb accordingly;

[0020]

[0021]

[0022]

[0023] Among them, Noun, Preposition, and Verb represent noun, preposition, and verb respectively.

[0024] Furthermore, the step S2 specifically includes the following steps:

[0025] Step S21: Calculate the loss function using the pre-annotated text-video training data set. Defined as:

[0026]

[0027] in, is the similarity calculation function, τ is the temperature coefficient, and n is the index of the negative sample;

[0028] Step S22: Joint training to minimize the loss functions in two directions at the same time. The complete loss function is:

[0029]

[0030] in, is the text-to-video loss, For the loss from video to text;

[0031] For a positive sample (T c ,V c ) or (V c ,T c ) and multiple negative samples (T c ,V n ) or (V c ,T n ), and The specific expressions are as follows:

[0032]

[0033]

[0034] Among them, c is the current sample index, representing a text-video pair or video-text pair, n is the index used to construct negative samples, (T c ,V c ) and (V c ,T c ) is a positive sample pair, and M is the number of texts or videos in the training dataset.

[0035] Furthermore, the step S3 specifically includes the following steps:

[0036] Step S31: Use the pre-trained BERT model to transform each word in the noun set N, preposition set P and verb set Vb w i Mapped to a fixed-dimensional vector space respectively:

[0037]

[0038] in, e ( w i ) is the word encoded by the BERT model w i The word vector representation of , d is the dimension of the word vector;

[0039] Step S32: Encode the words in the noun set N and the preposition set P respectively, and aggregate them to obtain the text static feature representation:

[0040]

[0041] in, Represents the static feature representation of the text, where |N| and |P| are the number of words in the noun set and preposition set respectively;

[0042] Step S33: Encode the words in the verb set Vb and aggregate them to obtain the text dynamic feature representation:

[0043]

[0044] in, Represents the dynamic feature representation of the text, and |Vb| is the number of words in the verb set.

[0045] Furthermore, the step S4 specifically includes the following steps:

[0046] Step S41: Input video sequence ,in F i represents the i-th frame of the video, q represents the total number of frames; each frame of the video is represented as a two-dimensional pixel matrix, or expanded into a vector form:

[0047]

[0048] Among them, H is the height of the frame, W is the width of the frame, and C is the number of channels of the frame;

[0049] Step S42: Use the pre-trained SwinTransformer model as the video encoder, denoted as f Swin , encode each frame of the video and obtain the static feature representation of each frame of the video:

[0050]

[0051] in, S i is the static feature representation of the i-th frame of the video, and d is the dimension of the static feature representation of the video;

[0052] Step S43: Perform global average pooling on the static feature representations of all frames of the video to obtain the static feature representation of the entire video:

[0053]

[0054] in, A static feature representation representing the entire video.

[0055] Furthermore, the step S5 specifically includes the following steps:

[0056] Step S51: Calculate the difference between each pair of adjacent frames in the video to construct an inter-frame difference sequence , where each difference matrix D i Represents the difference between the i-th frame and the i+1-th frame of the video. The calculation formula is as follows:

[0057]

[0058] Wherein, q represents the total number of video frames;

[0059] Step S52: for each difference matrix D i Perform normalization; first find the difference matrix D i The maximum and minimum values ​​in are denoted by D max and D min :

[0060]

[0061]

[0062] Then, the difference matrix D is calculated according to the following formula: i Normalize:

[0063]

[0064] in, represents the normalized difference matrix;

[0065] After normalization, the normalized inter-frame difference sequence is obtained. ;

[0066] Step S53: Use the pre-trained SwinTransformer model as the video encoder, denoted by f Swin, encode each difference matrix in the inter-frame difference sequence D' to obtain the corresponding dynamic feature representation:

[0067]

[0068] Among them, Z i Represents the i-th difference matrix D in the inter-frame difference sequence i The dynamic feature representation of the video, d is the dimension of the dynamic feature representation of the video;

[0069] Step S54: represent all dynamic features of the video Z i Perform global average pooling to obtain the dynamic feature representation Vdf of the entire video:

[0070]

[0071] in, Represents the dynamic feature representation of the entire video.

[0072] Furthermore, the step S6 specifically includes the following steps:

[0073] Step S61: Calculate the Euclidean distance between the text static feature representation Tsf and the video static feature representation Vsf :

[0074]

[0075] Among them, Tsf i and Vsf i Represent the i-th component of the text static feature representation Tsf and the video static feature representation Vsf respectively;

[0076] Step S62: Calculate the static similarity between the text static feature representation Tsf and the video static feature representation Vsf:

[0077]

[0078] Among them, Sim static is the static similarity;

[0079] Step S63: Calculate the Euclidean distance between the text dynamic feature representation Tdf and the video dynamic feature representation Vdf :

[0080]

[0081] Among them, Tdf i and Vdf i Respectively represent the i-th component of the text dynamic feature representation Tdf and the video dynamic feature representation Vdf;

[0082] Step S64: Calculate the similarity between the text dynamic feature representation Tdf and the video dynamic feature representation Vdf:

[0083]

[0084] Among them, Sim dynamic is the dynamic similarity;

[0085] Step S65: Similarity Sim of static features static Similarity with dynamic features Sim dynamic The weighted average is expressed as follows:

[0086]

[0087] Among them, Sim final It represents the final similarity score between text and video, α and β are the set weight parameters, α+β=1.

[0088] Furthermore, the step S7 specifically includes the following steps:

[0089] Step S71: For a given plurality of texts and videos, the similarity score between each text and video is calculated according to the method of steps S1-S6 to obtain a similarity score set. ,in represents the similarity score of the i-th text-video pair, and N is the number of all text-video pairs constructed by all given texts and videos;

[0090] Step S72: Sort the elements in the similarity score set from high to low according to the similarity score to obtain a similarity sorting set. ,in ;

[0091] Step S73: Select the first k elements from the similarity sorted set, find the text-video pairs corresponding to these k elements, and form a search result set, which is expressed as follows:

[0092]

[0093] in, Indicates selecting the first k elements from the set and mapping them to the corresponding text-video pairs to form a retrieval result set .

[0094] The present invention also provides a video and text cross-modal retrieval system that integrates static and dynamic features, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, the above method can be implemented.

[0095] Compared with the prior art, the present invention has the following beneficial effects:

[0096] (1) The present invention distinguishes nouns, prepositions and verbs in the text, and then extracts static text features and dynamic text features respectively. At the same time, it constructs and extracts static features and dynamic features of the video. Compared with the existing methods, the present invention separates the static and dynamic features of the text and video, which greatly improves the accuracy of semantic matching between video and text.

[0097] (2) The present invention proposes a video dynamic feature extraction framework based on inter-frame difference calculation. By calculating the difference between adjacent frames of the video and encoding it, it can more accurately capture the action and change information in the video. It not only reduces the dependence on complex feature extraction algorithms, but also simplifies the dynamic feature extraction process and improves the retrieval efficiency and accuracy.

[0098] (3) The present invention jointly trains the video encoder and the text encoder so that they can be mapped to a unified feature space, and calculates the final similarity score by weighted averaging the similarities of static and dynamic features. Compared with the existing methods, the present invention not only improves the accuracy of cross-modal retrieval, but also enhances the flexibility of the system through weighted strategies, and can better adapt to the needs of different scenarios.

[0099] (4) The present invention can be applied to cross-modal retrieval of text and video, and can effectively retrieve videos related to text descriptions from large-scale video datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] Figure 1 is a flow chart of a method implementation of an embodiment of the present invention;

[0101] Figure 2 Schematic diagram of the training process of the video encoder and the text encoder in the embodiment of the present invention. DETAILED DESCRIPTION

[0102] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0103] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0104] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0105] like Figure 1 As shown, this embodiment provides a video and text cross-modal retrieval method for fusing static and dynamic features, comprising the following steps:

[0106] Step S1: For a text T, construct a noun set N, a preposition set P and a verb set Vb.

[0107] In this embodiment, step S1 specifically includes the following steps:

[0108] Step S11: Input text ,in, w i Indicates the first i words, n is the total number of words in the text.

[0109] Step S12: assign a part-of-speech tag to each word in the text through the part-of-speech tagging function POS(x).

[0110] The part-of-speech tagging function POS(x) w i Mapped to its corresponding part-of-speech tag:

[0111]

[0112] in, l i Words w i Part-of-speech tags, such as nouns, prepositions, verbs, articles, numerals, etc.

[0113] Step S13, extract all words whose part-of-speech tags are nouns, prepositions and verbs respectively, and construct a noun set N, a preposition set P and a verb set Vb accordingly;

[0114]

[0115]

[0116]

[0117] Among them, Noun, Preposition, and Verb represent noun, preposition, and verb respectively.

[0118] Step S2: Jointly train the video encoder and the text encoder based on contrastive learning so that the video encoder can map the text and video into a unified feature space.

[0119] In this embodiment, step S2 specifically includes the following steps:

[0120] Step S21: Calculate the loss function using the pre-annotated text-video training data set. Defined as:

[0121]

[0122] in, is the similarity calculation function, τ is the temperature coefficient, and n is the index of the negative sample.

[0123] Step S22: Joint training to minimize the loss functions in two directions at the same time. The complete loss function is:

[0124]

[0125] in, is the text-to-video loss, is the loss from video to text.

[0126] For a positive sample (T c ,V c ) or (V c ,T c ) and multiple negative samples (T c ,V n ) or (V c ,T n ), and The specific expressions are as follows:

[0127]

[0128]

[0129] Among them, c is the current sample index, representing a text-video pair or video-text pair, n is the index used to construct negative samples, (T c ,V c ) and (V c ,T c ) is a positive sample pair, and M is the number of texts or videos in the training dataset.

[0130] The training process of the video encoder and the text encoder in this embodiment is as follows: Figure 2 shown.

[0131] Step S3: convert nouns and prepositions into text static feature representation Tsf, and convert verbs into text dynamic feature representation Tdf through a text encoder.

[0132] In this embodiment, step S3 specifically includes the following steps:

[0133] Step S31: Use the pre-trained BERT model to transform each word in the noun set N, preposition set P and verb set Vb w i Mapped to a fixed-dimensional vector space respectively:

[0134]

[0135] in, e ( w i ) is the word encoded by the BERT model w i The word vector representation of , d is the dimension of the word vector.

[0136] Step S32: Encode the words in the noun set N and the preposition set P respectively, and aggregate them to obtain the text static feature representation:

[0137]

[0138] in, Represents the static feature representation of the text, where |N| and |P| are the number of words in the noun set and preposition set respectively.

[0139] Step S33: Encode the words in the verb set Vb and aggregate them to obtain the text dynamic feature representation:

[0140]

[0141] in, Represents the dynamic feature representation of the text, and |Vb| is the number of words in the verb set.

[0142] In this embodiment, the specific calculation process of the BERT model is as follows:

[0143] For text sequences , the word embedding of the BERT model is represented as:

[0144]

[0145] Among them, W e is the word embedding matrix, b e is the bias term, E i is word x i word embedding representation.

[0146] Then perform position encoding and add the word embedding and position encoding to get the final input representation:

[0147]

[0148]

[0149] in, is the position embedding of position i, is the representation of the i-th word at layer 0.

[0150] Then enter the multi-layer Transformer encoder. Layer, the word representation is calculated through the self-attention mechanism:

[0151]

[0152] Among them, Q, K, Va are the linear transformation matrices of query, key and value respectively, d k is the dimension of the key vector.

[0153] The output of the self-attention mechanism is then processed by a feedforward neural network:

[0154]

[0155] Among them, FFN is a feed-forward neural network, is the i-th word in Layer representation.

[0156] After BERT is processed by multiple layers of Transformer encoders, the final representation of the text sequence T is the representation of each word in the last layer:

[0157]

[0158] Where L is the number of layers of the Transformer encoder.

[0159] Step S4: Encode the video V through a video encoder to obtain a video static feature representation Vsf.

[0160] In this embodiment, step S4 specifically includes the following steps:

[0161] Step S41: Input video sequence ,in F i represents the i-th frame of the video, q represents the total number of frames; each frame of the video is represented as a two-dimensional pixel matrix, or expanded into a vector form:

[0162]

[0163] Among them, H is the height of the frame, W is the width of the frame, and C is the number of channels of the frame (such as the number of RGB channels is 3).

[0164] Step S42: Use the pre-trained SwinTransformer model as the video encoder, denoted as f Swin , encode each frame of the video and obtain the static feature representation of each frame of the video:

[0165]

[0166] in, S i is the static feature representation of the i-th frame of the video, and d is the dimension of the static feature representation of the video.

[0167] Step S43: Perform global average pooling on the static feature representations of all frames of the video to obtain the static feature representation of the entire video:

[0168]

[0169] in, A static feature representation representing the entire video.

[0170] Step S5, calculating the inter-frame difference sequence D of adjacent frames of the video, and performing normalization processing, and then encoding the normalized inter-frame difference sequence D' through a video encoder to obtain a video dynamic feature representation Vdf.

[0171] In this embodiment, step S5 specifically includes the following steps:

[0172] Step S51: Calculate the difference between each pair of adjacent frames in the video to construct an inter-frame difference sequence , where each difference matrix D i Represents the difference between the i-th frame and the i+1-th frame of the video. The calculation formula is as follows:

[0173]

[0174] in, represents the pixel difference between the i-th pair of adjacent frames, and q represents the total number of video frames.

[0175] Step S52: for each difference matrix D i Perform normalization; first find the difference matrix D i The maximum and minimum values ​​in are denoted by D max and D min :

[0176]

[0177]

[0178] Then, the difference matrix D is calculated according to the following formula: i Normalize:

[0179]

[0180] in, Represents the normalized difference matrix.

[0181] After normalization, the normalized inter-frame difference sequence is obtained. .

[0182] Step S53: Use the pre-trained SwinTransformer model as the video encoder, denoted by f Swin , encode each difference matrix in the inter-frame difference sequence D' to obtain the corresponding dynamic feature representation:

[0183]

[0184] Among them, Z i Represents the i-th difference matrix D in the inter-frame difference sequence i The dynamic feature representation of , d is the dimension of the video dynamic feature representation.

[0185] Step S54: represent all dynamic features of the video Z i Perform global average pooling to obtain the dynamic feature representation Vdf of the entire video:

[0186]

[0187] in, Represents the dynamic feature representation of the entire video.

[0188] Step S6: Calculate the static similarity Sim between the text static feature representation and the video static feature representation static ; Calculate the dynamic similarity Sim between the text dynamic feature representation and the video dynamic feature representation dynamic ; Then, for the static similarity Sim static and dynamic similarity Sim dynamic Perform weighted average to get the final similarity score Sim final .

[0189] In this embodiment, step S6 specifically includes the following steps:

[0190] Step S61: Calculate the Euclidean distance between the text static feature representation Tsf and the video static feature representation Vsf :

[0191]

[0192] The specific expansion form is:

[0193]

[0194] Among them, Tsf i and Vsf i They represent the i-th component of the text static feature representation Tsf and the video static feature representation Vsf respectively.

[0195] Step S62: Calculate the static similarity between the text static feature representation Tsf and the video static feature representation Vsf:

[0196]

[0197] Among them, Sim static is the static similarity. The smaller the difference between the static feature vectors, the greater the static similarity.

[0198] Step S63: Calculate the Euclidean distance between the text dynamic feature representation Tdf and the video dynamic feature representation Vdf :

[0199]

[0200] The specific expansion form is:

[0201]

[0202] Among them, Tdf i and Vdf i They represent the i-th component of the text dynamic feature representation Tdf and the video dynamic feature representation Vdf respectively.

[0203] Step S64: Calculate the similarity between the text dynamic feature representation Tdf and the video dynamic feature representation Vdf:

[0204]

[0205] Among them, Sim dynamic The smaller the difference between the dynamic feature vectors, the greater the dynamic similarity.

[0206] Step S65: Similarity Sim of static features static Similarity with dynamic features Sim dynamic The weighted average is expressed as follows:

[0207]

[0208] Among them, Sim final It represents the final similarity score between text and video, α and β are the set weight parameters, α+β=1.

[0209] Step S7: For a given plurality of texts and videos, the similarity scores between each text and video are calculated respectively according to the method of steps S1-S6, and then the k text-video pairs with the highest similarity scores are selected as the retrieval results.

[0210] In this embodiment, step S7 specifically includes the following steps:

[0211] Step S71: For a given plurality of texts and videos, the similarity score between each text and video is calculated according to the method of steps S1-S6 to obtain a similarity score set. ,in represents the similarity score of the i-th text-video pair, and N is the number of all text-video pairs constructed by all given texts and videos.

[0212] Step S72: Sort the elements in the similarity score set from high to low according to the similarity score to obtain a similarity sorting set. ,in .

[0213] Step S73: Select the first k elements from the similarity sorted set, find the text-video pairs corresponding to these k elements, and form a search result set, which is expressed as follows:

[0214]

[0215] in, Indicates selecting the first k elements from the set and mapping them to the corresponding text-video pairs to form a retrieval result set .

[0216] This embodiment also provides a video and text cross-modal retrieval system that fuses static and dynamic features, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the above method can be implemented.

[0217] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0218] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0219] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0220] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0221] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A video and text cross-modal retrieval method with static and dynamic feature fusion, characterized in that: The following steps are involved: Step S1: for a text T, construct a noun set N, a preposition set P and a verb set Vb; Step S2, jointly training the video encoder and the text encoder based on contrastive learning; Step S3: convert nouns and prepositions into text static feature representations Tsf, and convert verbs into text dynamic feature representations Tdf through a text encoder; Step S4: Encode the video V through a video encoder to obtain a video static feature representation Vsf; Step S5, calculating the inter-frame difference sequence D of adjacent frames of the video, and performing normalization processing, and then encoding the normalized inter-frame difference sequence D' through a video encoder to obtain a video dynamic feature representation Vdf; Step S6: Calculate the static similarity Sim between the text static feature representation and the video static feature representation static ; Calculate the dynamic similarity Sim between the text dynamic feature representation and the video dynamic feature representation dynamic ; Then, for the static similarity Sim static and dynamic similarity Sim dynamic Perform weighted average to get the final similarity score Sim final ; Step S7: For a given plurality of texts and videos, the similarity scores between each text and video are calculated respectively according to the method of steps S1-S6, and then the k text-video pairs with the highest similarity scores are selected as the retrieval results.

2. According to the video and text cross-modal retrieval method of static and dynamic feature fusion according to claim 1, it is characterized in that: The step S1 specifically includes the following steps: Step S11: Input text ,in, w i Indicates the first i words, n is the total number of words in the text; Step S12: assign a part-of-speech tag to each word in the text through a part-of-speech tagging function POS(x); The part-of-speech tagging function POS(x) w i Mapped to its corresponding part-of-speech tag: in, l i Words w i The part-of-speech tag; Step S13, extract all words whose part-of-speech tags are nouns, prepositions and verbs respectively, and construct a noun set N, a preposition set P and a verb set Vb accordingly; Among them, Noun, Preposition, and Verb represent noun, preposition, and verb respectively.

3. The video and text cross-modal retrieval method of static and dynamic feature fusion according to claim 1 is characterized in that: The step S2 specifically includes the following steps: Step S21: Calculate the loss function using the pre-annotated text-video training data set. Defined as: in, is the similarity calculation function, τ is the temperature coefficient, and n is the index of the negative sample; Step S22: Joint training to minimize the loss functions in two directions at the same time. The complete loss function is: in, is the text-to-video loss, For the loss from video to text; For a positive sample (T c ,V c ) or (V c ,T c ) and multiple negative samples (T c ,V n ) or (V c ,T n ), and The specific expressions are as follows: Among them, c is the current sample index, representing a text-video pair or video-text pair, n is the index used to construct negative samples, (T c ,V c ) and (V c ,T c ) is a positive sample pair, and M is the number of texts or videos in the training dataset.

4. The video and text cross-modal retrieval method of static and dynamic feature fusion according to claim 1 is characterized in that: The step S3 specifically comprises the following steps: Step S31: Use the pre-trained BERT model to transform each word in the noun set N, preposition set P and verb set Vb w i Mapped to a fixed-dimensional vector space respectively: in, e ( w i ) is the word encoded by the BERT model w i The word vector representation of , d is the dimension of the word vector; Step S32: Encode the words in the noun set N and the preposition set P respectively, and aggregate them to obtain the text static feature representation: in, Represents the static feature representation of the text, where |N| and |P| are the number of words in the noun set and preposition set respectively; Step S33: Encode the words in the verb set Vb and aggregate them to obtain the text dynamic feature representation: in, Represents the dynamic feature representation of the text, and |Vb| is the number of words in the verb set.

5. The video and text cross-modal retrieval method of static and dynamic feature fusion according to claim 1 is characterized in that: The step S4 specifically comprises the following steps: Step S41: Input video sequence ,in F i represents the i-th frame of the video, q represents the total number of frames; each frame of the video is represented as a two-dimensional pixel matrix, or expanded into a vector form: Among them, H is the height of the frame, W is the width of the frame, and C is the number of channels of the frame; Step S42: Use the pre-trained SwinTransformer model as the video encoder, denoted as f Swin , encode each frame of the video and obtain the static feature representation of each frame of the video: in, S i is the static feature representation of the i-th frame of the video, and d is the dimension of the static feature representation of the video; Step S43: Perform global average pooling on the static feature representations of all frames of the video to obtain the static feature representation of the entire video: in, A static feature representation representing the entire video.

6. The video and text cross-modal retrieval method of static and dynamic feature fusion according to claim 1 is characterized in that: The step S5 specifically comprises the following steps: Step S51: Calculate the difference between each pair of adjacent frames in the video to construct an inter-frame difference sequence , where each difference matrix D i Represents the difference between the i-th frame and the i+1-th frame of the video. The calculation formula is as follows: Wherein, q represents the total number of video frames; Step S52: for each difference matrix D i Perform normalization; first find the difference matrix D i The maximum and minimum values ​​in are denoted by D max and D min : Then, the difference matrix D is calculated according to the following formula: i Normalize: in, represents the normalized difference matrix; After normalization, the normalized inter-frame difference sequence is obtained. ; Step S53: Use the pre-trained SwinTransformer model as the video encoder, denoted by f Swin , encode each difference matrix in the inter-frame difference sequence D' to obtain the corresponding dynamic feature representation: Among them, Z i Represents the i-th difference matrix D in the inter-frame difference sequence i The dynamic feature representation of the video, d is the dimension of the dynamic feature representation of the video; Step S54: Express all dynamic features of the video Z i Perform global average pooling to obtain the dynamic feature representation Vdf of the entire video: in, Represents the dynamic feature representation of the entire video.

7. The video and text cross-modal retrieval method of static and dynamic feature fusion according to claim 1 is characterized in that: The step S6 specifically comprises the following steps: Step S61: Calculate the Euclidean distance between the text static feature representation Tsf and the video static feature representation Vsf : Among them, Tsf i and Vsf i Represent the i-th component of the text static feature representation Tsf and the video static feature representation Vsf respectively; Step S62: Calculate the static similarity between the text static feature representation Tsf and the video static feature representation Vsf: Among them, Sim static is the static similarity; Step S63: Calculate the Euclidean distance between the text dynamic feature representation Tdf and the video dynamic feature representation Vdf : Among them, Tdf i and Vdf i Respectively represent the i-th component of the text dynamic feature representation Tdf and the video dynamic feature representation Vdf; Step S64: Calculate the similarity between the text dynamic feature representation Tdf and the video dynamic feature representation Vdf: Among them, Sim dynamic is the dynamic similarity; Step S65: Similarity Sim of static features static Similarity with dynamic features Sim dynamic The weighted average is expressed as follows: Among them, Sim final It represents the final similarity score between text and video, α and β are the set weight parameters, α+β=1.

8. The video and text cross-modal retrieval method of static and dynamic feature fusion according to claim 1 is characterized in that: The step S7 specifically includes the following steps: Step S71: For a given plurality of texts and videos, the similarity score between each text and video is calculated according to the method of steps S1-S6 to obtain a similarity score set. ,in represents the similarity score of the i-th text-video pair, and N is the number of all text-video pairs constructed by all given texts and videos; Step S72: Sort the elements in the similarity score set from high to low according to the similarity score to obtain a similarity sorting set. ,in ; Step S73: Select the first k elements from the similarity sorted set, find the text-video pairs corresponding to these k elements, and form a search result set, which is expressed as follows: in, Indicates selecting the first k elements from the set and mapping them to the corresponding text-video pairs to form a retrieval result set .

9. A video and text cross-modal retrieval system with static and dynamic feature fusion, characterized in that: The method comprises a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method according to any one of claims 1 to 8 can be implemented.

Images