Cross-modal video time sequence content positioning method and system based on situation memory prompt

Abstract

The invention discloses a cross-modal video time sequence content positioning method and system based on a situation memory prompt, and relates to the technical field of cross-modal understanding, a group of situation memories are dynamically generated through a first stage, and various behavior instances occurring in an input video are explicitly summarized; an unsupervised memory learning normal form is formulated by applying discriminative and diversity constraints, and dependence on extra behavior instance annotation is eliminated. Secondly, in the second stage, enhanced memory prompts are obtained based on frame-level detail content supplementation and language query interaction and further serve as anchor frames, and accurate and efficient regression of target moment fragment boundaries is achieved. Compared with a traditional strategy based on a sliding window or a multi-scale anchor frame, the cross-modal video time sequence content positioning method provided by the invention better conforms to cognitive habits, and has performance and efficiency advantages when processing a long video.

Description

Technical Field

[0001] The present invention relates to the field of cross-modal understanding technology, and in particular to a method and system for locating temporal content of a cross-modal video based on contextual memory prompts. Background Art

[0002] As one of the research hotspots in the field of visual-language understanding, cross-modal video temporal content localization technology aims to detect segments in the video that are semantically related to a given natural language query; and can further analyze and understand the localization results to serve downstream tasks such as video retrieval, video description generation, and video question answering. Its research results have good application value in security monitoring, autonomous driving, multimedia analysis, and human-computer interaction. Although cross-modal video temporal content localization technology has broad application prospects and research value, and researchers have made many research progress in recent years, most existing work focuses on videos with shorter time spans (e.g., minutes). With the rapid growth in the number and length of online videos, research on cross-modal temporal content localization for long-span videos (e.g., hours) has become increasingly important. However, existing analysis methods for short videos usually use downsampling strategies to obtain fixed-length sequences for processing, but this will inevitably aggravate information loss when processing long videos. Summary of the invention

[0003] In order to solve the deficiencies mentioned in the above background technology, the purpose of the present invention is to provide a method and system for localizing temporal content of cross-modal videos based on contextual memory prompts.

[0004] In a first aspect, the purpose of the present invention can be achieved by the following technical solution: a method for localizing temporal content of a cross-modal video based on contextual memory prompts, the method comprising the following steps:

[0005] Receive an input video sequence, input the input video sequence into a pre-trained visual backbone network model, output an initial video representation, input the initial video representation into a pre-established Transformer module, and output a dynamic contextual memory dataset;

[0006] Based on the memory items in the dynamic situational memory data set, the predicted behavior category score vector and the temporal center coordinates and span corresponding to each behavior instance are calculated, and the dynamic situational memory data set is reordered according to the sequence of the temporal center coordinates corresponding to each behavior instance to obtain a memory prompt set;

[0007] Receive input word data, input the input word data into a pre-trained language model, output a text query representation, calculate a matching score between the memory prompt set and the text query representation, and select several memory prompts with the highest matching scores to form an anchor frame subset;

[0008] The anchor box subset and the initial video representation are input into the Transformer module, and the anchor box representation with frame-level content enhancement is output. The text query representation is used as the guiding information and is element-wise multiplied with the anchor box subset and the anchor box representation with frame-level content enhancement to obtain the final cross-modal representation.

[0009] The final cross-modal representation is input into a pre-established feedforward network-based regressor, and the prediction deviation is output. The initial anchor box boundary is adjusted based on the prediction deviation to obtain the final target moment positioning result.

[0010] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the input video sequence length is T, and a pre-trained visual backbone network is used to extract the representation As the initial video representation, where D is the feature dimension.

[0011] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the process of inputting the initial video representation into a pre-established Transformer module:

[0012] The initial video representation X is fed into the encoder of the Transformer module to generate a video representation enhanced by self-attention. The decoder of the Transformer module takes the randomly initialized query and the video representation enhanced by self-attention as input, and obtains the dynamic context memory dataset through cross-attention interaction calculation. M represents the number of behavior instances appearing in the video.

[0013] In combination with the first aspect, in some implementations of the first aspect, the method further includes: a process of calculating the predicted behavior category score vector and the time series center coordinates and span corresponding to each behavior instance based on the memory items in the dynamic contextual memory data set:

[0014] Each memory entry M m It is sent to the linear classifier and after softmax normalization operation, the predicted behavior category score vector is obtained. C is the number of behavior categories that appear in the preset video;

[0015] Each memory entry M m It is fed into the time series predictor composed of a feedforward network and a ReLU activation function to generate prediction results corresponding to each behavior instance, including the time series center σ m and span ω m .

[0016] In combination with the first aspect, in some implementations of the first aspect, the method further includes: constructing a discriminative learning criterion and a semantic diversity criterion based on the behavior category score vector, and the discriminative learning criterion is as follows:

[0017]

[0018] In the formula, the behavior distribution p of the constraint prediction m Approaching one-hot encoding;

[0019] The semantic diversity learning criteria are as follows:

[0020]

[0021] By imposing an L2 norm constraint on the normalized average behavior category prediction vector of all memory entries, the occurrence probabilities of different behavior categories are made to approach a uniform distribution.

[0022] In combination with the first aspect, in some implementations of the first aspect, the method further includes: constructing a time series span diversity learning criterion based on the prediction result, as follows:

[0023]

[0024] in, Represents the calculation of the normalized temporal intersection-over-union ratio between two behavior instances.

[0025] In combination with the first aspect, in some implementations of the first aspect, the method further includes: a process of reordering the dynamic contextual memory data set according to the sequence of the temporal center coordinates corresponding to each behavior instance to obtain a memory prompt set:

[0026] According to the time series center coordinates σ of each behavior instance m The order of the initial situational memory set M is reordered to obtain Encode the temporal position Add to the corresponding memory items to construct a memory prompt set

[0027] In combination with the first aspect, in some implementations of the first aspect, the method further includes: a process of constructing the anchor box subset:

[0028] For the natural language query consisting of L words, a pre-trained language model is used to extract the text query representation.

[0029] Count each memory cue The matching score S between the text query representation q m ; Select the top K memory cues with the highest matching scores to form the anchor box subset

[0030] In combination with the first aspect, in some implementations of the first aspect, the method further includes: generating the anchor frame representation U in the following manner:

[0031] U=MCA(R,X,X)

[0032] Where MCA(·) represents the multi-head cross attention operation in the Transformer module decoder, whose anchor box subset R is used as the query Query input, and the key Key and value Value are derived respectively through the initial video representation X;

[0033] The text query-guided cross-modal representation generation method is as follows:

[0034] H k =[R k ⊙q;U k ⊙q]

[0035] Among them, R and U represent the filtered memory cue set and the content-enhanced anchor box representation respectively, ⊙ is the element-by-element multiplication calculation, and [·;·] represents the feature concatenation operation.

[0036] In a second aspect, in order to achieve the above-mentioned purpose, the present invention discloses a cross-modal video temporal content positioning system based on contextual memory prompts, comprising:

[0037] A data processing module is used to receive an input video sequence, input the input video sequence into a pre-trained visual backbone network model, output an initial video representation, input the initial video representation into a pre-established Transformer module, and output a dynamic contextual memory data set;

[0038] A data sorting module is used to calculate the predicted behavior category score vector and the time series center coordinates and span corresponding to each behavior instance based on the memory items in the dynamic situational memory data set, and re-sort the dynamic situational memory data set according to the order of the time series center coordinates corresponding to each behavior instance to obtain a memory prompt set;

[0039] A text extraction module is used to receive input word data, input the input word data into a pre-trained language model, output a text query representation, calculate a matching score between the memory prompt set and the text query representation, and select a number of memory prompts with the highest matching scores to form an anchor frame subset;

[0040] The feature calculation module is used to input the anchor box subset and the initial video representation into the Transformer module, output the anchor box representation with frame-level content enhancement, use the text query representation as the guide information, perform element-by-element multiplication with the anchor box subset and the anchor box representation with frame-level content enhancement, and obtain the final cross-modal representation;

[0041] The content localization module is used to input the final cross-modal representation into a pre-established regressor based on a feedforward network, output the prediction deviation, adjust the initial anchor box boundary based on the prediction deviation, and obtain the final target moment localization result.

[0042] Beneficial effects of the present invention:

[0043] The present invention can effectively alleviate the semantic gap between high-level concepts and low-level video features contained in natural language queries by generating a series of behavior-centered contextual memories, and provide behavioral prompt information for accurately and quickly locating the target moment. It is conducive to the formulation of an unsupervised contextual memory learning paradigm based on discriminative and diversity criteria, without the need for additional behavioral instance annotations, to ensure the scalability of the model. The constructed two-stage framework including contextual memory generation and query-guided moment positioning conforms to human cognitive habits. Compared with traditional sliding window-based or multi-scale anchor frame-based methods, the cross-modal video temporal content positioning method proposed in the present invention can better balance accuracy and computational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 It is a schematic flow chart of the method of the present invention;

[0046] Figure 2 It is a schematic diagram of the workflow of the present invention;

[0047] Figure 3 It is a video temporal content positioning effect diagram of the present invention on the MAD data set;

[0048] Figure 4 It is a video temporal content positioning effect diagram of the present invention on the Ego4d data set;

[0049] Figure 5 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] Embodiment 1:

[0052] like Figure 1 As shown, a method for localizing temporal content of a cross-modal video based on contextual memory cues comprises the following steps:

[0053] S101: receiving an input video sequence, inputting the input video sequence into a pre-trained visual backbone network model, outputting an initial video representation, inputting the initial video representation into a pre-established Transformer module, and outputting a dynamic contextual memory dataset;

[0054] The input video sequence length is T, and the pre-trained visual backbone network is used to extract the representation As the initial video representation, where D is the feature dimension;

[0055] The process of inputting the initial video representation into the pre-established Transformer module:

[0056] The initial video representation X is fed into the encoder of the Transformer module to generate a video representation enhanced by self-attention. The decoder of the Transformer module takes the randomly initialized query and the video representation enhanced by self-attention as input, and obtains the dynamic context memory dataset through cross-attention interaction calculation. M represents the number of behavior instances appearing in the video.

[0057] S102: Calculating a predicted behavior category score vector and a temporal center coordinate and span corresponding to each behavior instance based on the memory items in the dynamic situational memory data set, and reordering the dynamic situational memory data set according to the sequence of the temporal center coordinates corresponding to each behavior instance to obtain a memory prompt set;

[0058] The process of calculating the predicted behavior category score vector and the time series center coordinates and span corresponding to each behavior instance based on the memory items in the dynamic contextual memory data set:

[0059] Each memory entry M m It is sent to the linear classifier and after softmax normalization operation, the predicted behavior category score vector is obtained. C is the number of behavior categories that appear in the preset video; discriminative and semantic diversity criteria are constructed based on the behavior category score vector to guide the learning of contextual memory;

[0060] Each memory entry M m It is sent to the timing predictor composed of a feedforward network and a ReLU activation function to generate the timing center σ corresponding to each behavior instance m and span ω m ; Based on the prediction results, a temporal span diversity criterion is constructed to suppress the temporal overlap between behavioral instances corresponding to contextual memory;

[0061] The behavior category score vector is used as the basis to construct a discriminative learning criterion and a semantic diversity criterion. The discriminative learning criterion is as follows:

[0062]

[0063] The above formula will encourage a certain class to have a significantly higher confidence score than other classes, which is to constrain the predicted behavior distribution p m Approaching one-hot encoding forces each memory entry to correspond to a specific behavior category with relatively high certainty.

[0064] The semantic diversity learning criteria are as follows:

[0065]

[0066] The above formula imposes an L2 norm constraint on the normalized average behavior category prediction vector of all memory items to encourage the occurrence probabilities of different behavior categories to approach a uniform distribution. That is, it forces the model to evenly assign behavior category labels to all situational memories, thereby promoting semantic diversity among memory items.

[0067] The prediction results are used as the basis for constructing the time series span diversity learning criteria, as follows:

[0068]

[0069] in, Represents the calculation of the normalized temporal intersection-over-union ratio between two behavior instances.

[0070] The process of reordering the dynamic situational memory data set according to the sequence of the temporal center coordinates corresponding to each behavior instance to obtain the memory prompt set:

[0071] According to the time series center coordinates σ of each behavior instance m The order of the initial situational memory set M is reordered to obtain Encode the temporal position Add to the corresponding memory items to construct a memory prompt set

[0072] S103: receiving input word data, inputting the input word data into a pre-trained language model, outputting a text query representation, calculating a matching score between the memory prompt set and the text query representation, and selecting a number of memory prompts with the highest matching scores to form an anchor frame subset;

[0073] The construction process of the anchor box subset is as follows:

[0074] For the natural language query consisting of L words, a pre-trained language model is used to extract the text query representation.

[0075] Count each memory cue The matching score S between the text query representation q m ; Select the top K memory cues with the highest matching scores to form the anchor box subset

[0076] S104: input the anchor box subset and the initial video representation into the Transformer module, output the anchor box representation with frame-level content enhancement, use the text query representation as guidance information, perform element-by-element multiplication with the anchor box subset and the anchor box representation with frame-level content enhancement, and obtain the final cross-modal representation;

[0077] The filtered memory cue set R and the initial video representation X are used as the input of the Transformer module decoder, and the anchor box representation U with frame-level content enhancement is generated through cross-attention interaction calculation;

[0078] The text query representation q is used as the guiding information, and is element-wise multiplied with the filtered memory prompt set R and the content-enhanced anchor box representation U, and then further concatenated and fused to obtain the final cross-modal representation.

[0079]

[0080] The anchor frame representation U is generated as follows:

[0081] U=MCA(R,X,X)

[0082] Where MCA(·) represents the multi-head cross attention operation in the Transformer module decoder, whose anchor box subset R is used as the query Query input, and the key Key and value Value are derived respectively through the initial video representation X;

[0083] The text query-guided cross-modal representation generation method is as follows:

[0084] H k =[Rk ⊙q;U k ⊙q]

[0085] Among them, R and U represent the filtered memory cue set and the content-enhanced anchor box representation respectively, ⊙ is the element-by-element multiplication calculation, and [@;@] represents the feature concatenation operation.

[0086] S105: Input the final cross-modal representation into a pre-established regressor based on a feedforward network, output the prediction deviation, adjust the initial anchor box boundary based on the prediction deviation, and obtain the final target moment positioning result.

[0087] For each selected memory cue (anchor frame), its corresponding temporal center σ k and span ω k Convert to fragment boundaries The cross-modal representation H corresponding to the memory cue k Feed it into a regressor based on a feedforward network to predict the deviation of the start and end points of the segment boundary The initial anchor box boundary is adjusted based on the prediction deviation to produce the final target moment positioning result.

[0088] Based on prediction bias The calculation method for adjusting the initial anchor box boundary is as follows:

[0089]

[0090] in, and Represent the initial and adjusted segment boundaries, respectively.

[0091] The boundary regression loss used to supervise the target moment localization is defined as follows:

[0092]

[0093] in, Represents the ground truth boundary of the target video segment corresponding to the input text query.

[0094] The performance of the cross-modal video temporal content localization method based on a two-stage framework under contextual memory cues is shown in Figure 2. Figure 3-Figure 4 shown. Figure 3 The temporal localization results on the MAD dataset are given. When a long text query containing multiple continuous behaviors is given, the method proposed in the present invention can locate the semantically relevant contextual content more accurately. Figure 4 The temporal localization results on the Ego4d dataset are given. When applied to first-person perspective videos involving fine-grained human-object interactions, the proposed method can effectively cope with challenges such as tiny objects, cluttered backgrounds, and frequent camera motions, and achieve robust temporal content localization results.

[0095] Table 1 Comparison results of video temporal content localization accuracy of the present invention on the MAD dataset

[0096]

[0097] Table 2 Comparison results of video temporal content localization accuracy of the present invention on the Ego4d dataset

[0098]

[0099] Tables 1 and 2 respectively give the comparative results of the video temporal content localization accuracy of the present invention on the MAD and Ego4d datasets (where IoU represents the temporal intersection over union ratio, and R@n represents the recall rate when selecting the top n prediction results). On the MAD dataset, compared with sliding window-based methods (such as VLG-Net, CLIP, and CONE), the method of the present invention achieved performance improvements of no less than 3.8% and 2.2% on the R@1-0.3 and R@1-0.5 indicators, respectively. This shows that the method of the present invention has stronger long-range cross-modal temporal reasoning capabilities, especially considering that the MAD dataset covers long videos with an average length of approximately 111 minutes. In addition, on the Ego4d dataset containing first-person perspective videos, compared with the two-stage end-to-end model SOONet, The method of the present invention is in R@ 5-0.3 The performance of the two indicators, R@5-0.5, and R@5-0.5, respectively, was improved by 1.02% and 0.83%. This verifies that contextual memory can effectively bridge the semantic gap between the high-level behavioral concepts contained in text queries and the corresponding low-level visual content in long videos.

[0100] Embodiment 2: In the second aspect, as Figure 5 As shown, in order to achieve the above-mentioned purpose, the present invention discloses a cross-modal video temporal content positioning system based on contextual memory prompts, comprising:

[0101] The data processing module 11 is used to receive an input video sequence, input the input video sequence into a pre-trained visual backbone network model, output an initial video representation, input the initial video representation into a pre-established Transformer module, and output a dynamic contextual memory data set;

[0102] A data sorting module 12 is used to calculate the predicted behavior category score vector and the time series center coordinates and span corresponding to each behavior instance based on the memory items in the dynamic situational memory data set, and re-sort the dynamic situational memory data set according to the sequence of the time series center coordinates corresponding to each behavior instance to obtain a memory prompt set;

[0103] The text extraction module 13 is used to receive input word data, input the input word data into a pre-trained language model, output a text query representation, calculate a matching score between the memory prompt set and the text query representation, and select a number of memory prompts with the highest matching scores to form an anchor frame subset;

[0104] A feature calculation module 14 is used to input the anchor box subset and the initial video representation into the Transformer module, output the anchor box representation with frame-level content enhancement, use the text query representation as guidance information, perform element-by-element multiplication calculation with the anchor box subset and the anchor box representation with frame-level content enhancement, and obtain the final cross-modal representation;

[0105] The content positioning module 15 is used to input the final cross-modal representation into a pre-established regressor based on a feedforward network, output a prediction deviation, adjust the initial anchor box boundary based on the prediction deviation, and obtain the final target moment positioning result.

[0106] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors, and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is used to implement one or more instructions, specifically for loading and executing one or more instructions in a computer storage medium to implement the above method.

[0107] It needs to be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium, on which a computer program is stored, and the computer program is executed by a processor to execute the above method. The storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0108] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0109] The above shows and describes the basic principles, main features and advantages of the present disclosure. Those skilled in the art should understand that the present disclosure is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, the present disclosure may have various changes and improvements, and these changes and improvements fall within the scope of the present disclosure to be protected.

Claims

1. A cross-modal video temporal content localization method based on contextual memory cues, characterized in that: The method comprises the following steps: Receive an input video sequence, input the input video sequence into a pre-trained visual backbone network model, output an initial video representation, input the initial video representation into a pre-established Transformer module, and output a dynamic contextual memory dataset; Based on the memory items in the dynamic situational memory data set, the predicted behavior category score vector and the temporal center coordinates and span corresponding to each behavior instance are calculated, and the dynamic situational memory data set is reordered according to the sequence of the temporal center coordinates corresponding to each behavior instance to obtain a memory prompt set; Receive input word data, input the input word data into a pre-trained language model, output a text query representation, calculate a matching score between the memory prompt set and the text query representation, and select several memory prompts with the highest matching scores to form an anchor frame subset; The anchor box subset and the initial video representation are input into the Transformer module, and the anchor box representation with frame-level content enhancement is output. The text query representation is used as the guiding information and is element-wise multiplied with the anchor box subset and the anchor box representation with frame-level content enhancement to obtain the final cross-modal representation. The final cross-modal representation is input into a pre-established feedforward network-based regressor, and the prediction deviation is output. The initial anchor box boundary is adjusted based on the prediction deviation to obtain the final target moment positioning result.

2. The method for localizing temporal content of cross-modal videos based on contextual memory prompts according to claim 1, characterized in that: The input video sequence length is T, and the pre-trained visual backbone network is used to extract the representation As the initial video representation, where D is the feature dimension.

3. The method for localizing temporal content of cross-modal videos based on contextual memory prompts according to claim 1, characterized in that: The process of inputting the initial video representation into the pre-established Transformer module: The initial video representation X is fed into the encoder of the Transformer module to generate a video representation enhanced by self-attention. The decoder of the Transformer module takes the randomly initialized query and the video representation enhanced by self-attention as input, and obtains the dynamic context memory dataset through cross-attention interaction calculation. M represents the number of behavior instances appearing in the video.

4. The method for localizing temporal content of cross-modal videos based on contextual memory prompts according to claim 1, characterized in that: The process of calculating the predicted behavior category score vector and the time series center coordinates and span corresponding to each behavior instance based on the memory items in the dynamic contextual memory data set: Each memory entry M m It is sent to the linear classifier and after softmax normalization operation, the predicted behavior category score vector is obtained. C is the number of behavior categories that appear in the preset video; Each memory entry M m It is fed into the time series predictor composed of a feedforward network and a ReLU activation function to generate prediction results corresponding to each behavior instance, including the time series center σ m and span ω m .

5. The method for localizing temporal content of cross-modal video based on contextual memory prompts according to claim 4, characterized in that: The behavior category score vector is used as the basis to construct a discriminative learning criterion and a semantic diversity criterion. The discriminative learning criterion is as follows: In the formula, the behavior distribution p of the constraint prediction m Approaching one-hot encoding; The semantic diversity learning criteria are as follows: By imposing an L2 norm constraint on the normalized average behavior category prediction vector of all memory entries, the occurrence probabilities of different behavior categories are made to approach a uniform distribution.

6. The method for localizing temporal content of cross-modal videos based on contextual memory cues according to claim 5, characterized in that: The prediction results are used as the basis for constructing the time series span diversity learning criteria, as follows: in, Represents the calculation of the normalized temporal intersection-over-union ratio between two behavior instances.

7. The method for localizing temporal content of cross-modal videos based on contextual memory prompts according to claim 1, characterized in that: The process of reordering the dynamic situational memory data set according to the sequence of the temporal center coordinates corresponding to each behavior instance to obtain the memory prompt set: According to the time series center coordinates σ of each behavior instance m The order of the initial situational memory set M is reordered to obtain Encode the temporal position Add to the corresponding memory items to construct a memory prompt set 8. The method for localizing temporal content of cross-modal videos based on contextual memory prompts according to claim 1, characterized in that: The construction process of the anchor box subset is as follows: For the natural language query consisting of L words, a pre-trained language model is used to extract the text query representation. Count each memory cue The matching score S between the text query representation q m ; Select the top K memory cues with the highest matching scores to form the anchor box subset Memory Tips The matching degree with the text query representation q is calculated as follows: Here, σ(·) represents the sigmoid function.

9. The method for localizing temporal content of cross-modal videos based on contextual memory prompts according to claim 1, characterized in that: The anchor frame representation U is generated as follows: U=MCA(R,X,X) Where MCA(·) represents the multi-head cross attention operation in the Transformer module decoder, whose anchor box subset R is used as the query Query input, and the key Key and value Value are derived respectively through the initial video representation X; The text query-guided cross-modal representation generation method is as follows: H k =[R k ⊙q;U k ⊙q] Among them, R and U represent the filtered memory cue set and the content-enhanced anchor box representation respectively, ⊙ is the element-by-element multiplication calculation, and [·;·] represents the feature concatenation operation.

10. A cross-modal video temporal content localization system based on contextual memory cues, characterized in that: include: A data processing module is used to receive an input video sequence, input the input video sequence into a pre-trained visual backbone network model, output an initial video representation, input the initial video representation into a pre-established Transformer module, and output a dynamic contextual memory data set; A data sorting module is used to calculate the predicted behavior category score vector and the time series center coordinates and span corresponding to each behavior instance based on the memory items in the dynamic situational memory data set, and re-sort the dynamic situational memory data set according to the order of the time series center coordinates corresponding to each behavior instance to obtain a memory prompt set; A text extraction module is used to receive input word data, input the input word data into a pre-trained language model, output a text query representation, calculate a matching score between the memory prompt set and the text query representation, and select a number of memory prompts with the highest matching scores to form an anchor frame subset; The feature calculation module is used to input the anchor box subset and the initial video representation into the Transformer module, output the anchor box representation with frame-level content enhancement, use the text query representation as the guide information, perform element-by-element multiplication with the anchor box subset and the anchor box representation with frame-level content enhancement, and obtain the final cross-modal representation; The content localization module is used to input the final cross-modal representation into a pre-established regressor based on a feedforward network, output the prediction deviation, adjust the initial anchor box boundary based on the prediction deviation, and obtain the final target moment localization result.

Images