Cross-modal video temporal content localization method and system based on contextual memory cues

Abstract

The application discloses a cross-modal video timing content positioning method and system based on context memory prompts, relates to the technical field of cross-modal understanding, and generates a group of context memories dynamically in stage one, and explicitly summarizes various behavior instances occurring in the input video; an unsupervised memory learning paradigm is formulated by applying discriminative and diverse constraints to eliminate the dependence on additional behavior instance labeling. Secondly, in stage two, enhanced memory prompts are obtained based on frame-level detail content supplement and language query interaction, and further used as anchor frames to realize accurate and efficient regression of the target time segment boundary. Compared with the traditional strategy based on a sliding window or a multi-scale anchor frame, the cross-modal video timing content positioning method disclosed by the application is more in line with cognitive habits, and has performance and efficiency advantages in processing long videos.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cross-modal understanding, in particular to a cross-modal video time sequence content positioning method and system based on context memory prompts. BACKGROUND

[0002] As one of the research hotspots in the field of visual-language understanding, cross-modal video time sequence content positioning technology aims to detect the segments in a video that are semantically related to a given natural language query; and can further analyze and understand based on the positioning results to serve downstream tasks such as video retrieval, video description generation, and video question answering. The research results have good application value in security monitoring, autonomous driving, multimedia analysis, human-computer interaction, etc. Although cross-modal video time sequence content positioning technology has broad application prospects and research value, researchers have made many research progress in recent years, but most of the existing work focuses on videos with short time span (e.g., minutes). With the rapid growth in the number and length of network videos, research on cross-modal time sequence content positioning for long time span videos (e.g., hours) becomes increasingly important. However, the existing analysis methods for short videos usually use down-sampling strategies to obtain fixed-length sequences for processing, but this will inevitably exacerbate information loss when processing long videos. SUMMARY

[0003] To solve the problems mentioned in the background, the purpose of the present application is to provide a cross-modal video time sequence content positioning method and system based on context memory prompts.

[0004] In a first aspect, the purpose of the present application can be achieved by the following technical solution: a cross-modal video time sequence content positioning method based on context memory prompts, the method comprising the following steps:

[0005] Receiving an input video sequence, inputting the input video sequence into a pre-trained visual backbone network model, outputting an initial video representation, inputting the initial video representation into a pre-established Transformer module, and outputting a dynamic context memory dataset;

[0006] Based on the memory entries in the dynamic context memory dataset, a predicted behavior class score vector and the time sequence center coordinates and span corresponding to each behavior instance are calculated, the dynamic context memory dataset is reordered according to the order of the time sequence center coordinates corresponding to each behavior instance to obtain a memory prompt set;

[0007] Receiving read-in word data, inputting the read-in word data into a pre-trained language model, outputting a text query representation, calculating the matching degree score of the memory prompt set and the text query representation, and selecting a number of memory prompts with the highest matching degree score to form an anchor frame subset;

[0008] The anchor box subset and the initial video representation are input into a Transformer module, and an anchor box representation with frame-level content enhancement is output. The text query representation is multiplied element by element with the anchor box subset and the anchor box representation with frame-level content enhancement respectively as guide information, and a final cross-modal representation is obtained.

[0009] The final cross-modal representation is input into a pre-established regressor based on a feedforward network, and a predicted bias is output. The initial anchor box boundary is adjusted based on the predicted bias, and a final target time positioning result is obtained.

[0010] In combination with the first aspect, in some implementations of the first aspect, the method further includes that the input video sequence has a length of T, and a pre-trained visual backbone network is used to extract a representation as the initial video representation, where D is a feature dimension.

[0011] In combination with the first aspect, in some implementations of the first aspect, the method further includes that the process of inputting the initial video representation into a pre-established Transformer module includes:

[0012] The initial video representation X is input into an encoder of the Transformer module to generate a self-attention enhanced video representation. A decoder of the Transformer module takes a randomly initialized query and the self-attention enhanced video representation as input, and obtains a dynamic context memory dataset through cross-attention interaction calculation. M represents the number of behavior instances appearing in the video.

[0013] In combination with the first aspect, in some implementations of the first aspect, the method further includes that the process of calculating the predicted behavior class score vector and the time sequence center coordinates and span corresponding to each behavior instance based on the memory entries in the dynamic context memory dataset includes:

[0014] Each memory entry M m is input into a linear classifier, and after a softmax normalization operation, a predicted behavior class score vector C is a preset number of behavior classes appearing in the video.

[0015] Each memory entry M m is input into a time sequence predictor composed of a feedforward network and a ReLU activation function to generate a predicted result corresponding to each behavior instance, including a time sequence center m and a span m .

[0016] In conjunction with the first aspect, in certain implementations of the first aspect, the method further includes: constructing a discriminative learning criterion and a semantic diversity criterion based on the behavior category score vector, wherein the discriminative learning criterion is as follows:

[0017]

[0018] Where, the behavior distribution p of the constraint prediction is m Approaching one-hot encoding;

[0019] The semantic diversity learning criteria are as follows:

[0020]

[0021] By imposing an L2 norm constraint on the normalized average behavior category prediction vector of all memory entries, the occurrence probabilities of different behavior categories are made to approach a uniform distribution.

[0022] In conjunction with the first aspect, in certain implementations of the first aspect, the method further includes: constructing a time series span diversity learning criterion based on the prediction result, as follows:

[0023]

[0024] in, Represents the calculation of the normalized temporal intersection-over-union ratio between two behavior instances.

[0025] In conjunction with the first aspect, in certain implementations of the first aspect, the method further includes: a process of reordering the dynamic contextual memory data set according to the order of the temporal center coordinates corresponding to each behavior instance to obtain the memory prompt set:

[0026] According to the time series center coordinates σ of each behavior instance m The order of the initial context memory set M is reordered to obtain Encode the temporal position Add to the corresponding memory items to build a memory prompt set

[0027] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: a process of constructing the anchor box subset:

[0028] For the natural language query consisting of L words read in, a pre-trained language model is used to extract the text query representation

[0029] Count each memory cue The matching score S with the text query representation q m ; Select the top K memory cues with the highest matching scores to form the anchor box subset

[0030] With reference to the first aspect, in some implementations of the first aspect, the method further comprises that the anchor box representation U is generated in the following manner:

[0031] U = MCA(R, X, X)

[0032] wherein MCA(·) represents a multi-head cross-attention operation in a Transformer module decoder, an anchor box subset R is input as a query Query, and keys Key and values Value are derived respectively through an initial video representation X;

[0033] The text query guided cross-modal representation is generated in the following manner:

[0034] H k = [R k ⊙ q; U k ⊙ q]

[0035] wherein R and U represent a filtered memory prompt set and a content enhanced anchor box representation respectively, is an element-wise multiplication calculation, and [·;·] represents a feature concatenation operation.

[0036] The second aspect, in order to achieve the above object, the application discloses a cross-modal video time sequence content positioning system based on situational memory prompts, comprising:

[0037] A data processing module is configured to receive an input video sequence, input the input video sequence into a pre-trained visual backbone network model, output an initial video representation, input the initial video representation into a pre-established Transformer module, and output a dynamic situational memory dataset;

[0038] A data sorting module is configured to calculate a predicted behavior category score vector and time sequence center coordinates and spans corresponding to each behavior instance based on memory entries in the dynamic situational memory dataset, reorder the dynamic situational memory dataset according to the order of the time sequence center coordinates corresponding to each behavior instance to obtain a memory prompt set;

[0039] A text extraction module is configured to receive read-in word data, input the read-in word data into a pre-trained language model, output a text query representation, calculate a matching degree score of the memory prompt set and the text query representation, and select a plurality of memory prompts with the highest matching degree scores to form an anchor box subset;

[0040] The feature calculation module is used for inputting the anchor frame subset and the initial video representation into a Transformer module, outputting an anchor frame representation with frame-level content enhancement, and performing element-by-element multiplication calculation on the text query representation as guide information and the anchor frame subset and the anchor frame representation with frame-level content enhancement respectively to obtain a final cross-modal representation.

[0041] The content positioning module is used for inputting the final cross-modal representation into a pre-established regressor based on a feedforward network, outputting a predicted deviation, adjusting the initial anchor frame boundary based on the predicted deviation, and obtaining a final target time positioning result.

[0042] The present application has the following advantages:

[0043] The present application can effectively alleviate the semantic gap between high-level concepts and low-level video features contained in natural language queries by generating a series of behavior-centered context memories, and provide behavior prompt information for accurate and rapid positioning of the target time. It is beneficial to develop an unsupervised context memory learning paradigm based on discriminative and diverse criteria, without relying on additional behavior instance labeling, ensuring the scalability of the model. The two-stage framework for time positioning constructed by the present application includes context memory generation and query guidance, which conforms to human cognitive habits. Compared with traditional methods based on sliding windows or multi-scale anchor frames, the cross-modal video timing content positioning method proposed by the present application can better balance accuracy and computational efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, brief descriptions will be given below to the drawings needed to be used in the embodiments or prior art descriptions, and obviously, other drawings can also be obtained by those skilled in the art without creative effort on the premise of not paying creative effort;

[0045] Figure 1 is a method flowchart of the present application;

[0046] Figure 2 is a work flowchart of the present application;

[0047] Figure 3 is a video timing content positioning effect diagram of the present application on the MAD dataset;

[0048] Figure 4 is a video timing content positioning effect diagram of the present application on the Ego4d dataset;

[0049] Figure 5 is a system structure diagram of the present application. DETAILED DESCRIPTION

[0050] With reference to the accompanying drawings: the technical solutions in the embodiments of the present application will be described clearly and completely, obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0051] Embodiment one:

[0052] As shown in the cross-modal video timing content positioning method based on context memory prompt, the method comprises the following steps: Figure 1

[0053] S101: receiving an input video sequence, inputting the input video sequence into a pre-trained visual backbone network model, outputting an initial video representation, inputting the initial video representation into a pre-established Transformer module, and outputting a dynamic context memory dataset;

[0054] The input video sequence has a length of T, and a pre-trained visual backbone network is used to extract a representation as the initial video representation, wherein D is the feature dimension;

[0055] The process of inputting the initial video representation into the pre-established Transformer module comprises:

[0056] The initial video representation X is input into the encoder of the Transformer module to generate a self-attention enhanced video representation; the decoder of the Transformer module takes the randomly initialized query and the self-attention enhanced video representation as input, and calculates the dynamic context memory dataset through cross-attention interaction. M represents the number of behavior instances appearing in the video.

[0057] S102: calculating a predicted behavior category score vector and a timing center coordinate and span corresponding to each behavior instance based on the memory entries in the dynamic context memory dataset, and reordering the dynamic context memory dataset according to the order of the timing center coordinates corresponding to each behavior instance to obtain a memory prompt set;

[0058] The process of calculating a predicted behavior category score vector and a timing center coordinate and span corresponding to each behavior instance based on the memory entries in the dynamic context memory dataset comprises:

[0059] Each memory entry M m is input into a linear classifier, and after a softmax normalization operation, a predicted behavior category score vector ​C is the preset number of behavior categories appearing in the video; based on the behavior category score vector, discriminative and semantic diversity criteria are constructed to guide the learning of contextual memory;

[0060] Each memory entry M m is fed into a temporal predictor composed of a feedforward network and a ReLU activation function to generate the temporal center σ m and span ω m corresponding to each behavior instance; based on the prediction results, a temporal span diversity criterion is constructed to suppress the temporal overlap between the behavior instances corresponding to the contextual memory;

[0061] The behavior category score vector is used as the basis to construct a discriminative learning criterion and a semantic diversity criterion, and the discriminative learning criterion is as follows:

[0062]

[0063] The above formula will encourage the confidence score of a certain specific category to be significantly higher than that of other categories, that is, to constrain the predicted behavior distribution p m to a one-hot encoding approximation, thereby forcing each memory entry to establish a corresponding relationship with a specific behavior category at a relatively high certainty.

[0064] The semantic diversity learning criterion is as follows:

[0065]

[0066] The above formula applies an L2 norm constraint to the normalized average behavior category prediction vector of all memory entries to encourage the appearance probability of different behavior categories to approach a uniform distribution, that is, to force the model to uniformly assign behavior category labels to all contextual memories, thereby promoting semantic diversity among memory entries.

[0067] The prediction results are used as the basis to construct a temporal span diversity learning criterion, as follows:

[0068]

[0069] wherein, denotes the normalized temporal intersection over union calculation between two behavior instances.

[0070] The process of reordering the dynamic contextual memory dataset to obtain the memory prompt set according to the chronological order of the temporal center coordinates of each behavior instance is as follows:

[0071] According to the chronological order of the temporal center coordinates σ m of each behavior instance, the initial contextual memory set M is reordered to obtain The temporal position encoding is added to the corresponding memory entry to construct the memory prompt set

[0072] S103: receiving the read-in word data, inputting the read-in word data into a pre-trained language model, outputting a text query representation, calculating a matching degree score of the memory prompt set and the text query representation, and selecting a few memory prompts with the highest matching degree scores to form an anchor frame subset;

[0073] The construction process of the anchor frame subset is as follows:

[0074] For the read-in natural language query composed of L words, a pre-trained language model is used to extract a text query representation

[0075] Calculate the matching degree score S between each memory prompt and the text query representation q m ; select the top K memory prompts with the highest matching degree scores to form an anchor frame subset

[0076] S104: inputting the anchor frame subset and the initial video representation into a Transformer module, outputting a frame-level content-enhanced anchor frame representation, taking the text query representation as guide information, and performing element-by-element multiplication calculation with the anchor frame subset and the frame-level content-enhanced anchor frame representation respectively to obtain a final cross-modal representation;

[0077] Taking the filtered memory prompt set R and the initial video representation X as inputs of a Transformer module decoder, a frame-level content-enhanced anchor frame representation U is generated through cross-attention interaction calculation;

[0078] Taking the text query representation q as guide information, element-by-element multiplication calculation is performed with the filtered memory prompt set R and the content-enhanced anchor frame representation U respectively, and further splicing and fusion are performed to obtain a final cross-modal representation

[0079]

[0080] The generation method of the anchor frame representation U is as follows:

[0081] U=MCA(R,X,X)

[0082] Wherein, MCA(·) represents a multi-head cross-attention operation in the Transformer module decoder, the anchor frame subset R is input as a query Query, and the key Key and value Value are derived through the initial video representation X respectively;

[0083] The generation method of the text query guided cross-modal representation is as follows:

[0084] H k =[Rk ⊙q;U k ⊙q]

[0085] where R and U represent the filtered memory cues set and the content-augmented anchor representation respectively, ⊙ denotes element-wise multiplication, and [email protected] denotes feature concatenation.

[0086] S105: input the final cross-modal representation into a pre-established regression based on a feedforward network, output the predicted bias, adjust the initial anchor boundary based on the predicted bias, and obtain the final target time positioning result.

[0087] For each selected memory cue (anchor), its corresponding time sequence center σ k and span ω k are converted into segment boundaries The cross-modal representation H k corresponding to the memory cue is input into a regression based on a feedforward network to predict the bias of the start and end points of the segment boundary Based on the predicted bias, the initial anchor boundary is adjusted to generate the final target time positioning result.

[0088] Based on the predicted bias The calculation method of adjusting the initial anchor boundary is as follows:

[0089]

[0090] wherein, and represent the initial and adjusted segment boundaries respectively.

[0091] The boundary regression loss for supervised target time positioning is defined as follows:

[0092]

[0093] wherein, denotes the true value boundary of the target video segment corresponding to the input text query.

[0094] The performance of the cross-modal video time sequence content positioning method based on the two-stage framework of situational memory cues is shown in Figures 3-4 . Figure 3 The time sequence positioning results on the MAD dataset are given, and when a long text query containing multiple continuous behaviors is given, the method proposed in the application can accurately locate the contextually relevant content; Figure 4 The time sequence positioning results on the Ego4d dataset are given, and when applied to the first-person video involving fine-grained human-object interaction behaviors, the method proposed in the application can effectively cope with small objects, cluttered backgrounds, frequent camera movements and other challenges, and achieve robust time sequence content positioning effect.

[0095] Table 1 Video temporal content localization accuracy comparison results of the present application on the MAD dataset

[0096]

[0097] Table 2 Video temporal content localization accuracy comparison results of the present application on the Ego4d dataset

[0098]

[0099] Tables 1 and 2 respectively give the video temporal content localization accuracy comparison results of the present application on the MAD and Ego4d datasets (where IoU represents the temporal intersection over union, and R@n represents the recall rate when selecting the top n prediction results). On the MAD dataset, compared with the sliding window-based methods (such as VLG-Net, CLIP and CONE), the present application method achieves a performance improvement of not less than 3.8% and 2.2% on the R@1-0.3 and R@1-0.5 indicators, respectively. This shows that the present application method has stronger long-range cross-modal temporal reasoning capability, especially considering that the MAD dataset covers long videos with an average duration of about 111 minutes. In addition, on the Ego4d dataset containing first-person videos, compared with the two-stage end-to-end model SOONet, The process of the present application is at R 5-0.3 and R@5-0.5 indicators, respectively, a performance improvement of 1.02% and 0.83% is achieved. This verifies that through context memory, the semantic gap between the high-level behavior concepts implied in the text query and the corresponding low-level visual content in the long video can be effectively bridged.

[0100] Embodiment Two: In the second aspect, as Figure 5 shown in order to achieve the above-mentioned purpose, the present application discloses a cross-modal video temporal content localization system based on context memory prompts, comprising:

[0101] The data processing module 11 is used for receiving an input video sequence, inputting the input video sequence into a pre-trained visual backbone network model, outputting an initial video representation, inputting the initial video representation into a pre-established Transformer module, and outputting a dynamic context memory dataset;

[0102] The data sorting module 12 is used for calculating a predicted behavior class score vector and the temporal center coordinates and span corresponding to each behavior instance based on the memory entries in the dynamic context memory dataset, and reordering the dynamic context memory dataset according to the chronological order of the temporal center coordinates corresponding to each behavior instance to obtain a memory prompt set;

[0103] The text extraction module 13 is configured to receive the read-in word data, input the read-in word data into a pre-trained language model, output a text query representation, calculate a matching degree score of the memory prompt set and the text query representation, and select a plurality of memory prompts with the highest matching degree scores to form an anchor frame subset;

[0104] The feature calculation module 14 is configured to input the anchor frame subset and the initial video representation into a Transformer module, output an anchor frame representation with frame-level content enhancement, multiply the text query representation and the anchor frame subset and the anchor frame representation with frame-level content enhancement respectively as guide information to obtain a final cross-modal representation.

[0105] The content positioning module 15 is configured to input the final cross-modal representation into a pre-established regression regressor based on a feedforward network, output a predicted deviation, adjust the initial anchor frame boundary based on the predicted deviation, and obtain a final target time positioning result.

[0106] Based on the same inventive concept, the application further provides a computer device, which comprises one or more processors and a memory for storing one or more computer programs; the program comprises program instructions, and the processor is configured to execute the program instructions stored in the memory. The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, and is configured to implement one or more instructions, and is specifically configured to load and execute one or more instructions in the computer storage medium to implement the above method.

[0107] It should be further noted that based on the same inventive concept, the present application further provides a computer storage medium, which stores a computer program, and the computer program is run by a processor to execute the above method. The storage medium can adopt any combination of one or more computer readable media. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples (non-exhaustive list) of the computer readable storage medium include: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus.

[0108] In the description of the present application, the description of the terms "one embodiment", "an example", "a specific example" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present disclosure. In the present specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0109] The basic principles, main features and advantages of the present disclosure are shown and described above. It should be understood by those skilled in the art that the present disclosure is not limited by the above embodiments, and the above embodiments and descriptions in the specification are only to illustrate the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, various changes and improvements can be made to the present disclosure, and these changes and improvements all fall within the scope of the claimed present disclosure.

Claims

1. A cross-modal video temporal content localization method based on contextual memory cues, characterized by: The method comprises the following steps: Receive an input video sequence, input the input video sequence into a pre-trained visual backbone network model, output an initial video representation, input the initial video representation into a pre-established Transformer module, and output a dynamic contextual memory dataset; Based on the memory items in the dynamic contextual memory dataset, the predicted behavior category score vector and the temporal center coordinates and span corresponding to each behavior instance are calculated. The dynamic contextual memory dataset is reordered according to the order of the temporal center coordinates corresponding to each behavior instance to obtain a memory prompt set. Receive input word data, input the input word data into a pre-trained language model, output a text query representation, calculate the matching score between the memory prompt set and the text query representation, and select several memory prompts with the highest matching scores to form an anchor box subset; The anchor box subset and the initial video representation are input into the Transformer module, and the output is the anchor box representation with frame-level content enhancement. The text query representation is used as guidance information and multiplied element-by-element with the anchor box subset and the anchor box representation with frame-level content enhancement to obtain the final cross-modal representation. The final cross-modal representation is input into a pre-established feedforward network-based regressor, and the output is the prediction deviation. The initial anchor box boundary is adjusted based on the prediction deviation to obtain the final target moment positioning result.

2. The method for localizing temporal content of cross-modal videos based on contextual memory prompts according to claim 1 is characterized in that: The input video sequence length is T, and the pre-trained visual backbone network is used to extract the representation As the initial video representation, where D is the feature dimension.

3. The method for localizing temporal content of cross-modal videos based on contextual memory prompts according to claim 1, characterized in that: The process of inputting the initial video representation into the pre-built Transformer module: The initial video representation X is fed into the encoder of the Transformer module to generate a video representation enhanced by self-attention. The decoder of the Transformer module takes the randomly initialized query and the video representation enhanced by self-attention as input, and obtains the dynamic context memory dataset M= through cross-attention interaction calculation. M represents the number of behavior instances appearing in the video.

4. The method for localizing temporal content of cross-modal videos based on contextual memory prompts according to claim 1, characterized in that: The process of calculating the predicted behavior category score vector and the time series center coordinates and span corresponding to each behavior instance based on the memory entries in the dynamic contextual memory dataset: Each memory item M m Send it to the linear classifier and after softmax normalization operation, the predicted behavior category score vector is obtained C is the number of behavior categories that appear in the preset video; Each memory item M m It is fed into the time series predictor composed of a feedforward network and a ReLU activation function to generate prediction results corresponding to each behavior instance, including the time series center σ m and span ω m .

5. The method for localizing temporal content of cross-modal videos based on contextual memory prompts according to claim 4 is characterized in that: The discriminative learning criterion and the semantic diversity criterion are constructed based on the behavior category score vector. The discriminative learning criterion is as follows: Where, the behavior distribution p of the constraint prediction is m Approaching one-hot encoding; The semantic diversity learning criteria are as follows: By imposing an L2 norm constraint on the normalized average behavior category prediction vector of all memory entries, the occurrence probabilities of different behavior categories are approached to a uniform distribution.

6. The method for localizing temporal content of cross-modal videos based on contextual memory prompts according to claim 5, characterized in that: The prediction results are used as the basis to construct the time series span diversity learning criteria, as follows: in, represents the calculation of the normalized temporal intersection-over-union ratio between two behavior instances, σ i ,ω i , σ j ,ω j Represent the temporal center and span of two behavior instances i and j respectively.

7. The method for localizing temporal content of cross-modal videos based on contextual memory prompts according to claim 1, characterized in that: The process of reordering the dynamic contextual memory data set according to the order of the temporal center coordinates corresponding to each behavior instance to obtain the memory prompt set: According to the time series center coordinates σ of each behavior instance m The order of the initial context memory set M is reordered to obtain Encode the temporal position Add to the corresponding memory items to build a memory prompt set 8. The method for localizing temporal content of cross-modal videos based on contextual memory prompts according to claim 1, characterized in that: The construction process of the anchor box subset: For the natural language query consisting of L words read in, a pre-trained language model is used to extract the text query representation Count each memory cue The matching score S with the text query representation q m ; Select the top K memory cues with the highest matching scores to form the anchor box subset Memory tips The matching degree with the text query representation q is calculated as follows: Here, σ(·) represents the sigmoid function.

9. The method for localizing temporal content of cross-modal videos based on contextual memory prompts according to claim 1, characterized in that: The anchor box representation U is generated as follows: U=MCA(R,X,X) Where MCA(·) represents the multi-head cross attention operation in the Transformer module decoder, whose anchor box subset R is used as the query input, and the key Key and value Value are derived respectively through the initial video representation X; The text query-guided cross-modal representation generation method is as follows: H k =[R k ⊙q;U k ⊙q] Among them, R and U represent the filtered memory prompt set and content-enhanced anchor box representation, respectively, ⊙ is the element-by-element multiplication calculation, and [·;·] represents the feature concatenation operation.

10. A cross-modal video temporal content localization system based on contextual memory prompts, characterized by: include: A data processing module is used to receive an input video sequence, input the input video sequence into a pre-trained visual backbone network model, output an initial video representation, input the initial video representation into a pre-established Transformer module, and output a dynamic contextual memory dataset; A data sorting module is used to calculate the predicted behavior category score vector and the temporal center coordinates and span corresponding to each behavior instance based on the memory entries in the dynamic contextual memory dataset, and re-sort the dynamic contextual memory dataset according to the order of the temporal center coordinates corresponding to each behavior instance to obtain a memory prompt set; A text extraction module is used to receive input word data, input the input word data into a pre-trained language model, output a text query representation, calculate the matching score between the memory prompt set and the text query representation, and select several memory prompts with the highest matching scores to form an anchor box subset; The feature computation module is used to input the anchor box subset and the initial video representation into the Transformer module, output the anchor box representation with frame-level content enhancement, and use the text query representation as guidance information to perform element-by-element multiplication with the anchor box subset and the frame-level content enhancement anchor box representation to obtain the final cross-modal representation; The content localization module is used to input the final cross-modal representation into a pre-established feedforward network-based regressor, output the prediction deviation, adjust the initial anchor box boundary based on the prediction deviation, and obtain the final target moment localization result.

Images