Training method for event-aware cross-modal video text retrieval model

Through event perception technology, the content and timing alignment of video frames and text is solved, and the problem of ignoring event understanding in video text retrieval in the prior art is improved, and the accuracy and efficiency of cross-modal video text retrieval is improved.

CN118410210BActive Publication Date: 2024-11-19INST OF AUTOMATION CHINESE ACAD OF SCI
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202410845065.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-27
Publication Date
2024-11-19
Estimated Expiration
2044-06-27

AI Technical Summary

Technical Problem

Existing video text retrieval methods rely only on coarse-grained video-level and text comparisons, ignoring the ability to understand video events, resulting in limited retrieval potential.

Method used

A cross-modal video text retrieval model based on event perception is adopted. By obtaining sample video and initial search model, the frame features and video features of video frames are extracted, and the event content and timing alignment are performed are performed, and the model is iteratively trained to improve event understanding ability.

Benefits of technology

It significantly improves the search potential and robustness of cross-modal video text retrieval, can match videos and text more accurately, and improves search capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118410210B_ABST
    Figure CN118410210B_ABST
Patent Text Reader

Abstract

The present invention relates to the technical field of machine learning, and provides a training method for a cross-modal video text retrieval model based on event perception, the method comprising: obtaining a sample video and an initial retrieval model, the sample video comprising a frame description of each video frame; extracting frame features of each video frame and video features of the sample video; performing event content alignment based on the frame features of each video frame and the frame text features of each frame description to determine the event content perception loss; performing event timing alignment based on the video features and the overall text features of the sample video to determine the event timing perception loss; and obtaining a cross-modal video text retrieval model based on the event content perception loss and the event timing perception loss. The method provided by the present invention greatly improves the event understanding ability of the cross-modal video text retrieval model by performing event content alignment at the video frame granularity and event timing alignment at the video granularity, thereby improving the cross-modal video text retrieval ability and robustness.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and in particular to a training method for a cross-modal video text retrieval model based on event perception. Background Art

[0002] Cross-modal video text retrieval is a new type of information retrieval method that has emerged in recent years. Its basic principle is to quickly retrieve the most semantically relevant videos or texts from the database given a text or video as a query. Current work on video text retrieval can be divided into two categories: single-stream models and two-stream models. Among them, the single-stream model can model detailed visual text associations during the reasoning process, with higher accuracy but at the expense of speed; the two-stream model lacks modal interaction, and although it sacrifices accuracy, it can perform efficient retrieval, so it is suitable for actual retrieval systems.

[0003] However, current work on video-text retrieval relies only on coarse-grained comparisons between video and text levels. There may be matching videos or texts that are not retrieved, which means that the retrieval potential of existing methods is limited. Summary of the invention

[0004] The present invention provides a training method for a cross-modal video text retrieval model based on event perception, so as to solve the defect that the cross-modal video text retrieval potential is limited in the prior art.

[0005] The present invention provides a training method for a cross-modal video text retrieval model based on event perception, comprising the following steps:

[0006] Obtaining a sample video and an initial retrieval model, wherein the sample video includes a frame description of each video frame;

[0007] Extracting frame features of each video frame in the sample video and video features of the sample video based on the initial retrieval model;

[0008] Based on the initial retrieval model, event content alignment is performed by applying frame features of each video frame and frame text features of each frame description to determine the event content perception loss of the sample video;

[0009] Based on the initial retrieval model, the event timing alignment is performed by applying the video features and the overall text features of the sample video to determine the event timing perception loss of the sample video;

[0010] Based on the event content perception loss and the event timing perception loss of the sample video, the initial retrieval model is iteratively trained to obtain a cross-modal video text retrieval model.

[0011] According to a training method for an event-aware cross-modal video text retrieval model provided by the present invention, the frame features of each video frame in the sample video and the video features of the sample video are extracted based on the initial retrieval model, including:

[0012] Extracting image block features of all image blocks of each video frame in the sample video based on the initial retrieval model;

[0013] Aggregating the image block features based on the spatiotemporal position information of each image block, and simultaneously obtaining the video features of the sample video and the frame features of each video frame;

[0014] The spatiotemporal position information includes video frame information to which each image block belongs and position information of the video frame to which the image block belongs in the sample video.

[0015] According to a training method for an event-aware cross-modal video text retrieval model provided by the present invention, the step of obtaining a sample video includes:

[0016] Get the initial sample video;

[0017] Selecting a plurality of video frames from the initial sample video, wherein the plurality of video frames are non-overlapping;

[0018] Performing image descriptions on the multiple video frames respectively to obtain frame descriptions of the multiple video frames;

[0019] The sample video is obtained based on the initial sample video and the frame description corresponding to the initial sample video.

[0020] According to a training method for an event-aware cross-modal video text retrieval model provided by the present invention, the sample video is obtained based on the initial sample video and the frame description corresponding to the initial sample video, including:

[0021] A preset number of continuous video frames are randomly selected from a plurality of initial sample videos respectively, and the sample video is obtained based on the preset number of continuous video frames and frame descriptions corresponding to the preset number of continuous video frames.

[0022] According to a training method for an event-aware cross-modal video text retrieval model provided by the present invention, based on the initial retrieval model, event content alignment is performed by applying frame features of each video frame and frame text features of each frame description to determine the event content perception loss of the sample video, including:

[0023] Based on the initial retrieval model, comparative learning is performed on the frame features of each video frame and the frame text features of each frame description to achieve event content alignment and determine the event content perception loss of the sample video.

[0024] According to a training method for an event-aware cross-modal video text retrieval model provided by the present invention, based on the initial retrieval model, applying the video features and the overall text features of the sample video to perform event timing alignment, and determining the event timing perception loss of the sample video, the method includes:

[0025] Based on the initial retrieval model, comparative learning is performed on the video features and the overall text features of the sample video to achieve event timing alignment and determine the event timing perception loss of the sample video;

[0026] The overall text features of the sample video are obtained by sampling based on the frame text features of each video frame of the sample video.

[0027] The present invention also provides a training device for a cross-modal video text retrieval model based on event perception, comprising:

[0028] An acquisition unit, which acquires a sample video and an initial retrieval model, wherein the sample video includes a frame description of each video frame;

[0029] A feature extraction unit, which extracts frame features of each video frame in the sample video and video features of the sample video based on the initial retrieval model;

[0030] An event content alignment unit, based on the initial retrieval model, performs event content alignment by applying frame features of each video frame and frame text features of each frame description to determine the event content perception loss of the sample video;

[0031] An event timing alignment unit, based on the initial retrieval model, performs event timing alignment by applying the video features and the overall text features of the sample video to determine the event timing perception loss of the sample video;

[0032] The training unit iteratively trains the initial retrieval model based on the event content perception loss and the event timing perception loss of the sample video to obtain a cross-modal video text retrieval model.

[0033] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the training method of the event-aware cross-modal video text retrieval model as described in any one of the above is implemented.

[0034] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a training method for an event-aware cross-modal video text retrieval model as described in any one of the above.

[0035] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements a training method for an event-aware cross-modal video text retrieval model as described in any one of the above.

[0036] The training method of the event-aware cross-modal video text retrieval model provided by the present invention obtains sample videos and an initial retrieval model, extracts frame features and video features of video frames, and performs event content alignment based on the frame features of each video frame and the frame text features described by each frame, so as to perform event content alignment at the video frame granularity. In addition, event timing alignment is performed based on the video features and the overall text features of the sample video, so as to perform event timing alignment at the video granularity, so that the event understanding ability of the cross-modal video text retrieval model is greatly improved, thereby improving the cross-modal video text retrieval ability and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 It is one of the flow charts of the training method of the event-aware cross-modal video text retrieval model provided by the present invention;

[0039] Figure 2 This is the second flow chart of the training method of the event-aware cross-modal video text retrieval model provided by the present invention;

[0040] Figure 3 It is a structural schematic diagram of a training device for an event-aware cross-modal video text retrieval model provided by the present invention;

[0041] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] Unlike traditional retrieval that relies solely on a single modality, such as text retrieval, image retrieval, and video retrieval, cross-modal video text retrieval can effectively achieve alignment at the text and visual semantic levels, and can implement cross-modal search for different user input forms, bringing users different search experiences and information gain. For example, on short video platforms, previous retrieval methods relied entirely on text matching between the text entered by the user and the title of the video itself on the platform. One disadvantage of this method is that if the video title is of poor quality or it is difficult to cover the entire content of the video, the user's desired video cannot be retrieved. Cross-modal video text retrieval can align the user's text search directly to the actual content of the video. Even if the video does not have a high-quality title, it can be matched through high-level visual semantics.

[0044] Thanks to the large-scale pre-training datasets crawled from the Internet, the video text retrieval model based on the dual-stream encoder structure has made rapid progress in recent years. Its basic structure includes a video encoder and a text encoder, where the video encoder is used to encode the spatiotemporal visual features in the video, and the text encoder is used to encode the text features. The visual and text features are then used to calculate the cosine similarity to obtain a cross-modal similarity score between video and text pairs, and cross-modal video text retrieval is achieved by comparing the cross-modal similarity scores between different video and text pairs. The dual-stream retrieval model will independently encode information of different modalities without any interaction between the modalities. Therefore, it supports offline extraction and storage of sample features of the entire database to be retrieved. When a query arrives, it only needs to calculate the cosine similarity between the features to achieve fast retrieval. Therefore, it has high practicality in actual application scenarios.

[0045] However, the current work of video text retrieval based on the two-stream model only relies on the coarse-grained comparison between the video level and the text, ignoring the ability to understand video events. There may be matching videos or texts, but they are not retrieved, that is, the retrieval potential of the existing methods is limited. To address this problem, the present invention provides a training method for a cross-modal video text retrieval model based on event perception, which realizes cross-modal video text retrieval focusing on events in the video, so as to improve the retrieval potential of cross-modal video text retrieval. Figure 1 is one of the flow charts of the training method of the event-aware cross-modal video text retrieval model provided by the present invention, such as Figure 1 As shown, the method includes the following:

[0046] Step 110, obtaining a sample video and an initial retrieval model, wherein the sample video includes a frame description of each video frame;

[0047] The frame description here refers to the event information used to describe the video frame, and the event information here may include event content and event timing information. It should be noted that the frame description used to train the cross-modal video text retrieval model in the prior art generally only focuses on the background information in the video frame, and lacks the text description of the event content and event timing in the video frame. Therefore, the frame description here may include the event content and event timing information in the video frame. It is understandable that when describing the event content of an event, a description of the background in the image is usually included, such as a description of the location, person, and time of the event, and the description of the location, person, and time of the event usually constitutes the background of the video frame. In addition, the initial retrieval model here can be a cross-modal video text retrieval model based on a dual-stream encoder structure, and the basic structure includes a video encoder and a text encoder.

[0048] Specifically, sample videos can be obtained from existing video-text pair datasets, such as from Webvid datasets. The sample videos here also include frame descriptions of multiple video frames. The frame descriptions of each video frame here can be the description text in the dataset, or manually annotated text, or text generated by an image description model.

[0049] Step 120, extracting frame features of each video frame in the sample video and video features of the sample video based on the initial retrieval model;

[0050] Here, each video frame in the sample video can be a non-overlapping video frame randomly selected from the sample video. The frame features here can be used to reflect the visual information in the video frame image, especially focusing on reflecting the event content in the video frame and the time when a single event occurs. In addition, the video features here can be used to reflect the event content in the entire sample video and the timing logic between each event. It should be noted that a sample video can contain multiple events, so to achieve fast and accurate video text retrieval requires the model to have a comprehensive understanding of the events occurring at different times in the video and the timing logic between different video events.

[0051] Specifically, the video encoder in the initial retrieval model can perform feature encoding on multiple video frames of the sample video respectively to obtain the frame features of each video frame. Similarly, the video encoder can perform feature encoding on the sample video as a whole to obtain the video features of the sample video. Alternatively, the sample video can be feature encoded as a whole to obtain both the video features and the frame features.

[0052] Step 130, based on the initial retrieval model, aligning event content by applying frame features of each video frame and frame text features of each frame description, and determining the event content perception loss of the sample video;

[0053] Here, the frame text feature can be used to reflect the semantic information of the frame description of each video frame, which can include the events occurring in the video frame. In addition, the event content-aware loss here can represent the difference between the semantic information reflected by the frame feature at the video frame granularity and the semantic information reflected by the text feature of the frame description.

[0054] Specifically, the initial retrieval model can be used to align the event content of the frame features of each video frame and the frame text features of the frame description corresponding to each video frame at the granularity of the video frame. For example, the semantic loss between the frame features of each video frame and the frame description corresponding to each video frame can be calculated respectively, and the event content perception loss of the sample video can be obtained based on the semantic loss. It can be understood that when the event content perception loss at the granularity of the video frame is smaller, it means that the event content and event timing information contained in the frame features extracted based on the video frame are more consistent with the frame description of the video frame itself, that is, the accuracy and comprehensiveness of the extracted frame features are higher; when the event content perception loss at the granularity of the video frame is larger, it means that the event content and event timing information contained in the frame features extracted based on the video frame are less consistent with the frame description of the video frame itself, that is, the accuracy and comprehensiveness of the extracted frame features are lower.

[0055] It should be noted that compared with the prior art, the video features of the entire video are compared with the description text, and only the information constituting the video screen is focused on during the comparison, such as the static information such as the background and people in the video screen, that is, the text video matching based on the image composition is performed from the video granularity. It can be understood that when performing cross-modal video text search, it is easy for users to search by using the events that occurred as the search text. The model training method provided by the embodiment of the present invention, in the model training stage, compares the frame features of each video frame with the corresponding frame description from the video frame granularity, and the frame features and frame description contain descriptions of the event content and event timing that occurred in the image, thereby improving the search potential of the cross-modal video text retrieval model, and can greatly improve the search capability of the model, so that more practical and extensive video text search can be achieved in the application stage of the cross-modal video text retrieval model.

[0056] Step 140, based on the initial retrieval model, applying the video features and the overall text features of the sample video to perform event timing alignment, and determining the event timing perception loss of the sample video;

[0057] Here, the overall text feature can reflect the event content contained in the entire sample video and the temporal logic between events. In addition, the event temporal perception loss here can represent the difference between the event temporal logic reflected by the sample video and the event temporal logic reflected by the overall text feature.

[0058] Specifically, the initial retrieval model can be used to align the event timing logic of the video features and the overall text features of the sample video to determine the event timing perception loss of the sample video. For example, the event timing difference between the video features and the overall text features can be calculated to obtain the event timing perception loss. It can be understood that when the event timing perception loss is smaller, it means that the event timing logic information between the video features extracted based on the sample video and the overall text features extracted based on the overall description of the video is more consistent, and the event timing information contained in the video features extracted based on the sample video is more accurate and comprehensive; when the event timing perception loss is larger, it means that the event timing logic information between the video features extracted based on the sample video and the overall text features extracted based on the overall description of the video is more inconsistent, and the event timing information contained in the video features extracted based on the sample video is less accurate and comprehensive.

[0059] It should be noted that compared with the prior art which only performs video-text comparison on background information in the video, the method provided in the embodiment of the present invention improves the retrieval capability of the cross-modal video-text retrieval model by performing cross-modal comparison of video and text from the dimension of event timing.

[0060] In addition, it should be noted that compared with the cross-modal video-text comparison based on a single granularity in the prior art, the method provided by the embodiment of the present invention compares the event content from the video frame granularity and the event timing from the complete video granularity, that is, compares the video and text from multiple granularities, thereby achieving a more comprehensive and accurate event information comparison, thereby improving the retrieval capability of the cross-modal video text retrieval model.

[0061] Step 150, based on the event content perception loss and the event timing perception loss of the sample video, iteratively train the initial retrieval model to obtain a cross-modal video text retrieval model.

[0062] Specifically, the model parameters used for encoding frame features in the initial retrieval model can be adjusted by perceiving the event content loss of the sample video, so that the extracted frame features can more comprehensively and accurately reflect the event information in the video frame, that is, the semantic similarity between the extracted frame features and the frame text features of the frame description corresponding to the video frame is higher. Similarly, the model parameters used for encoding video features in the initial retrieval model can be adjusted by perceiving the event timing loss of the sample video, so that the extracted video features can more comprehensively and accurately reflect the event timing information in the sample video, that is, the similarity between the extracted video features and the overall text features in event timing is higher.

[0063] The method provided by the embodiment of the present invention obtains sample videos and an initial retrieval model, extracts frame features and video features of video frames, and performs event content alignment based on the frame features of each video frame and the frame text features described by each frame, so as to perform event content alignment at the video frame granularity. In addition, event timing alignment is performed based on the video features and the overall text features of the sample video, so as to perform event timing alignment at the video granularity, so that the event understanding ability of the cross-modal video text retrieval model is greatly improved, thereby improving the cross-modal video text retrieval ability and robustness.

[0064] Based on any of the above embodiments, step 120 includes:

[0065] Extracting frame features of each video frame based on the initial retrieval model;

[0066] Aggregating frame features of the video frames based on the spatiotemporal position information of the video frames to obtain video features of the sample video;

[0067] The spatiotemporal position information is used to reflect the position of the video frame in the sample video.

[0068] Specifically, a predefined random feature of a randomly initialized image block [CLS] Token can be inserted before the image block features of all image block Patch Tokens of each video frame in the sample video. The random feature can be used to interact with the image block features of each image block under the corresponding video frame to extract the visual features of the video frame that are of interest in the retrieval task. It should be noted that since a random feature of a randomly initialized image block is introduced for each video frame, but the number of video frames is much smaller than the number of image blocks contained in a single video frame, the additional computational cost brought by the introduced random feature can be ignored.

[0069] Next, the video features of the sample video can be obtained by aggregating the image features of the image blocks of all the video frames in the sample image. In detail, the image block representations of all the image blocks under the sample video can be aggregated according to the spatiotemporal position information of each image block, the video frame information to which the image block belongs, and the position information of the video frame in the sample video to obtain an aggregated overall representation. For example, the random representations of the randomly initialized image blocks inserted into each video frame can be connected at the beginning to obtain the frame representations of each video frame, and the video representation of the sample video can be obtained by aggregating the aggregated frame representations of each video frame. The aggregated overall representation can be expressed as: .in, Indicates The overall representation of the aggregate of sample videos; Indicates Video representation of sample videos; Indicates Sample Videos Image patch representation of video frames; Indicates Sample Videos Random representation of randomly initialized image patches of video frames; Indicates the total number of video frames in the sample video.

[0070] Furthermore, the aggregated overall representation can be input into a video encoder to convert the overall visual features of the sample video. Finally, the video features of the sample video and the frame features of each video frame can be directly extracted from the overall visual features of the sample video. For example, the video features of the sample video and the frame features of each video frame can be obtained by the following formula:

[0071]

[0072] In the formula, Indicates the first The overall visual features of the sample videos; Represents a video encoder; Represents the first Video features of sample videos; Represents the first Sample Videos Image block features of video frames; Represents the first Sample Videos Random features of randomly initialized image patches of video frames; Indicates the total number of video frames in the sample video.

[0073] The method provided by the embodiment of the present invention aggregates the image block features of each image block in units of video frames and the entire video through the spatiotemporal position information of each image block under the sample video. The video features of the sample video and the frame features of each video frame can be directly obtained from the aggregated features, thereby achieving simultaneous encoding of video features and frame features without consuming excessive computing costs, thereby improving the encoding performance of the cross-modal video text retrieval model, and further improving the retrieval efficiency of cross-modal video text retrieval.

[0074] Considering that mainstream pre-training datasets such as WebVid lack specific and complete text descriptions of the content of events in the video, the quality of the training sample data is poor, which restricts the model's ability to understand the events in the video. To address this problem, based on any of the above embodiments, in step 110, obtaining the sample video includes:

[0075] Get the initial sample video;

[0076] Selecting a plurality of video frames from the initial sample video, wherein the plurality of video frames are non-overlapping;

[0077] Performing image descriptions on the multiple video frames respectively to obtain frame descriptions of the multiple video frames;

[0078] The sample video is obtained based on the initial sample video and the frame description corresponding to the initial sample video.

[0079] Specifically, an initial sample video may be obtained from an existing data set, such as a WebVid data set. Then, the obtained initial sample video may be divided into a plurality of non-overlapping video segments, and a video frame may be sampled from each of the plurality of non-overlapping video segments, thereby obtaining a plurality of non-overlapping video frames.

[0080] Next, the event content can be enhanced for the multiple video frames obtained, and image description can be performed on each video frame to obtain a frame description of each video frame. For example, the video frames can be input into an image description model respectively, and the event content in the image can be described by the image description model to output a frame description of each video frame. It should be noted that in order to generate more diverse frame descriptions to obtain a cross-modal video text model with strong generalization ability, a non-deterministic generation strategy can be used to generate frame descriptions of each video frame.

[0081] Furthermore, multiple video frames sampled from a single initial sample video may be synthesized into a sample video, and the video description of the sample video may be obtained through the corresponding frame descriptions of the multiple video frames. Alternatively, multiple video frames sampled from multiple initial sample videos may be synthesized into a single sample video, and the video description of the sample video may be obtained through the corresponding frame descriptions. Thus, a sample video and sample data including frame descriptions of video frames and video descriptions may be obtained.

[0082] For example, the frame description after event content enhancement can be: frame description of the first frame: cars driving on city streets at night, frame description of the second frame: a map crossing the street, frame description of the Nth frame: several people walking on the street at night. It should be noted that there is a prior art that creates a title for each video, simulating videos with relevant text information in real scenes for retrieval, such as adding titles and tags. Alternatively, a fine-tuned large language model is used to add narration to long videos of the sample data set to assist in training. However, the purpose of the above work is to expand general text information, rather than to specifically increase event information in pre-training data.

[0083] The method provided by the embodiment of the present invention randomly selects multiple non-overlapping video frames from the initial sample video, and performs event content enhancement on the multiple video frames to obtain frame descriptions containing event content information, so as to enrich the visual event content information of the sample video and the event content information of the frame description.

[0084] Based on any of the above embodiments, obtaining the sample video based on the initial sample video and the frame description corresponding to the initial sample video includes:

[0085] A preset number of continuous video frames are randomly selected from a plurality of initial sample videos respectively, and the sample video is obtained based on the preset number of continuous video frames and frame descriptions corresponding to the preset number of continuous video frames.

[0086] Specifically, multiple initial sample videos can be randomly selected from an initial sample data set in a training batch, for example, two initial sample videos. Then, a preset number of video frames can be randomly sampled from the initial sample videos having multiple continuous video segments, for example, a preset number of video frames can be sampled from the first initial sample video. consecutive video frames, sampled from the second initial sample video The number of consecutive video frames sampled here can be based on the total number of frames of the sample video. Confirmation means satisfaction Finally, the preset number of continuous video frames sampled from the multiple initial sample videos can be synthesized into a sample video, and the frame descriptions corresponding to the preset number of continuous video frames can be connected to obtain the video description of the sample video.

[0087] It should be noted that the continuous video frames in multiple initial sample videos can represent an event that occurs at a certain time, and sampling a preset number of continuous video frames from different initial videos can be considered as combining multiple events that occur at different times in the sample video. Therefore, by randomly selecting preset number of continuous video frames from multiple initial sample videos, the initial sample video is synthesized to enrich the temporal changes between various events in the sample video, so that the trained cross-modal video text retrieval model can learn the temporal logic between multiple events in the sample video, and then in the application process, it can enhance the ability to understand the event temporal logic of the video to be retrieved, and obtain ideal retrieval results in the retrieval of event timing.

[0088] Based on any of the above embodiments, step 130 includes:

[0089] Based on the initial retrieval model, comparative learning is performed on the frame features of each video frame and the frame text features of each frame description to achieve event content alignment and determine the event content perception loss of the sample video.

[0090] Specifically, the text encoder in the initial retrieval model can be used to encode the text of each frame description to obtain the frame text features of each frame description. Then, the frame features of each video frame and the frame text features of each frame description can be compared and learned to achieve event content alignment of the video frames and determine the event content perception loss of the sample video. For example, the event content perception loss can be calculated by the following formula, as shown in the following formula:

[0091]

[0092]

[0093] In the formula, Indicates Sample Videos Event content-aware loss for frame-by-frame vision-to-text; represents the temperature hyperparameter; Indicates the size of the training batch; Indicates Sample Videos Frame features of video frames; Indicates Sample Videos Frame text features of the frame description corresponding to the frame video frame; Indicates Sample Videos The frame text features of the frame description corresponding to the frame video frame, where Represents any sample video in the batch , can be used with The same, but also different; Indicates video frames; in addition, Indicates Sample Videos Event content-aware loss for frame-by-frame text-to-vision; Indicates Sample Videos Frame features of video frames; Represents event content-aware loss.

[0094] Based on any of the above embodiments, the step of applying the video features and the overall text features of the sample video to perform event timing alignment based on the initial retrieval model to determine the event timing perception loss of the sample video includes:

[0095] Based on the initial retrieval model, comparative learning is performed on the video features and the overall text features of the sample video to achieve event timing alignment and determine the event timing perception loss of the sample video;

[0096] The overall text features of the sample video are obtained by sampling based on the frame text features of each video frame of the sample video.

[0097] Specifically, the text encoder in the initial retrieval model can be used to sample the frame features of each video frame of the sample video to obtain the overall text features of the sample video. Then, the initial retrieval model can be used to compare and learn the video features and the overall text features of the sample video to achieve the logical alignment of the event timing of multiple events and obtain the event timing perception loss of the sample video. The event timing perception loss here can be calculated by the following formula, as shown in the following formula:

[0098]

[0099]

[0100]

[0101] In the formula, Indicates The event-time-aware vision-to-text loss for synthesized sample videos; Indicates Video features of synthesized sample videos; Indicates The overall text features of the sample videos; Indicates ( Can be used with The overall text features of the synthesized sample videos (the same or different); Indicates A text-to-visual event-timing-aware loss for synthesized sample videos; Indicates Video features of synthesized sample videos; Represents event timing-aware loss.

[0102] Based on any of the above embodiments, Figure 2 FIG. 2 is a flow chart of the training method of the event-aware cross-modal video text retrieval model provided by the present invention. Figure 2As shown in Figure 1, the model structure of the event-aware cross-modal video text retrieval model is a dual-stream encoder structure, which includes a multi-granularity video encoder and a text encoder. The training method of the cross-modal video text retrieval model is divided into two parts. The first part is the enhancement and learning of event content, and the second part is the enhancement and learning of event timing.

[0103] In the process of event content enhancement and learning, first, multiple video frames are described by an image descriptor to achieve event content enhancement of frame description. Then, each video frame is divided into multiple frame image blocks, and a randomly initialized frame special identifier is added to the beginning of each video frame. In addition, the frame descriptions of each video frame can be spliced ​​to obtain a video description of the sample video. Then, the multiple frame image blocks of the sample video can be feature encoded by a multi-granularity video encoder to obtain the frame features of each video frame. And the video features of the sample video Similarly, the video description of the sample video and the frame description of each video frame can be feature encoded by the text encoder to obtain the overall text feature of the sample video and frame text features of each video frame .in, Indicates the need for sample videos; Indicates the frame number of the video frame. Furthermore, the event content learning can be performed by aligning the frame features of each video frame with the frame text features described by each frame, and the event content perception loss of the sample video can be obtained.

[0104] In the event timing enhancement and learning stage, first, a preset number of continuous video frames and frame descriptions corresponding to the preset number of continuous video frames can be randomly extracted from multiple initial sample videos (video #1, video #2). Then, all the extracted preset number of continuous video frames and the corresponding frame descriptions of the preset number of continuous video frames are used as sample videos, and the video description of the synthesized sample video is obtained by splicing the frame descriptions corresponding to the preset number of continuous video frames. Then, the synthesized sample video is encoded by a multi-granularity video encoder to obtain the video features of the synthesized sample video. Similarly, the synthesized video description is feature encoded by the text encoder to obtain the synthesized text features of the synthesized sample video . Further, by combining the video features of the synthesized sample video and synthetic text features Alignment is performed for event timing learning, and an event timing-aware loss is obtained for sample videos.

[0105] Therefore, the parameters of the cross-modal video text retrieval model can be iterated based on the event content perception loss and event timing perception loss of the sample video to complete the training process of the cross-modal video text retrieval model. It should be noted that in addition to event content learning and event timing learning, the commonly used video-level contrastive learning can also be used to align the sample videos crawled from the network and the corresponding frame descriptions.

[0106] In order to verify the technical effect of the training method of the event-aware cross-modal video text retrieval model provided by an embodiment of the present invention, Table 1 shows the results of zero-sample text to video retrieval of multiple methods on four mainstream downstream video text retrieval datasets (MSRVTT, DiDeMo, LSMDC and MSVD), where higher R@k (k=1, 5,10) and lower MedR (Median Rank) indicate better performance.

[0107] Table 1

[0108]

[0109] In the table, the MSRVTT dataset is a large video description dataset used to bridge video and language; the DiDeMo dataset uses natural language to locate moments in videos. The dataset contains a large number of video-sentence pairs, each sentence describes a specific moment or event in the video; the LSMDC (Large Scale Movie Description Challenge) dataset contains a large number of movie clips and corresponding natural language descriptions; the MSVD (Microsoft Video Description Corpus) dataset contains a large number of video clips and corresponding natural language descriptions, collecting highly parallel data to evaluate interpretation. Among them, the methods used for comparison include single-stream encoders: TACO (TACo: Token-Aware Cascade Contrastive Learning for Video-Text Alignment), VIOLET (VIOLET: End-to-End Video-Language Transformers with Masked Visual-token Modeling), ALPRO (Align and Prompt: Video-and-Language Pre-training with Entity prompts), Rap (Redundancy-aware Video-language Pre-training for Text-Video Retrieval), Clover (Towards A Unified Video-Language Alignment and Fusion Model), TW-BERT (Learning Trajectory-Word Alignments for Video-Language) and other methods. Tasks, learning track-word alignment for video language tasks).In addition, the two-stream encoder class methods include: Frozen (Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval), LaT (LaT: Latent Translation with Cycle-Consistency for Video-Text Retrieval), RegionL (Video-Text Pre-training with Learned Regions), OA-Trans (Object-aware Video-language Pre-training for Retrieval), MCQ (Bridging Video-Text Retrieval with Multiple Choice Questions), and Miles (Visual BERT Pre-training with Injected Language Semantics for Video-text Retrieval).

[0110] It can be clearly seen from Table 1 that the method proposed in the embodiment of the present invention greatly surpasses the previous dual-stream retrieval model. Specifically, the event-aware cross-modal video text retrieval model provided in the embodiment of the present invention is 2.0%, 5.9% and 5.4% higher than the previous optimal method Miles in R@1, R@5 and R@10 of the MSRVTT dataset, respectively. Its advantage is even more obvious on the DiDeMo dataset with rich temporal information, with R@1, R@5 and R@10 respectively 5.5%, 8.6% and 5.3% higher than the previous optimal method, which to a certain extent reflects that the model provided by the embodiment of the present invention can better capture the event temporal logic. In addition, even compared with the recent single-stream retrieval method, the performance of the event-aware cross-modal video text retrieval model provided in the embodiment of the present invention is still superior, although the joint encoder method is generally considered to have better performance at the expense of efficiency.

[0111] Furthermore, Table 2 shows the comparison of text-to-video retrieval results of different methods after fine-tuning on MSRVTT, DiDeMo, LSMDC and MSVD, where the evaluation setting is the same as that of zero-shot text-to-video retrieval, as shown in the following table:

[0112] Table 2

[0113]

[0114] As shown in Table 2, the conclusions drawn from zero-shot retrieval still hold after fine-tuning.

[0115] In order to verify the generalization ability of the method, zero-sample action recognition evaluation can be performed on the video action recognition datasets UCF101 (UCF101: ADataset of 101 Human Actions Classes From Videos in The Wild) and HMDB51 (HMDB: A large video database for human motion recognition). To this end, the video category can be converted into a text description with a template prompt, and 16 frames of the video are uniformly sampled as input. The results are shown in Table 3, which is a comparison of the zero-sample video action recognition results of different methods on UCF101 and HMDB51, as shown in the following table:

[0116] Table 3

[0117]

[0118] As shown in Table 3, compared with Miles, the average top-1 classification accuracy (the "Average" column in the table) of the method provided by the embodiment of the present invention on UCF101 and HMDB51 is improved by 8.3% and 0.5% respectively. This can be attributed to the fact that actions themselves exist in video events, so the event enhancement and learning method provided by the embodiment of the present invention naturally enhances the understanding of video actions.

[0119] In addition, the goal of the multi-event video text retrieval task is to find all events related to a given video. This task effectively reflects the model's comprehensive perception of all video events. Zero-shot evaluation can be performed on the ActivityNet (ActivityNet: A Large-Scale Video Benchmark for HumanActivity Understanding) dataset by uniformly sampling 16 video frames as input. Thus, the zero-shot multi-event video text retrieval results of different methods on ActivityNet can be compared. Table 4 shows the comparison of zero-shot multi-event video text retrieval results of different methods on ActivityNet, as shown in the following table:

[0120] Table 4

[0121]

[0122] The results are shown in Table 4. It can be clearly seen that in the setting of video-to-text retrieval, the method provided by the embodiment of the present invention significantly outperforms other methods in all three indicators (Recall@k-Average / One-Hit / All-Hit). At the same time, the results of the setting of text-to-video retrieval show a similar trend. This verifies that the event content learning proposed in the present invention is indeed conducive to improving the model's ability to capture all events in the video.

[0123] Each sample in the Test of Time task contains a video and two video descriptions with the order of events reversed, and requires the model to select the correct subtitle for the video. This task aims to test the model's ability to understand the temporal order of video events, and can be evaluated with zero samples on the ActivityNet dataset provided by the author.

[0124] Table 5 shows the comparison of zero-shot evaluation results of different methods on the ActivityNet dataset of the Test of Time task, as shown in the following table:

[0125] Table 5

[0126]

[0127] The results in Table 5 show that this is a rather challenging task, as the performance of the previous two-stream retrieval model is close to random guessing (50.0%). It can be seen that the method provided by the embodiment of the present invention has a significant advantage in determining the temporal relationship of events, showing 61.6% temporal consistency in all samples, leading the second-place method by 11.3%. This shows that event timing learning can effectively enhance the model's ability to understand event timing logic.

[0128] Finally, an ablation experiment can also be performed on the effectiveness and compatibility of different components in the method proposed in the present invention. Table 6 shows the ablation experiment on the effectiveness and compatibility of different components in the method proposed in the present invention, as shown in the following table:

[0129] Table 6

[0130]

[0131] From the results in Table 6, we can draw the following conclusions: 1) Both Models B and C outperform the baseline Model A trained using only video-level contrastive learning. Specifically, Model B, which combines event content learning, shows greater improvements on MSRVTT, while Model C, which combines event timing learning, shows more significant gains on DiDeMo. 2) In addition, Model D, which combines event content learning and event timing learning, achieves further performance improvements over Models B and C on both datasets, indicating that event content learning and event timing learning are mutually beneficial and compatible.

[0132] Based on any of the above embodiments, Figure 3 is a structural schematic diagram of a training device for an event-aware cross-modal video text retrieval model provided by the present invention, such as Figure 3 As shown, the device comprises:

[0133] An acquisition unit 310 acquires a sample video and an initial retrieval model, wherein the sample video includes a frame description of each video frame;

[0134] A feature extraction unit 320 extracts frame features of each video frame in the sample video and video features of the sample video based on the initial retrieval model;

[0135] An event content alignment unit 330, based on the initial retrieval model, performs event content alignment by applying frame features of each video frame and frame text features of each frame description to determine the event content perception loss of the sample video;

[0136] An event timing alignment unit 340, based on the initial retrieval model, performs event timing alignment by applying the video features and the overall text features of the sample video to determine the event timing perception loss of the sample video;

[0137] The training unit 350 iteratively trains the initial retrieval model based on the event content perception loss and the event timing perception loss of the sample video to obtain a cross-modal video text retrieval model.

[0138] The device provided by the embodiment of the present invention obtains sample videos and an initial retrieval model, extracts frame features and video features of video frames, and performs event content alignment based on the frame features of each video frame and the frame text features described by each frame, so as to perform event content alignment at the video frame granularity. In addition, event timing alignment is performed based on the video features and the overall text features of the sample video, so as to perform event timing alignment at the video granularity, so that the event understanding ability of the cross-modal video text retrieval model is greatly improved, thereby improving the cross-modal video text retrieval ability and robustness.

[0139] Based on any of the above embodiments, the feature extraction unit is specifically used for:

[0140] Extracting image block features of all image blocks of each video frame in the sample video based on the initial retrieval model;

[0141] Aggregating the image block features based on the spatiotemporal position information of each image block, and simultaneously obtaining the video features of the sample video and the frame features of each video frame;

[0142] The spatiotemporal position information includes video frame information to which each image block belongs and position information of the video frame to which the image block belongs in the sample video.

[0143] Based on any of the above embodiments, the acquiring unit is specifically used for:

[0144] Get the initial sample video;

[0145] Selecting a plurality of video frames from the initial sample video, wherein the plurality of video frames are non-overlapping;

[0146] Performing image descriptions on the multiple video frames respectively to obtain frame descriptions of the multiple video frames;

[0147] The sample video is obtained based on the initial sample video and the frame description corresponding to the initial sample video.

[0148] Based on any of the above embodiments, the acquiring unit is further specifically configured to:

[0149] A preset number of continuous video frames are randomly selected from a plurality of initial sample videos respectively, and the sample video is obtained based on the preset number of continuous video frames and frame descriptions corresponding to the preset number of continuous video frames.

[0150] Based on any of the above embodiments, the event content alignment unit is specifically used for:

[0151] Based on the initial retrieval model, comparative learning is performed on the frame features of each video frame and the frame text features of each frame description to achieve event content alignment and determine the event content perception loss of the sample video.

[0152] Based on any of the above embodiments, the event timing alignment unit is specifically used for:

[0153] Based on the initial retrieval model, comparative learning is performed on the video features and the overall text features of the sample video to achieve event timing alignment and determine the event timing perception loss of the sample video;

[0154] The overall text features of the sample video are obtained by sampling based on the frame text features of each video frame of the sample video.

[0155] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor (processor) 410 , a communication interface (Communications Interface) 420 , a memory (memory) 430 and a communication bus 440 , wherein the processor 410 , the communication interface 420 , and the memory 430 communicate with each other through the communication bus 440 . The processor 410 can call the logic instructions in the memory 430 to execute a training method for an event-aware cross-modal video text retrieval model, the method comprising: obtaining a sample video and an initial retrieval model, the sample video comprising a frame description of each video frame; extracting frame features of each video frame in the sample video and video features of the sample video based on the initial retrieval model; based on the initial retrieval model, applying the frame features of each video frame and the frame text features of each frame description to perform event content alignment to determine the event content-aware loss of the sample video; based on the initial retrieval model, applying the video features and the overall text features of the sample video to perform event timing alignment to determine the event timing-aware loss of the sample video; based on the event content-aware loss of the sample video and the event timing-aware loss, iteratively training the initial retrieval model to obtain a cross-modal video text retrieval model.

[0156] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0157] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the training method of the event-aware cross-modal video text retrieval model provided by the above-mentioned methods, and the method includes: obtaining a sample video and an initial retrieval model, wherein the sample video includes a frame description of each video frame; extracting frame features of each video frame in the sample video and video features of the sample video based on the initial retrieval model; based on the initial retrieval model, aligning event content by applying the frame features of each video frame and the frame text features of each frame description to determine the event content perception loss of the sample video; based on the initial retrieval model, aligning event timing by applying the video features and the overall text features of the sample video to determine the event timing perception loss of the sample video; based on the event content perception loss of the sample video and the event timing perception loss, iteratively training the initial retrieval model to obtain a cross-modal video text retrieval model.

[0158] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the training method of the event-aware cross-modal video text retrieval model provided by the above-mentioned methods, the method comprising: obtaining a sample video and an initial retrieval model, the sample video comprising a frame description of each video frame; extracting frame features of each video frame in the sample video and video features of the sample video based on the initial retrieval model; based on the initial retrieval model, aligning event content by applying the frame features of each video frame and the frame text features of each frame description, and determining the event content perception loss of the sample video; based on the initial retrieval model, aligning event timing by applying the video features and the overall text features of the sample video, and determining the event timing perception loss of the sample video; based on the event content perception loss of the sample video and the event timing perception loss, iteratively training the initial retrieval model to obtain a cross-modal video text retrieval model.

[0159] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0160] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for an event-aware cross-modal video text retrieval model, characterized in that: include: Obtaining a sample video and an initial retrieval model, wherein the sample video includes a frame description of each video frame; Extracting frame features of each video frame in the sample video and video features of the sample video based on the initial retrieval model; Based on the initial retrieval model, comparative learning is performed on the frame features of each video frame and the frame text features of each frame description to achieve event content alignment and determine the event content perception loss of the sample video; Based on the initial retrieval model, the video features and the overall text features of the sample video are compared and learned to achieve event timing alignment and determine the event timing perception loss of the sample video; the overall text features of the sample video are obtained by sampling based on the frame text features of each video frame of the sample video; Iteratively training the initial retrieval model based on the event content perception loss and the event timing perception loss of the sample video to obtain a cross-modal video text retrieval model; The event content perception loss is calculated by the following formula: ; ; ; in, Indicates Sample Videos Event content-aware loss for frame-by-frame vision-to-text; represents the temperature hyperparameter; Indicates the size of the training batch; Indicates Sample Videos Frame features of video frames; Indicates Sample Videos Frame text features of the frame description corresponding to the frame video frame; Indicates Sample Videos The frame text features of the frame description corresponding to the frame video frame, where Represents any sample video in the batch ; Indicates video frames; in addition, Indicates Sample Videos Event content-aware loss for text-to-vision frame-by-frame video; Indicates Sample Videos Frame features of video frames; Represents event content-aware loss; The event timing perception loss is calculated by the following formula: ; ; ; in, Indicates The event-time-aware vision-to-text loss for synthesized sample videos; Indicates Video features of synthesized sample videos; Indicates The overall text features of the sample videos; Indicates The overall text features of the synthesized sample videos; Indicates A text-to-visual event-timing-aware loss for synthesized sample videos; Indicates Video features of synthesized sample videos; Represents event timing-aware loss.

2. The training method of the event-aware cross-modal video text retrieval model according to claim 1 is characterized in that: The extracting the frame features of each video frame in the sample video and the video features of the sample video based on the initial retrieval model includes: Extracting image block features of all image blocks of each video frame in the sample video based on the initial retrieval model; Aggregating the image block features based on the spatiotemporal position information of each image block, and simultaneously obtaining the video features of the sample video and the frame features of each video frame; The spatiotemporal position information includes video frame information to which each image block belongs and position information of the video frame to which the image block belongs in the sample video.

3. The training method of the event-aware cross-modal video text retrieval model according to claim 1 is characterized in that: The obtaining of the sample video comprises: Get the initial sample video; Selecting a plurality of video frames from the initial sample video, wherein the plurality of video frames are non-overlapping; Performing image descriptions on the multiple video frames respectively to obtain frame descriptions of the multiple video frames; The sample video is obtained based on the initial sample video and the frame description corresponding to the initial sample video.

4. The training method of the event-aware cross-modal video text retrieval model according to claim 3 is characterized in that: The obtaining the sample video based on the initial sample video and the frame description corresponding to the initial sample video includes: A preset number of continuous video frames are randomly selected from a plurality of initial sample videos respectively, and the sample video is obtained based on the preset number of continuous video frames and frame descriptions corresponding to the preset number of continuous video frames.

5. A training device for an event-aware cross-modal video text retrieval model, characterized in that: include: An acquisition unit, which acquires a sample video and an initial retrieval model, wherein the sample video includes a frame description of each video frame; A feature extraction unit, which extracts frame features of each video frame in the sample video and video features of the sample video based on the initial retrieval model; An event content alignment unit, based on the initial retrieval model, performs comparative learning on the frame features of each video frame and the frame text features of each frame description to achieve event content alignment and determine the event content perception loss of the sample video; An event timing alignment unit, based on the initial retrieval model, performs comparative learning on the video features and the overall text features of the sample video to achieve event timing alignment and determine the event timing perception loss of the sample video; the overall text features of the sample video are obtained by sampling based on the frame text features of each video frame of the sample video; A training unit, which iteratively trains the initial retrieval model based on the event content perception loss and the event timing perception loss of the sample video to obtain a cross-modal video text retrieval model; The event content perception loss is calculated by the following formula: ; ; ; in, Indicates Sample Videos Event content-aware loss for frame-by-frame vision-to-text; represents the temperature hyperparameter; Indicates the size of the training batch; Indicates Sample Videos Frame features of video frames; Indicates Sample Videos Frame text features of the frame description corresponding to the frame video frame; Indicates Sample Videos The frame text features of the frame description corresponding to the frame video frame, where Represents any sample video in the batch ; Indicates video frames; in addition, Indicates Sample Videos Event content-aware loss for text-to-vision frame-by-frame video; Indicates Sample Videos Frame features of video frames; Represents event content-aware loss; The event timing perception loss is calculated by the following formula: ; ; ; in, Indicates The event-time-aware vision-to-text loss for synthesized sample videos; Indicates Video features of synthesized sample videos; Indicates The overall text features of the sample videos; Indicates The overall text features of the synthesized sample videos; Indicates A text-to-visual event-timing-aware loss for synthesized sample videos; Indicates Video features of synthesized sample videos; Represents event timing-aware loss.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the training method of the event-aware cross-modal video text retrieval model is implemented as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the training method of the event-aware cross-modal video text retrieval model as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the training method of the event-aware cross-modal video text retrieval model as described in any one of claims 1 to 4.

Citation Information

Patent Citations

  • Text-video retrieval method based on fine-grained cross-modal alignment model

    CN114048350A

  • Video text retrieval method, electronic equipment and medium

    CN116737995A

  • Video text cross-modal retrieval method and device, equipment and medium

    CN117725261A

  • Unsupervised hashing method for cross-modal video-text retrieval with clip

    WO2023004206A1