Text video retrieval method of fine-grained relation learning network based on energy perception

Abstract

The invention provides a text video retrieval method of a fine-grained relation learning network based on energy perception. The method comprises the following steps: giving a query text and a video clip; inputting the query text into a text encoder of the CLIP, inputting the video clip into a text encoder image encoder of the CLIP, and extracting to obtain text embedding and frame embedding; inputting the text embedding and the frame embedding into a fine-grained relation learning network for text enhancement operation to obtain enhanced text embedding; taking the enhanced text embedding as a frame fusion condition to carry out frame fusion operation on the frame embedding to obtain video embedding; calculating the similarity of a text-video pair formed by the query text and the video embedding based on a cosine similarity function; and selecting the text-video pair with the highest similarity as a retrieval output result of text video retrieval. According to the method, the problem of randomness of the random text of single sampling is solved, so that semantic information of text coding is better expanded, and the final retrieval effect is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of text video retrieval, and particularly relates to a text video retrieval method based on an energy-aware fine-grained relationship learning network. BACKGROUND

[0002] Recent mainstream methods require fine-tuning a pre-trained large text-image model (CLIP). First, the pre-trained CLIP is used to encode the text and video, and then project them into a common space, and then align the corresponding text and video. Previous methods focus on extracting better video features, often ignoring the amount of information in the text features. Compared with information-rich videos, the semantic information contained in the text is less, and it is difficult to fully reflect the content of the video, thus increasing the difficulty of retrieval. There are already some methods that focus on overcoming this difficulty, such as TMASS using random text modeling to generate text encodings containing more semantic information, thereby achieving the purpose of enriching the amount of text information. But TMASS only samples a random text encoding, and the randomness brought by the random modeling used by it may make the text encoding carry some irrelevant noise, affecting the final retrieval effect. SUMMARY

[0003] The present application aims to overcome the above technical deficiencies, and provides a text video retrieval method based on an energy-aware fine-grained relationship learning network, which solves the technical problem that the randomness of random text encoding in the prior art may make the text encoding carry some irrelevant noise, affecting the final retrieval effect.

[0004] To achieve the above technical purpose, in a first aspect, the technical solution of the present application provides a text video retrieval method based on an energy-aware fine-grained relationship learning network, comprising the steps of:

[0005] Given a query text and a video segment;

[0006] input the query text into the text encoder of CLIP, and input the video segment into the image encoder of the text encoder of CLIP to extract text embedding and frame embedding;

[0007] input the text embedding and the frame embedding into a fine-grained relationship learning network for text enhancement operation to obtain an enhanced text embedding;

[0008] perform frame fusion operation on the frame embedding with the enhanced text embedding as a condition for frame fusion to obtain a video embedding;

[0009] calculate the energy score of the text-video pair composed of the enhanced text embedding of the query text and the video embedding based on a cosine similarity function;

[0010] Select the text-video pair with the highest similarity as the retrieval output result of the text-video retrieval.

[0011] Compared with the prior art, the present application has the following advantages.

[0012] The present application can make the text encoding more stable and reasonable by solving the randomness problem of single sampling of random text. The previous random text may have the problems of incomplete and inaccurate semantic expression, and the method can optimize it, so that the semantic information contained in the text can be better mined and utilized, avoiding the semantic loss or deviation caused by randomness, and laying a foundation for subsequent accurate retrieval. The present application uses a fine-grained relationship learning network to perform text enhancement operation, which can further expand the semantic information of the text encoding. It is not simply processing the original text, but deeply mining the more detailed correlation between the text and the video, so as to integrate more dimensional semantic elements, so that the text embedding can carry more rich content and more accurately match and compare with the video content.

[0013] According to some embodiments of the present application, the query text is input into the text encoder of CLIP, the video segment is input into the image encoder of the text encoder of CLIP, and the text embedding and the frame embedding are extracted, including the steps of:

[0014] The video segment is uniformly sampled into multiple video frames, the query text is input into the text encoder of CLIP, the video frames are input into the image encoder of the text encoder of CLIP, and the text embedding and the frame embedding are extracted.

[0015] According to some embodiments of the present application, the text embedding and the frame embedding are input into a fine-grained relationship learning network to perform text enhancement operation to obtain an enhanced text embedding, including the steps of:

[0016] Randomly sampling the text embedding with random text modeling to obtain multiple text embedding candidates;

[0017] The text embedding candidates and the frame embedding are regarded as nodes, and a heterogeneous text-frame graph composed of various types of nodes and relationships is constructed, the text-frame graph including multiple node pairs;

[0018] The relationship between the text embedding and the frame embedding is learned by a relationship graph attention network: text-frame weight;

[0019] Then, the multiple text embedding candidates are aggregated into a final enhanced text embedding using the text-frame weight.

[0020] According to some embodiments of the present application, the node pairs have three types of relationships: text-text, frame-frame, and text-frame.

[0021] According to some embodiments of the present application, a symmetric cross-entropy loss function, a support loss function and an energy loss function are set during the training process of the CLIP, and a final matching loss function is calculated based on the symmetric cross-entropy loss function, the support loss function and the energy loss function.

[0022] The calculation step of the symmetric cross-entropy loss function comprises:

[0023] Based on the enhanced text embedding, a text-to-video retrieval loss function and a video-to-text retrieval loss function are calculated;

[0024] The symmetric cross-entropy loss function is calculated based on the text-to-video retrieval loss function and the video-to-text retrieval loss function;

[0025] The calculation step of the support loss function comprises:

[0026] Based on the sampling radius of the random text modeling, a random text embedding on the sampling boundary is calculated as a support text embedding;

[0027] The symmetric cross-entropy loss function is calculated based on the support text embedding and the video embedding as the support loss function.

[0028] The calculation step of the energy loss function comprises:

[0029] A similarity function is selected as an energy function in the energy model;

[0030] Based on the energy function and the Monte Carlo sampling method, a plurality of embedding of false text-video pairs are generated;

[0031] Based on the energy function, the energy scores of all real text-video pair embeddings and false text-video pair embeddings are calculated, and the energy loss function is calculated.

[0032] According to some embodiments of the present application, the energy function is any one of the following:

[0033] The negative cosine similarity function, the bilinear score function or the MLP energy function.

[0034] In a second aspect, the technical scheme of the present application provides a text-video retrieval system based on an energy-aware fine-grained relationship learning network, comprising:

[0035] An embedding extraction module is provided, which inputs a query text and a video segment into the text encoder of the CLIP and the image encoder of the text encoder of the CLIP, respectively, to extract text embedding and frame embedding;

[0036] a fine-grained relation learning network, performing a text enhancement operation on the text embedding and the frame embedding input to obtain an enhanced text embedding;

[0037] a video fusion module, performing a frame fusion operation on the frame embedding with the enhanced text embedding as a condition of frame fusion to obtain a video embedding;

[0038] a retrieval result output module, selecting a text-video pair with the highest similarity as a retrieval output result of text-video retrieval;

[0039] an energy-aware matching module, modeling the distribution of text-video pairs based on an energy model, used to assist model training.

[0040] In a third aspect, a computer-readable storage medium is provided, and the computer-readable storage medium stores computer executable instructions, and the computer executable instructions are used to make a computer execute the text-video retrieval method based on the energy-aware fine-grained relation learning network according to any one of the first aspect.

[0041] Additional aspects and advantages of the present application will be given in part in the following description, become apparent from the following description, or be understood through practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0042] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description, taken in conjunction with the accompanying drawings, in which:

[0043] Figure 1 a flowchart of the text-video retrieval method based on the energy-aware fine-grained relation learning network provided by an embodiment of the present application;

[0044] Figure 2 an energy efficiency comparison diagram of multiple latest models on the MSRVTT dataset provided by another embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the present application more clear, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0046] It should be noted that although the functional modules are divided in the system schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in a manner different from the module division in the system or the order in the flowchart. The terms "first", "second", etc. in the description and claims and the above figures are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0047] Referring to Figure 1 and Figure 2 , Figure 1 The flowchart of the text video retrieval method based on the energy-aware fine-grained relationship learning network provided by an embodiment of the present application; Figure 2 The energy efficiency comparison chart of multiple latest models on the MSRVTT data set provided by another embodiment of the present application. Figure 1 In the figure: (a) training process overview. (b) Fine-grained relationship learning (FRL) first samples multiple text candidates and learns text-frame weights, and then uses the weights to aggregate the text candidates into a final rich text embedding containing context related to the video frames. (c) Energy-aware matching (EAM) improves the relationship learning in FRL by capturing detailed text-frame interactions and accurately modeling the real text-video pair distribution, thereby improving the final matching performance from a fine-grained perspective.

[0048] In an embodiment, the text video retrieval method based on the energy-aware fine-grained relationship learning network comprises the steps of: given a query text and a video segment; input the query text into the text encoder of CLIP and input the video segment into the image encoder of the text encoder of CLIP to extract text embedding and frame embedding; input the text embedding and the frame embedding into the fine-grained relationship learning network to perform text enhancement operation to obtain enhanced text embedding; perform frame fusion operation on the frame embedding based on the enhanced text embedding as the condition of frame fusion to obtain video embedding; calculate the similarity of the text-video pair composed of the enhanced text embedding of the query text and the video embedding based on the cosine similarity function; select the text-video pair with the highest similarity as the retrieval output result of the text video retrieval; and additionally model the distribution of the text-video pair based on the energy model during training, which is only used to assist model training and is not used in the final retrieval.

[0049] The present application can make the text encoding more stable and reasonable by solving the problem of random text randomness caused by single sampling. The previous random text may have the problems of incomplete and inaccurate semantic expression, and the present method can optimize it, so that the semantic information contained in the text can be better mined and utilized, avoiding the semantic loss or deviation caused by randomness, and laying a foundation for subsequent accurate retrieval.

[0050] This invention utilizes a fine-grained relation learning network for text enhancement, further expanding the semantic information of the encoded text. It doesn't simply process the original text, but delves deeper into the text itself and the more nuanced relationships between text and video, incorporating more multi-dimensional semantic elements. This allows the text embedding to carry richer content and more accurately match and contrast with the video content. After obtaining the enhanced text embedding, this invention uses it as a condition for frame fusion to obtain the video embedding. This process establishes a closer and more reasonable connection between text and video. The energy score of the text-video pair, composed of the text and video embeddings, is calculated based on a similarity function to select retrieval results. Compared to traditional methods, this more accurately measures the degree of matching between text and video, reduces false matches, and thus improves retrieval accuracy.

[0051] In summary, the effective expansion of semantic information and the improvement of matching accuracy mentioned above ultimately enhance the overall effectiveness of text video retrieval. Improvements are seen in both the relevance of search results to query requirements and the rationality of result ranking, enabling users to more efficiently obtain video content that meets their expectations.

[0052] Faced with today's diverse and complex text and video data scenarios, this invention, through operations such as fine-grained relation learning, can better adapt to texts of different styles and themes, as well as various types of video clips. It possesses strong versatility and adaptability, and can be widely applied to various text and video retrieval scenarios. This invention proposes innovative approaches based on energy perception and the construction of fine-grained relation learning networks, which differ from traditional, relatively simple and direct retrieval methods. It provides new technical means and research directions for the field of text and video retrieval, contributing to the further development and improvement of this technology.

[0053] I. General Process of Text and Video Retrieval Tasks

[0054] 1.1 Text and Video Representation

[0055] The goal of text-video retrieval is to align text and video into a joint space. The first step is to learn representations of text and video using a large-scale pre-trained vision-language model, CLIP. Specifically, given a text-video pair {t, v}, the standard approach is to first uniformly sample M frames [f1, f2, ..., f...]. M Then input it into CLIP's text encoder φ t (·) and image encoder φ v In (·), the text embedding t and the frame embedding F are obtained, represented as:

[0056]

[0057] where d is the embedded hidden dimension.

[0058] Many previous works employ a fusion module to learn global video features to reduce the redundant information in video frames and align t with v. A simple solution is average pooling. However, it treats all frames equally, ignoring important frames while introducing noisy information from irrelevant frames. This invention follows the practice of XPool [2] and uses text embedding t as a condition so that the fusion module can pay more attention to text-related frames:

[0059]

[0060] This method uses cross-attention to achieve (hierarchical) text-frame interaction and temporal modeling, etc.

[0061] 1.2 Training and inference

[0062] Given query text t and video segment v, the text embedding and video embedding are obtained using the multi-modal encoder of CLIP Then the similarity measure function s(t, v) (such as cosine similarity) can be used to calculate the correlation between text and video.

[0063] In the training phase, for each training batch and its batch size B, a symmetric cross-entropy loss will be applied to minimize the distance between relevant text-video pairs while maximizing the distance between irrelevant pairs, similar to contrastive learning. Here this invention considers both text-to-video retrieval and video-to-text retrieval :

[0064]

[0065] where λ is a learnable factor. The overall symmetric cross-entropy loss function is:

[0066]

[0067] In the inference process, given n t text and n v video segments for a query text t query , the matching video segment v * is the one with the highest score, and vice versa.

[0068]

[0069] II. Energy-Aware Fine-Grained Relationship Learning Network (EagleNet)

[0070] 2.1 Fine-Grained Relationship Learning (FRL)

[0071] In the process of rich text embedding, relying on a single random text embedding can introduce unnecessary noise. Specifically, to mitigate this individual bias, the present invention first models multiple text embedding candidates with random texts, and then treats the text and frames as nodes to form a heterogeneous Text-Frame Graph composed of various types of nodes and edges. Then the present invention designs a simple and efficient Relation Graph Attention Network (RGAT) to learn the relationship between the text and the frame (Text-Frame Weights), so that the present invention can aggregate text candidates and generate a more semantically rich coherent text embedding.

[0072] 2.1.1 Text-Frame Graph Construction.

[0073] Given the text embedding t and the frame embedding F = [f1, f2, …, fM], the present invention uses the random text modeling technique [3] to sample S random text embeddings M Then the text embedding and the frame embedding are connected into a node matrix as follows:

[0074]

[0075] where “;” is the row concatenation operation, and n = 1 + S + M is the number of nodes. Therefore, there are three types of edges (relationships) between nodes: text-text, frame-frame, and text-frame. This results in three adjacency matrices where r ∈ {tt, ff, tf}, and in each A r , the node pairs linked by the relationship r are fully connected. For example:

[0076]

[0077] 2.1.2 Relationship Learning

[0078] To extract useful information from the complex relationship between the text and the video frame, the present invention proposes a simple and effective Relation Graph Attention Network (RGAT). RGAT has a multi-layer structure, and in each layer, for any node pair (i, j) connected by the relationship r (at this time ), the edge weight is given by:

[0079] ​

[0080] where h = 1, 2, …, H, H is the number of attention heads, h i is the feature of node i (in the first layer, h i is the i-th row of matrix x), “||” denotes the column concatenation operation, W r，h is a learnable matrix, to obtain edge weights. Then, the attention score is calculated as follows:

[0081]

[0082] where, is the neighbor set connected with node i through relation r. Finally, the output node of this layer is denoted as

[0083]

[0084] where, σ(·) is the activation function. Then h′ i is the input of the next layer RGAT.

[0085] Note that h′ i of each layer here is only an intermediate medium, while the ultimate goal of FRL is to learn the relationship between text and video frames These relationships are considered as the importance of different text embeddings under the guidance of all frames. Then these weights are used to aggregate text candidates into a final rich text embedding. Specifically, the text-frame edge weights from the last layer are averaged among attention heads to obtain the final text-frame weights, i.e.

[0086]

[0087] Then the invention takes the average of the text-frame weights of each text node to obtain the attention score of each text node.

[0088]

[0089] Therefore, the final generated text embedding is

[0090]

[0091] The enhanced text embedding t gen is used as the condition for frame fusion (equation (2)) to obtain the video embedding v, then t gen and v are used in the symmetric loss (equation (3)). According to [3], the invention further obtains a support text t sup and calculates the support loss sup using t and v through equation (3), while tsup As the boundary of randomness, the random text embedding is sampled within this range. Thus, is used to limit the degree of randomness. The final matching loss is

[0092]

[0093] where λ sup is a coefficient.

[0094] 2.2 Energy-aware matching (EAM)

[0095] The present invention proposes energy-aware matching (EAM) to facilitate FRL to learn text-frame relationships better. EAM uses an energy-based model (EBM) to model the distribution of text-video pairs, thus exploiting the fine-grained interactions between text and video frames. Since EBMs perform well in modeling data distribution, EAM also improves the matching performance from a fine-grained perspective.

[0096] Specifically, given a text embedding t and frame embeddings F = [f1, f2,..., f M ], the EBM uses the Boltzmann distribution to effectively simulate the distribution p(t, F) of real text-video pairs, where Z θ is the normalization constant, and θ is the parameter, denotes the energy score of a text-video pair. The present invention first computes the energy score of each text-frame pair, and then takes their average to obtain the energy score of the text-video pair In this way, EAM can fully exploit all the information in the fine-grained text and video frame interactions. On the other hand, since the truly matched text-video pairs have lower energy scores, while the unmatched ones have higher energy scores, EBMs can judge whether a text-video pair is matched according to the energy score, further helping the model to accurately identify the corresponding video (or text) of the query text (or video).

[0097] 2.2.1 Selection of energy function

[0098] Since the energy score indicates the relevance of the text-video pair, the present invention can adopt a similarity function as the E θ . For example, the negative cosine similarity can be used as a parameter-free energy function:

[0099]

[0100] Or the present invention can use a bilinear score function with learnable parameters W:

[0101]

[0102] For better flexibility, one MLP can also be used as the energy function:

[0103]

[0104] 2.2.2 Training EAM

[0105] The EBM is usually trained by minimizing the negative log-likelihood loss (NLL) during actual learning, i.e. which can be reformulated as:

[0106]

[0107] where B is the batch size, is B' fake text-video pairs sampled from , which can be obtained by K-step Markov Chain Monte Carlo (MCMC) sampling:

[0108]

[0109] where and are given initial samples, η is the step size, and has a given noise σ 2 In practice, the present application stops updating the gradients of and after sampling. The specific process of the sampling algorithm is shown as follows.

[0110]

[0111] The final loss of the present application is

[0112]

[0113] where λ eam is a coefficient. EAM is only used during the training process, so it will not participate in the inference phase.

[0114] In an embodiment, the text-video retrieval method based on the energy-aware fine-grained relation learning network comprises the following steps: given a query text and a video segment; input the query text into the text encoder of CLIP and input the video segment into the image encoder of the text encoder of CLIP to extract text embedding and frame embedding; input the text embedding and the frame embedding into the fine-grained relation learning network to perform text enhancement operation to obtain enhanced text embedding; perform frame fusion operation on the frame embedding by taking the enhanced text embedding as the condition of frame fusion to obtain video embedding; calculate the similarity of the text-video pair composed of the enhanced text embedding of the query text and the video embedding based on the cosine similarity function; select the text-video pair with the highest similarity as the retrieval output result of the text-video retrieval; and additionally model the distribution of the text-video pair based on the energy model during training, which is only used to assist model training and is not used in the final retrieval.

[0115] In the method, the inputting of the query text into the text encoder of CLIP and the inputting of the video segment into the image encoder of the text encoder of CLIP to extract the text embedding and the frame embedding comprise the following steps: uniformly sampling multiple video frames from the video segment; inputting the query text into the text encoder of CLIP and inputting the video frames into the image encoder of the text encoder of CLIP to extract the text embedding and the frame embedding.

[0116] In the method, the inputting of the text embedding and the frame embedding into the fine-grained relation learning network to perform text enhancement operation to obtain enhanced text embedding comprises the following steps: randomly sampling multiple text embedding candidates from the text embedding by random text modeling; regarding the text embedding candidates and the frame embedding as nodes to construct a heterogeneous text-frame graph composed of various types of nodes and relations, the text-frame graph comprising multiple node pairs; learning the relation between the text embedding and the frame embedding by a relation graph attention network: text-frame weight; and then aggregating the multiple text embedding candidates into a final enhanced text embedding by using the text-frame weight.

[0117] The present application generates multiple text embedding candidates through random text modeling, breaking through the semantic deviation or information incompleteness problem that may exist in single text encoding. The diversified candidates formed by random sampling can cover multiple semantic angles of the text, avoid semantic loss caused by randomness of single encoding, and make the text expression more comprehensive and stable. For example, the same query text may have multiple dimensions of semantics (such as “running puppy” contains both the action “running” and the subject “puppy”), and multiple candidate embeddings can focus on different dimensions, providing a richer semantic basis for subsequent fusion.

[0118] Fine-grained capturing the association between text and video frames, strengthening cross-modal semantic alignment, constructing text embedding candidates and frame embedding into a heterogeneous text-frame graph, and explicitly modeling the relationship between the two as the connection between node pairs in the graph can fine-grainedly mine the potential association between text and each frame of video content (such as the correspondence between text keywords and objects, actions, and scenes in video frames). Compared with the traditional global matching method, this graph structure can preserve the local semantic relationship between text and video frames, avoid the "drowning" of fine-grained information by overall features, and make cross-modal semantic alignment more accurate.

[0119] In the training process of CLIP, a symmetric cross-entropy loss function, a support loss function and an energy loss function are set, and a final matching loss function is calculated based on the symmetric cross-entropy loss function, the support loss function and the energy loss function; the calculation steps of the symmetric cross-entropy loss function include: based on the enhanced text and the video embedding, calculating the text-to-video retrieval loss function and the video-to-text retrieval loss function; the symmetric cross-entropy loss function is calculated based on the text-to-video retrieval loss function and the video-to-text retrieval loss function; the calculation steps of the support loss function include: based on the sampling radius of the random text modeling, the random text embedding on the sampling boundary is calculated as the support text embedding; the symmetric cross-entropy loss function is calculated based on the support text embedding and the video embedding, which is used as the support loss function; the calculation steps of the energy loss include: selecting a similarity function as an energy function in the energy model, generating multiple false text-video pair embeddings based on the energy function and the Monte Carlo sampling method, calculating the energy scores of all real text-video pair embeddings and false text-video pair embeddings based on the energy function, and calculating the energy loss function, which is only used for auxiliary model training.

[0120] In an embodiment, the text-video retrieval system based on the energy-aware fine-grained relationship learning network, characterized in that it comprises: an embedding extraction module, which inputs a query text and a video segment into the text encoder of CLIP and the image encoder of CLIP, respectively, and extracts text embedding and frame embedding; a fine-grained relationship learning network, which performs text enhancement on the text embedding and the frame embedding to obtain enhanced text embedding; a video fusion module, which performs frame fusion on the frame embedding based on the enhanced text embedding to obtain video embedding; a similarity calculation module, which calculates the similarity of the text-video pair composed of the enhanced text embedding of the query text and the video embedding based on the cosine similarity function; a retrieval result output module, which selects the text-video pair with the highest similarity as the retrieval output result of text-video retrieval. The energy-aware matching module models the distribution of the text-video pair based on the energy model during training, which is only used for auxiliary model training and is not used for final retrieval.

[0121] The system disassembles the core function into six independent modules: embedding extraction, fine-grained relation learning, video fusion, similarity calculation, retrieval result, and energy-aware matching output. Each module works collaboratively through standardized input and output interfaces. This architecture allows individual optimization of a module (such as replacing a more efficient embedding extraction model or upgrading the relation learning network) without modifying the overall framework, significantly reducing the cost of system iteration. The system constructs a complete semantic transmission chain from the original input to the final retrieval result through the module series of "text enhancement → conditional fusion → similarity scoring": the enhanced text embedding generated by the fine-grained relation learning network is directly used as the condition for the video fusion module, ensuring accurate guidance of the text semantics to the video frame fusion; and the energy-aware matching module models the real text-video pair based on the output of the previous modules, assisting the model training, so that the semantic information is progressively transmitted and echoed among the modules. This collaborative design avoids the semantic fragmentation between modules in traditional systems (such as independent processing of text and video features), ensuring continuous strengthening of the semantic association between text and video throughout the entire process.

[0122] The modular implementation of the energy-aware mechanism improves the reliability of the matching decision. The system treats the energy-aware matching as an independent module, allowing it to focus on optimizing the similarity measurement logic of the text-video pair. This module can flexibly adapt to different similarity functions (such as cosine similarity, bilinear score function, etc.) and adjust the energy score calculation strategy for specific scenarios (such as short video retrieval and long video segment matching). The independent energy-aware module also facilitates the introduction of regularization mechanisms or dynamic threshold adjustment strategies, reducing the interference of abnormal samples (such as noisy video frames and ambiguous text) on the retrieval results and improving the stability of the system decision.

[0123] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include high-speed random access memory and can also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory that is remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0124] The apparatus embodiments described above are merely illustrative, and units described as separate components can or can not be physically separated, and can be located in one place or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment.

[0125] In addition, one embodiment of the present application further provides a computer readable storage medium storing computer executable instructions, which are executed by a processor or a controller, for example, a processor in the terminal embodiment, so that the processor executes the text video retrieval method based on the energy-aware fine-grained relationship learning network in the above embodiment.

[0126] Those skilled in the art can understand that all or some steps in the above disclosed method and system can be implemented as software, firmware, hardware and appropriate combinations thereof. Some or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software can be distributed on a computer readable medium, which can include computer storage media (or non-transitory media) and communication media (or transitory media). As known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, as known to those skilled in the art, communication media generally includes computer readable instructions, data structures, program modules or other data in modulated data signals such as carrier waves or other transport mechanisms, and can include any information delivery medium.

[0127] The above is a specific description of the preferred embodiment of the present application, but the present application is not limited to the above embodiment. Those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

[0128] The above specific embodiments of the present application do not constitute a limitation on the protection scope of the present application. Any other corresponding changes and modifications made according to the technical concept of the present application should be included in the protection scope of the claims of the present application.

Claims

1. A text video retrieval method based on an energy-aware fine-grained relationship learning network, characterized in that, The method comprises the steps of: inputting a query text and a video clip into a text encoder of CLIP and inputting the video clip into an image encoder of the text encoder of CLIP to extract a text embedding and a frame embedding; inputting the text embedding and the frame embedding into a fine-grained relation learning network to perform a text enhancement operation to obtain an enhanced text embedding; performing a frame fusion operation on the frame embedding by taking the enhanced text embedding as a condition of frame fusion to obtain a video embedding; calculating a similarity of a text-video pair composed of the enhanced text of the query text and the video embedding based on a cosine similarity function; selecting a text-video pair with the highest similarity as a retrieval output result of text-video retrieval. The method comprises the steps of: 2.The text video retrieval method based on the energy-aware fine-grained relation learning network according to claim 1, wherein, uniformly sampling a plurality of video frames from the video clip, inputting the query text into the text encoder of CLIP, and inputting the video frames into the image encoder of the text encoder of CLIP to extract the text embedding and the frame embedding. The method comprises the steps of: 3.The text video retrieval method based on energy-aware fine-grained relation learning network according to claim 1, wherein, randomly sampling a plurality of text embedding candidates from the text embedding by using random text modeling; regarding the text embedding candidates and the frame embedding as nodes to construct a heterogeneous text-frame graph composed of various types of nodes and relations, the text-frame graph comprising a plurality of node pairs; learning a relation between the text embedding and the frame embedding by using a relation graph attention network, i.e., a text-frame weight; then aggregating the plurality of text embedding candidates into a final enhanced text embedding by using the text-frame weight. There are three types of relations between the node pairs: text-text, frame-frame, and text-frame. 4.The text video retrieval method based on the energy-aware fine-grained relation learning network according to claim 1, wherein, During the training process of CLIP, a symmetric cross-entropy loss function, a support loss function, and an energy loss function are set, and a final matching loss function is calculated based on the symmetric cross-entropy loss function, the support loss function, and the energy loss function. 5.The text-based video retrieval method of energy-aware fine-grained relation learning network according to claim 1, wherein, The calculation steps of the symmetric cross-entropy loss function comprise: based on the enhanced text embedding, calculating a text-to-video retrieval loss function and a video-to-text retrieval loss function; calculating the symmetric cross-entropy loss function based on the text-to-video retrieval loss function and the video-to-text retrieval loss function; The calculation steps of the support loss function comprise: based on the sampling radius of the random text modeling, calculating a random text embedding on the sampling boundary as a support text embedding; calculating the symmetric cross-entropy loss function based on the support text embedding and the video embedding as the support loss function; The calculation steps of the energy loss function comprise: selecting a similarity function as an energy function in the energy model; based on the energy function and the Monte Carlo sampling method, generating embeddings of a plurality of false text-video pairs; based on the energy function, calculating energy scores of all real text-video pair embeddings and false text-video pair embeddings, and calculating the energy loss function. The energy function is any one of the following: 6.The text-based video retrieval method of energy-aware fine-grained relation learning network according to claim 5, wherein, ​ A negative cosine similarity function, a bilinear score function, or an MLP energy function.

7. A text video retrieval system based on an energy-aware fine-grained relation learning network, characterized in that, The method comprises the following steps: embedding extraction module, given a query text and a video clip, input the query text into the text encoder of CLIP, input the video clip into the image encoder of CLIP, and extract text embedding and frame embedding; fine-grained relation learning network, input the text embedding and the frame embedding to perform text enhancement operation to obtain enhanced text embedding; video fusion module, perform frame fusion operation on the frame embedding by taking the enhanced text embedding as the condition of frame fusion to obtain video embedding; calculate the similarity of the text-video pair composed of the query text and the video embedding based on the cosine similarity function; retrieval result output module, select the text-video pair with the highest similarity as the retrieval output result of text-video retrieval; energy-aware matching module, model the distribution of the text-video pair based on the energy model, and used for assisting model training.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer executable instructions for causing the computer to execute the text-video retrieval method based on the energy-aware fine-grained relation learning network according to any one of claims 1 to 6.

Images