Weak supervision video clip retrieval method and system based on reinforcement learning

Abstract

The invention relates to a weak supervision video clip retrieval method and system based on reinforcement learning, belongs to the technical field of video clip retrieval in computer vision, integrates two intelligent agent-based adaptive modules, namely an intelligent agent controlled convertible encoder, and enhances the flexibility and accuracy of video feature coding. Besides, a relevance-oriented return generator is designed for weak supervision, and the relevance between videos is accurately evaluated by utilizing fine-grained interaction between video frames, so that reliable returns are distributed to multiple agents. By adopting the algorithm designed by the invention, more efficient and more accurate video clip retrieval can be realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of video clip retrieval in computer vision, and in particular relates to a weakly supervised video clip retrieval method and system based on reinforcement learning. Background Art

[0002] The task of video clip retrieval aims to retrieve relevant clips from an uncut long video based on given query information. In video clip retrieval, query information can take different forms, such as text, images, and video clips. Text and images as query information, although simple and intuitive to a certain extent, have limitations in information expression. It is difficult for text to accurately describe complex actions or event sequences in a video, while images cannot convey temporal continuity and dynamic changes. Therefore, although they are very effective in some scenarios, they have certain limitations when highly accurate and detailed retrieval is required. To overcome these limitations, video query-based video clip retrieval uses video clips as query information, which can intuitively display the required scene dynamics and contextual information, thereby more effectively meeting the user's retrieval needs and achieving more accurate retrieval. The input of this task is a query video and a reference video, where the query video is a short video clip of interest to the user, and the reference video is an uncut long video. The task goal is to retrieve the target clip that is semantically related to the query video from the reference video and determine the temporal boundaries of the clip, i.e., the start time and end time.

[0003] Most current methods are implemented in a fully supervised setting. Although they can achieve excellent retrieval performance, they rely on the temporal boundary annotation of the target segment for training. The annotation process is time-consuming and laborious, and lacks scalability in large-scale practical applications. Therefore, a weakly supervised method is proposed. During training, only the matching information between the query video and the reference video is used, but there is no temporal boundary annotation of the target segment. This low-cost annotation method makes the model more practical.

[0004] The weakly supervised video clip retrieval task faces many challenges for the following reasons:

[0005] First, videos are diverse, and the total length of each video and the length of the action segments in it are different. Therefore, a fixed feature encoding module may not be suitable for all videos. Therefore, how to obtain effective video feature representation is a major challenge.

[0006] Second, in the weakly supervised setting, there is no temporal boundary annotation of the target segment as supervision information, and the frames near the boundary are often very similar, so it is difficult to identify the frames located on the boundary. Therefore, it is challenging to accurately determine the temporal boundary of the target segment. Summary of the invention

[0007] The purpose of the present invention is to overcome the deficiencies of the prior art and to propose a weakly supervised video clip retrieval method and system based on reinforcement learning to achieve efficient and accurate retrieval under weakly supervised settings.

[0008] The present invention solves the technical problem by the following technical solutions:

[0009] A weakly supervised video clip retrieval method based on reinforcement learning, the steps of the method are:

[0010] S1, a switchable feature extractor generates state features of the current environment based on the input video data;

[0011] S2, the multi-agent generates corresponding action strategies and state values ​​based on the state features of S1, adjusts the time boundary of the current retrieval segment, and selects a feature encoder for the video from the candidate encoder set;

[0012] S3. The relevance-oriented reward generator calculates the similarity scores between the query video and the retrieved segments and assigns corresponding rewards to the multi-agents.

[0013] Moreover, the S1 is specifically:

[0014] 1) The video data is the query video V Q And reference video V R , the current environment is a collection of query video, reference video and current positioning segment, query video V Q And reference video V R Transformed into corresponding feature sequences through the backbone network, respectively expressed as and , where l q and l r represents the length of the query video and the reference video, d f represents the dimension of each feature vector;

[0015] 2) At each time step t, the reference video V R It is divided into three parts: the left segment, the current segment and the right segment. The feature sequences of the above three segments are obtained by selecting the corresponding parts from the reference video feature sequence, which are respectively expressed as , and ;

[0016] 3) Enhance the video features using feature encoders. The query video uses a fixed encoder, and the reference video and its three segments use a convertible encoder, which contains n sub-encoders e 1, e 2, e 3,... e n , at each time step, the conversion agent controls the convertible encoder to select a specific sub-encoder for video feature encoding, and the specific sub-encoder is described by the following formula:

[0017] (1);

[0018] Where: x represents the input of the transformable encoder, SA represents the transformable agent, and s t Indicates state characteristics;

[0019] 4) Perform average pooling on the feature sequences of the query video, reference video, left segment, current segment, and right segment to obtain fixed-length features, denoted as f q 、f r , , and These features are used as the input of the state encoding. In order to clearly represent the state of the current retrieval fragment, the time boundary of the normalization , t=1,2,...T max That is, the position feature is also used for state encoding, where N represents the length of the reference video; the video feature and the position feature are concatenated and input into a fully connected layer Φ to obtain the final state feature, which is expressed as:

[0020] (2).

[0021] Moreover, the S2 is specifically:

[0022] 1) The multi-agent includes a boundary agent and a transition agent. Each agent has two fully connected layer branches, namely the actor branch and the critic branch. They share a GRU layer. At each time step, the input of the GRU is the state feature generated by the transition encoder, and its hidden state is used for strategy generation and state value estimation. The actor branch outputs the action strategy , which represents the probability distribution of all actions in the corresponding action space, and the critic branch outputs the state value , which represents the estimated value of the cumulative return of the current state, where i∈{boundary, switch}, and are the actor branch parameters and critic branch parameters of agent i respectively;

[0023] 2) Each agent selects an action from the corresponding action space according to the learned strategy. The action space of the conversion agent contains n actions: select the sub-encoder e in the convertible encoder i , where i∈{1,2,...n};

[0024] The action space of the boundary agent contains 4 actions: 1) move the starting boundary forward by δ; 2) move the starting boundary backward by δ; 3) move the ending boundary forward by δ; 4) move the ending boundary backward by δ, and set δ to N / 10.

[0025] Moreover, the S3 is specifically:

[0026] 1) Estimate the correlation between the query video and the retrieval clip and obtain the corresponding similarity score;

[0027] Given a query video and a feature sequence of retrieved clips and , use the BiLSTM layer to perform bidirectional sequence encoding, calculate the attention weighted features of the query video and the retrieval segment, and the specific calculation process is expressed as follows:

[0028] (3);

[0029] 2) Based on the similarity scores, corresponding rewards are assigned to multiple agents. The reward obtained by the boundary agent at the t time step is defined as:

[0030] (4);

[0031] Among them: τ is the factor that determines the size of the return;

[0032] The reward of the transition agent at time step t is obtained based on the reward of the boundary agent at time step t+1, expressed as:

[0033] (5);

[0034] Considering the sequential decision-making process, it is necessary to trace the rewards of subsequent time steps back to the current time step. The cumulative rewards of multiple agents at the tth time step are expressed as:

[0035] (6);

[0036] Among them, γ is the discount factor.

[0037] A weakly supervised video segment retrieval system based on reinforcement learning, executing the weakly supervised video segment retrieval method based on reinforcement learning, comprising:

[0038] A switchable feature extractor generates state features of the current environment based on the input video data;

[0039] Multiple agents, namely boundary agents and conversion agents, generate corresponding action strategies and state values ​​based on state features, where the boundary agent adjusts the time boundary of the current retrieval segment, and the conversion agent selects a feature encoder for the video from a set of candidate encoders;

[0040] A relevance-oriented reward generator calculates similarity scores between query videos and retrieved segments, and based on this, assigns corresponding rewards to multiple agents to measure the effectiveness of the actions they perform.

[0041] The advantages and beneficial effects of the present invention are:

[0042] The present invention adaptively guides video feature encoding and segment boundary location by training multiple agents. Specifically, an agent-controlled switchable encoder is designed, which enables the encoder to adaptively select the optimal encoding strategy for the current task goal through a novel goal-oriented switchable mechanism, ensuring context-aware feature encoding and improving flexibility. Secondly, an agent-guided boundary locator is designed to gradually refine the temporal boundary of the retrieval moment, thereby improving the accuracy of boundary location. In addition, a correlation-oriented reward generator is designed for weak supervision, which uses fine-grained interactions between video frames to estimate the correlation between the retrieved segment and the query video, thereby assigning reliable rewards to multiple agents to guide them to identify segments that are more relevant to the query video. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a schematic diagram of the system network structure of the present invention. DETAILED DESCRIPTION

[0044] The present invention is further described in detail below through specific examples. The following examples are only illustrative and not restrictive, and the protection scope of the present invention cannot be limited thereto.

[0045] The network structure of the weakly supervised video clip retrieval system based on reinforcement learning in the present invention is as follows: Figure 1 As shown, including:

[0046] 1) Switchable feature extractor to output state features of the environment;

[0047] 2) Multiple agents, including boundary agents and transition agents, generate corresponding state values ​​and action strategies;

[0048] 3) A correlation-oriented reward generator that assigns corresponding rewards to multiple agents.

[0049] Specifically, video clip retrieval is regarded as a sequential decision-making process involving multiple agents, and the retrieval of the target clip is achieved through the following iterative process.

[0050] First, an initial retrieval segment is determined in the reference video. ,in and Represent the starting boundary and the ending boundary respectively.

[0051] After that, the multi-agent gradually updates the initial state. At each iterative time step, the feature extractor can be transformed to output the current state of the environment for the multi-agent to generate an action strategy. The multi-agent takes actions according to the corresponding action strategy, where the transformation agent selects an encoder for the video from the candidate encoder set, and the boundary agent adjusts the time boundary of the current retrieval segment.

[0052] At the same time, the correlation-oriented reward generator assigns corresponding rewards to multiple agents, reflecting the effects of the actions performed. max When , the iteration process terminates.

[0053] The various components of the present invention as well as the training and reasoning stages are described in detail below.

[0054] 1. Convertible Feature Extractor:

[0055] For the switchable feature encoder, its input is a query video and a reference video pair. These video pairs are encoded as the current state features of the environment. Here the environment refers to the set of query video, reference video and current positioning segment.

[0056] Query Video V Q And reference video V R Through the backbone network, the pre-trained C3D network is converted into the corresponding feature sequence, which is represented as and , where l q and l r represents the length of the query video and the reference video, d f represents the dimension of each feature vector. At each time step t, the reference video is divided into three parts: the left segment, the current segment, and the right segment. The feature sequences of these three segments are obtained by selecting the corresponding parts from the feature sequence of the reference video, and are represented as , and .

[0057] Next, the video features are enhanced using a feature encoder. Specifically, the query video uses a fixed encoder, such as 1D convolution, while the reference video and its three segments use a transformable encoder. Figure 1 As shown, the convertible encoder includes n sub-encoders e 1, e 2, e3, ... e n Each sub-encoder has a unique convolution kernel and receptive field, capturing different ranges of temporal information. At each time step, the conversion agent controls the convertible encoder to select a specific sub-encoder for video feature encoding. This encoder is described by the following formula:

[0058] (1);

[0059] Where x represents the input of the transformable encoder, SA represents the transformable agent, and s t Indicates state characteristics.

[0060] Afterwards, the feature sequences of the query video, reference video, left segment, current segment, and right segment are averaged and pooled to obtain fixed-length features, which are represented as f q 、f r , , and These features are used as the input of the state encoding, which makes full use of the context information of the video. In addition, in order to explicitly represent the state of the current retrieval segment, its normalized temporal boundary , t=1,2,...T max (called position feature) is also used for state encoding, where N represents the length of the reference video. The video feature and position feature are concatenated and input into a fully connected layer Φ to obtain the final state feature, which is expressed as:

[0061] (2).

[0062] 2. Transition and Boundary Agents:

[0063] The present invention designs multi-agents based on Advantage Actor-Critic (A2C): conversion agents and boundary agents. Figure 1 As shown in Figure 1, each agent has two fully connected layer branches, the actor branch and the critic branch, which share a GRU layer. At each time step, the input of the GRU is the state features generated by the transformable encoder, and its hidden state is used for policy generation and state value estimation. Specifically, the actor branch outputs the action policy , which represents the probability distribution of all actions in the corresponding action space, and the critic branch outputs the state value , which represents the estimated value of the current state cumulative reward. Where i∈{boundary, switch}, and Agents The actor branch parameters and critic branch parameters of .

[0064] Each agent selects actions from the corresponding action space according to the learned policy. The action space of the conversion agent contains n actions: select the sub-encoder e in the conversion encoder i , where i∈{1,2,...n}. For example, if the switching agent takes the action of “selecting a sub-encoder e in the switchable encoder” at time step t 1 ", then in the next time step, e 1 will be used as the feature encoder. The action space of the boundary agent contains 4 actions: 1) move the starting boundary forward by δ; 2) move the starting boundary backward by δ; 3) move the ending boundary forward by δ; 4) move the ending boundary backward by δ, where δ is set to N / 10. For example, if the boundary agent takes the action of "move the starting boundary forward by δ" at time step t, then the time boundary of the retrieved segment at the next time step will be moved from becomes .

[0065] 3. Correlation-oriented reward generator:

[0066] The relevance-oriented reward generator is used to assign rewards to multi-agents to encourage them to identify segments that are more relevant to the query. First, the relevance between the query video and the retrieved segments is estimated to obtain the corresponding similarity scores. Then, corresponding rewards are assigned to multi-agents based on the similarity scores.

[0067] like Figure 1 As shown, given the query video and the feature sequence of the retrieval segment and , first use the BiLSTM layer to perform bidirectional sequence encoding. Then, in order to highlight the frames containing rich information, the attention weighted features of the query video and the retrieved fragment are calculated. The specific calculation process is expressed as follows:

[0068] (3);

[0069] The similarity score between the query video and the retrieval segment is used to assign corresponding rewards to the multi-agents. The goal of the boundary agent is to refine the temporal boundary of the current retrieval segment to make it closer to the target segment, that is, more relevant to the query video. Therefore, if the similarity score between the retrieval segment and the query video increases after performing the action at the tth time step, the boundary agent will receive a positive reward τ. On the contrary, if the similarity score decreases, the boundary agent will receive a negative reward of -0.1×τ. In addition, if the starting boundary is after the ending boundary, the boundary agent will receive a greater degree of negative reward -τ. In general, the reward obtained by the boundary agent at the tth time step is defined as:

[0070] (4);

[0071] Where: τ is a factor that determines the size of the reward. The goal of the transition agent is to obtain better state features to help the boundary agent take better actions. Therefore, the reward of the transition agent at time step t is obtained based on the reward of the boundary agent at time step t+1, expressed as:

[0072] (5);

[0073] Considering the sequential decision-making process, it is necessary to trace the rewards of subsequent time steps back to the current time step. Therefore, the cumulative reward of the agent at the tth time step is calculated as:

[0074] (6);

[0075] Among them, γ is the discount factor.

[0076] 4. Training process:

[0077] The present invention uses the A2C algorithm to optimize multi-agents, and the corresponding loss is expressed as L A2C At the same time, a combined ranking loss is designed , to optimize the reward generator for correlation.

[0078] (1) A2C loss

[0079] During the training phase, the multi-agent system takes T max The optimization goal of the actor branch is to maximize the expected advantage, and its loss is defined as:

[0080] (7);

[0081] in, represents the advantage function, which measures how good it is to perform an action in a given state compared to the average situation; H(·) represents the entropy of the action strategy, which is introduced into the loss to enhance the agent’s exploration ability.

[0082] The optimization goal of the critic branch is to minimize the mean squared error (MSE) between the cumulative return and the estimated state value, and its loss is defined as:

[0083] (8);

[0084] The overall loss of the A2C algorithm is the sum of the losses of the actor branch and the critic branch:

[0085] (9).

[0086] (2) Combination sorting loss:

[0087] The present invention designs a combined ranking loss to optimize a relevance-oriented reward generator, which includes an inter-video ranking loss and an intra-video ranking loss.

[0088] For the inter-video ranking loss, the matching information between the query video and the reference video is used. This loss constrains the relevance score of the matched query-reference video pairs to be higher than that of the unmatched query-reference video pairs, and is defined as:

[0089] (10);

[0090] Where S(·,·) represents the correlation score; (V Q ,V R ) represents a matched query-reference video pair; and represents unmatched query-reference video pairs; and corresponds to all unmatched instances in the current batch; μ represents an interval value to ensure that there is a certain gap between the relevance scores of matching pairs and unmatched pairs.

[0091] The goal of the intra-video ranking loss is to distinguish different contents within the same video. At each time step, the reference video is divided into three parts: the left segment, the current segment, and the right segment. Since there is no target segment annotation in the weakly supervised setting, the proximity of these three segments to the target segment, that is, their relevance to the query video, is unknown. Therefore, in order to distinguish the three segments, the following assumption is made: if the relevance score of a query-segment pair exceeds that of the query-reference pair, then the pair should have a higher score than other pairs. To simplify the representation, the relevance score of the query-reference pair is denoted as S R, the relevance scores of the three query-fragment pairs are recorded as , and The intra-video ranking loss is defined as:

[0092] (11);

[0093] in, Indicates that From the collection ; φ(·) is a binary indicator function that outputs 1 when the inequality in the brackets holds, otherwise it outputs 0.

[0094] The combined ranking loss is a combination of the inter-video and intra-video ranking losses, expressed as:

[0095] (12);

[0096] Here, λ represents the weight factor, which is used to weigh the inter-video and intra-video ranking losses.

[0097] (3) Alternating training:

[0098] In the present invention, the network training process follows an alternating optimization mechanism. For each set of 2K iterations, the parameters of the transformable feature extractor and the multi-agent are first fixed, and L rank To optimize the model, we ensure that the initial rewards of the multi-agents are reliable. When we reach K iterations, we fix the parameters of the correlation-oriented reward generator and use To optimize the model. At the same time, the conversion agent and the boundary agent are also optimized alternately. Specifically, for a total of 2T training rounds, refers to , and then in T rounds refers to .

[0099] 5. Testing process:

[0100] In the test phase, the multi-agent selects actions through a greedy strategy to guide video feature encoding and segment boundary positioning, that is, the action selected by agent i at time step t is The inter-video relevance scores calculated in the relevance-oriented reward generator are used to determine the final retrieval segment. Specifically, the multi-agent performs T max After the steps, a series of segments and corresponding relevance scores are obtained. These scores reflect the relevance between the segment and the query video. Therefore, the segment with the highest relevance score is selected as the final retrieval result.

[0101] Furthermore, the ActivityNet-VMR dataset is used to train and test the algorithm. The dataset contains 9530 uncut videos, each of which contains only one action clip. The dataset is divided according to action categories, where the training set contains 160 action categories, while the validation set and test set contain 20 action categories respectively. In the video clip retrieval task, each set of data consists of a pair of query videos and reference videos. The action clip in each video is used as a query video, and the paired reference video is selected from the uncut videos with the same action category. In the training phase, the query video and the reference video are randomly matched, while in the validation and testing phases, the pairing of the query video and the reference video is fixed.

[0102] The present invention uses the video features released by the 2016 ActivityNet Challenge, which are extracted by the pre-trained C3D network, downsamples the C3D features to ensure that there is no temporal overlap, and projects the feature dimensions to 500 dimensions through principal component analysis (PCA). The initial retrieval segment is determined based on the C3D features. Specifically, the similarity between each reference feature vector and the average query feature is calculated, and the video segment with the highest similarity is selected as the initial retrieval segment. The convertible encoder includes 5 sub-encoders, whose convolution kernel sizes are {1, 3, 5, 7, 9}.

[0103] In the training phase, the present invention uses the Adam optimizer, and the batch size and learning rate are set to 16 and 1e-4 respectively. max is set to 10. The factor τ in the reward function is set to 1, and the interval value μ in the ranking loss is set to 0.2. The hyperparameters γ, α, η, and λ are set to 0.4, 0.1, 0.1, and 1, respectively. In alternating optimization, K is 200 and T is 100.

[0104] Furthermore, the performance of the present invention is compared with other existing methods, including weakly supervised methods and fully supervised methods. The performance evaluation index is mean average precision (mAP), and the tIoU threshold is set to {0.5, 0.6, 0.7, 0.8, 0.9}. The mAP under different tIoU thresholds is calculated respectively, and then the average of these mAPs is calculated. The performance comparison results are shown in Table 1. First, compared with other weakly supervised methods, the present invention achieves better results than others. Second, the results of the present invention exceed those of some fully supervised methods.

[0105] Table 1 Performance comparison results of the present invention and other methods

[0106]

[0107] Although the embodiments and drawings of the present invention are disclosed for illustrative purposes, those skilled in the art will appreciate that various substitutions, changes and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the scope of the present invention is not limited to the contents disclosed in the embodiments and drawings.

Claims

1. A weakly supervised video clip retrieval method based on reinforcement learning, characterized in that: The steps of the method are: S1, a switchable feature extractor generates state features of the current environment based on the input video data; S2, the multi-agent generates corresponding action strategies and state values ​​based on the state features of S1, adjusts the time boundary of the current retrieval segment, and selects a feature encoder for the video from the candidate encoder set; S3. The relevance-oriented reward generator calculates the similarity scores between the query video and the retrieved segments and assigns corresponding rewards to the multi-agents.

2. The weakly supervised video segment retrieval method based on reinforcement learning according to claim 1, characterized in that: The S1 is specifically: 1) The video data is the query video V Q And reference video V R , the current environment is a collection of the query video, the reference video and the current positioning segment, the query video V Q And reference video V R Transformed into corresponding feature sequences through the backbone network, respectively expressed as and , where l q and l r represents the length of the query video and the reference video, d f represents the dimension of each feature vector; 2) At each time step t, the reference video V R It is divided into three parts: the left segment, the current segment and the right segment. The feature sequences of the above three segments are obtained by selecting the corresponding parts from the reference video feature sequence, which are respectively expressed as , and ; 3) Enhance the video features using feature encoders. The query video uses a fixed encoder, and the reference video and its three segments use a convertible encoder, which contains n sub-encoders e 1, e 2, e 3,... e n , at each time step, the conversion agent controls the convertible encoder to select a specific sub-encoder for video feature encoding, and the specific sub-encoder is described by the following formula: （1）； Where: x represents the input of the transformable encoder, SA represents the transformable agent, and s t Indicates state characteristics; 4) Perform average pooling on the feature sequences of the query video, reference video, left segment, current segment, and right segment to obtain fixed-length features, denoted as f q 、f r , , and These features are used as the input of the state encoding. In order to clearly represent the state of the current retrieval fragment, the time boundary of the normalization , t=1,2,...T max That is, the position feature is also used for state encoding, where N represents the length of the reference video; the video feature and the position feature are concatenated and input into a fully connected layer Φ to obtain the final state feature, which is expressed as: （2）。 3. The weakly supervised video segment retrieval method based on reinforcement learning according to claim 2, characterized in that: The S2 is specifically: 1) The multi-agent includes a boundary agent and a transition agent. Each agent has two fully connected layer branches, namely the actor branch and the critic branch. They share a GRU layer. At each time step, the input of the GRU is the state feature generated by the transition encoder, and its hidden state is used for strategy generation and state value estimation. The actor branch outputs the action strategy , which represents the probability distribution of all actions in the corresponding action space, and the critic branch outputs the state value , which represents the estimated value of the cumulative return of the current state, where i∈{boundary, switch}, and are the actor branch parameters and critic branch parameters of agent i respectively; 2) Each agent selects an action from the corresponding action space according to the learned strategy. The action space of the conversion agent contains n actions: select the sub-encoder e in the convertible encoder i , where i∈{1,2,...n}; The action space of the boundary agent contains 4 actions: 1) move the starting boundary forward by δ; 2) move the starting boundary backward by δ; 3) move the ending boundary forward by δ; 4) move the ending boundary backward by δ, and set δ to N / 10.

4. The weakly supervised video segment retrieval method based on reinforcement learning according to claim 2, characterized in that: The S3 is specifically: 1) Estimate the correlation between the query video and the retrieval clip and obtain the corresponding similarity score; Given a query video and a feature sequence of retrieved clips and , use the BiLSTM layer to perform bidirectional sequence encoding, calculate the attention weighted features of the query video and the retrieval segment, and the specific calculation process is expressed as follows: （3）； 2) Based on the similarity scores, corresponding rewards are assigned to multiple agents. The reward obtained by the boundary agent at the t time step is defined as: （4）； Among them: τ is the factor that determines the size of the return; The reward of the transition agent at time step t is obtained based on the reward of the boundary agent at time step t+1, expressed as: （5）； Considering the sequential decision-making process, it is necessary to trace the rewards of subsequent time steps back to the current time step. The cumulative rewards of multiple agents at the tth time step are expressed as: （6）； Among them, γ is the discount factor.

5. A weakly supervised video clip retrieval system based on reinforcement learning, characterized in that: Executing the weakly supervised video clip retrieval method based on reinforcement learning as claimed in any one of claims 1 to 4, comprising: A switchable feature extractor generates state features of the current environment based on the input video data; Multiple agents, namely boundary agents and conversion agents, generate corresponding action strategies and state values ​​based on state features, where the boundary agent adjusts the time boundary of the current retrieval segment, and the conversion agent selects a feature encoder for the video from a set of candidate encoders; A relevance-oriented reward generator calculates similarity scores between query videos and retrieved segments, and based on this, assigns corresponding rewards to multiple agents to measure the effectiveness of the actions they perform.

Images