Method and system for generating candidate sets of video action segments based on common sense enhancement

Abstract

The present invention belongs to the technical field of video behavior segment retrieval. A method and system for generating a candidate set of video behavior segments based on common sense enhancement is provided, which obtains the semantic feature representation of the query statement and the video to be retrieved; interacts with the semantic representation of the visual feature and the semantic representation of the text feature through a cross-modal interaction module to fuse multimodal information; predicts the probability of each video unit being retained, and retains the video units with high information validity based on the probability; maps the features of the retained video frames to the input space of the large image-text pre-training model through the visual adaptation layer; fine-tunes the large image-text pre-training model by inserting the adaptation layer, and constructs an instruction to instruct the model to complete the task of generating a candidate set of video behavior segments. The present invention introduces a large image-text pre-training model to utilize the rich external knowledge therein to improve the understanding of the visual content, while taking into account the generation speed and accuracy of the candidate set of video behavior segments.

Description

Technical Field

[0001] The present invention relates to the technical field of video behavior segment retrieval, and in particular to a method and system for generating a candidate set of video behavior segments based on common sense enhancement. Background Art

[0002] The statements in this section merely provide background art related to the present invention and do not necessarily constitute prior art.

[0003] An efficient and reliable strategy for generating candidate video action segments can ensure the real-time and accuracy of the system, which is crucial for achieving fast and accurate retrieval of video action segments. Most early research works directly use multi-scale sliding windows to generate candidate sets of video action segments. Although this type of method is simple in design, it cannot avoid redundant calculations of overlapping parts between windows, thus reducing the efficiency of retrieval of video action segments and making it difficult to apply in actual monitoring systems.

[0004] In order to reduce redundant calculations, the video unit-based method divides the video into several non-overlapping video units according to predefined time intervals, evaluates the confidence of all video units through multiple methods, and generates a candidate set of video behavior segments with flexible temporal boundaries according to the confidence of the video units through a multi-threshold clustering algorithm; however, although the above method avoids redundant calculations between candidate video behavior segments, end-to-end training cannot be achieved due to the introduction of multi-threshold clustering.

[0005] The anchor-based method further regards each video unit as an anchor point, and simultaneously predicts the confidence of multiple candidate video behavior segments of different scales centered on the anchor point at each anchor point. In each round of iteration, only a single traversal of the video unit is required, so it can be trained end-to-end, greatly improving the efficiency of training and retrieval. At the same time, in order to further increase the flexibility of candidate set generation, the generative network-based method designs a variety of different network structures to sample and merge video units, but its accuracy still cannot meet the requirements.

[0006] In summary, although existing research work has made some progress in the generation of candidate video behavior segments, it still faces the following problems:

[0007] (1) The generation of candidate sets of video action segments faces a trade-off between efficiency and accuracy. On the one hand, the quality of candidate set generation determines the upper limit of positioning accuracy. Therefore, most existing methods improve positioning accuracy by generating dense multi-scale candidate segments. However, highly overlapping candidate segments will lead to a large amount of redundant calculations, thereby reducing positioning efficiency. On the other hand, some methods choose to directly predict the start and end time of the target segment to improve positioning efficiency. This strategy avoids the generation of redundant candidate segments, thereby saving computing time. However, since judging the start and end time is more challenging than evaluating the confidence of the candidate segments, its positioning accuracy is low.

[0008] (2) The generation of candidate sets of video behavior segments faces the challenge of applying common sense knowledge. On the one hand, the meaning and importance of many behavior segments may not be intuitive and require the combination of common sense knowledge to be correctly understood and evaluated. On the other hand, different cultural backgrounds and individual experiences may lead to different interpretations of the same behavior, which increases the difficulty of generating accurate candidate sets of behavior segments. In addition, common sense knowledge is usually difficult to learn directly from video data and requires additional external knowledge or prior information to assist, which further increases the complexity of generating candidate sets of behavior segments. Summary of the invention

[0009] In order to address the deficiencies of the prior art, the present invention provides a method and system for generating a candidate set of video behavior segments based on common sense enhancement, which improves the accuracy of generating a candidate set of video behavior segments while ensuring retrieval efficiency.

[0010] In order to achieve the above object, the present invention adopts the following technical solution:

[0011] In a first aspect, the present invention provides a method for generating a candidate set of video behavior segments based on common sense enhancement.

[0012] A method for generating a candidate set of video action segments based on common sense enhancement includes the following steps:

[0013] The video to be retrieved is divided into continuous but non-overlapping video units, and the semantic feature representation of the query sentence and the semantic feature representation of each video unit are obtained;

[0014] Mapping the semantic feature representation of the query statement to a cross-modal common semantic space to obtain a modality-aligned semantic feature representation of the query statement, and mapping the semantic feature representation of the video unit to a cross-modal common semantic space to obtain a modality-aligned semantic feature representation of the video unit;

[0015] The semantic feature representation of the modality-aligned query statement is interacted with the semantic feature representation of the modality-aligned video unit to obtain the semantic feature representation of the video unit fused with text information, which is used to predict the probability of each video unit being retained;

[0016] According to the probability of video units being retained, the video units are sorted and the top video units, the previous The semantic feature representation of the fused text information of each video unit is mapped to the input space of the pre-trained model, and is concatenated with the semantic feature representation of the query sentence aligned with the modality and the preset instructions as the input of the pre-trained model to obtain a candidate set of video behavior segments.

[0017] As a further limitation of the first aspect of the present invention, the semantic feature representation of the query sentence aligned with the modality is interacted with the semantic feature representation of the video unit aligned with the modality to obtain the semantic feature representation of the video unit integrating the text information, including:

[0018] Computing cross-modal attention scores for visual and textual semantic feature representations via cosine similarity , the corresponding text information is aggregated for each video unit through the cross-modal attention score, and multi-modal information is fused: ,in, To represent the semantic features of video units that integrate text information, The semantic feature representation of video units aligned with modality, It is the semantic feature representation of query sentences aligned with modality, is the cross-modal attention score The transpose of represents the set of real numbers, represents the dimension of the cross-modal common semantic space, Indicates the number of video units.

[0019] As a further limitation of the first aspect of the present invention, the prediction layer is used to predict the probability of each video unit being retained. ,include: ,in, is the weight matrix of the prediction layer, is the activation function.

[0020] As a further limitation of the first aspect of the present invention, generating a supervisory signal according to a timestamp of a target video behavior segment, and calculating a loss function to train a pre-trained model, comprises:

[0021] Calculate the intersection and union ratio of the timestamp of each video unit and the timestamp of the target video behavior segment;

[0022] When When the IoU ratio of the video unit is greater than the specified IoU threshold, The label assigned to the i-th video unit is 1, indicating that the visual content of the i-th video unit matches the query text, and it is determined to be a positive video unit at this time. Otherwise, the label assigned is 0, and it is determined to be a negative video unit at this time.

[0023] Balanced cross entropy loss is used to calculate the model loss size.

[0024] As a further limitation of the first aspect of the present invention, the balanced cross entropy loss is used to calculate the model loss size, including:

[0025] Defining the median probability : ,in, It is the predicted The probability that a video unit is retained, This is for The labels assigned to the video units are used to calculate the balanced cross entropy loss: ,in, is the weight of the video unit, which is inversely proportional to the number of positive and negative video units.

[0026] As a further limitation of the first aspect of the present invention, the retained The semantic feature representation of the fused text information of each video unit is mapped to the input space of the pre-trained model, including: ,in, It is the visual semantic feature representation of the retained video unit after being aligned to the input space of the large pre-trained image and text model. are the trainable weights of the visual adaptation layer.

[0027] As a further limitation of the first aspect of the present invention, the preset instruction is: determine whether the content in each provided video unit is consistent with the text description in the query sentence, and output yes or no for each video unit.

[0028] As a further limitation of the first aspect of the present invention, the image and text pre-training large model is lightly fine-tuned through an adaptation layer to adapt to the task of generating a candidate set of video behavior segments, and the adapter adopts a bottleneck structure, including: ,in, is the intermediate hidden state matrix output by the intermediate layer of the image and text, is a nonlinear activation function, and They are the downsampling and upsampling mapping matrices of the bottleneck structure, is the intermediate dimension of the bottleneck structure, Indicates the text length of the query statement. represents the dimension of the cross-modal common semantic space, Indicates the length of the instruction after word segmentation.

[0029] As a further limitation of the first aspect of the present invention, the probability that the content in each video unit prediction matches the text description in the query sentence is obtained according to the image-text pre-training large model. , according to the probability sequence, a candidate set of video behavior segments is obtained by using a watershed algorithm ,in, and Respectively represent the start and end time of the first candidate video segment, and Respectively represent the start and end time of the second candidate video segment, and Respectively represent The start and end times of the candidate video segments, Indicates the number of candidate video segments finally generated.

[0030] In a second aspect, the present invention provides a system for generating a candidate set of video behavior segments based on common sense enhancement.

[0031] A video action segment candidate set generation system based on common sense enhancement, comprising:

[0032] The semantic feature representation generating unit is configured to: divide the video to be retrieved into continuous but non-overlapping video units, obtain the semantic feature representation of the query sentence and the semantic feature representation of each video unit;

[0033] A modality alignment unit is configured to: map the semantic feature representation of the query statement to a cross-modality common semantic space to obtain a modality-aligned semantic feature representation of the query statement, and map the semantic feature representation of the video unit to a cross-modality common semantic space to obtain a modality-aligned semantic feature representation of the video unit;

[0034] A feature fusion unit is configured to: interact the semantic feature representation of the modality-aligned query sentence with the semantic feature representation of the modality-aligned video unit to obtain the semantic feature representation of the video unit fused with the text information, which is used to predict the probability of each video unit being retained;

[0035] The candidate set generation unit is configured to: sort the video units according to the probability of the video units being retained, and retain the top video units, the previous The semantic feature representation of the fused text information of each video unit is mapped to the input space of the pre-trained model, and is concatenated with the semantic feature representation of the query sentence aligned with the modality and the preset instructions as the input of the pre-trained model to obtain a candidate set of video behavior segments.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] 1. The present invention innovatively proposes a method and system for generating a candidate set of video behavior segments based on common sense enhancement, including a dynamic sampling process of video units based on query sentences and a process for generating a candidate set of video behavior segments based on a large pre-trained model of graphics and text, while taking into account the generation speed and accuracy of the candidate set of video behavior segments.

[0038] 2. The query-based video unit dynamic sampling process in the present invention can perform fine-grained cross-modal interaction between the query and the video to be retrieved, and preliminarily evaluate the relevance between each video unit and the query, thereby effectively screening out more representative video unit semantic feature representations.

[0039] 3 The video behavior segment candidate set generation process based on the image-text pre-training large model in the present invention can make full use of the rich common sense knowledge in the image-text pre-training large model, improve the behavior understanding ability based on the image-text pre-training large model, and thus improve the generation accuracy of the video behavior segment candidate set.

[0040] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0042] Figure 1 This is a flow chart of a method for generating a candidate set of video behavior segments based on common sense enhancement in Example 1 of the present invention;

[0043] Figure 2 Schematic diagram of a video unit dynamic sampling module based on a query statement in Embodiment 1 of the present invention;

[0044] Figure 3 Schematic diagram of a module for generating a candidate set of video behavior segments based on a large image-text pre-training model in Embodiment 1 of the present invention;

[0045] Figure 4 Schematic diagram of the adaptation layer insertion method in Example 1 of the present invention. DETAILED DESCRIPTION

[0046] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0047] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0048] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0049] Embodiment 1:

[0050] This implementation proposes a method for generating candidate sets of video action segments based on common sense enhancement, such as Figure 1 As shown in the figure, the query sentence is "the man in the red shirt crossed the road", and the video to be retrieved is divided into continuous but non-overlapping video units. The video units are dynamically sampled based on the query sentence to obtain key video frames, and then the final video behavior segment candidate set is obtained through the large image and text pre-training model.

[0051] More specifically, the process includes the following:

[0052] S1: The query-based video unit dynamic sampling module samples the query-related parts from the video and filters out the noise information irrelevant to the query, thereby reducing the data input to achieve the purpose of reducing the calculation amount of subsequent modules, such as Figure 2 shown.

[0053] S1.1: Obtain the semantic feature representation of the query statement and the video to be retrieved.

[0054] S1.1.1: Use the pre-trained RoBERTa model as a text feature extractor to extract the semantic feature representation of the query statement ;in, Indicates the text length of the query statement. Represents the dimension of the text feature space;

[0055] S1.1.2: Divide the video to be retrieved into continuous but non-overlapping video units;

[0056] S1.1.3: Use the pre-trained Video-MAE model as a visual feature extractor to extract the semantic feature representation of video units ;in, Indicates the number of video units, Represents the dimension of the visual feature space;

[0057] S1.1.4: Use a multilayer perceptron (MLP) as the text mapping layer to map the text semantic feature representation into a cross-modal common semantic space to obtain a modality-aligned query statement semantic feature representation. ;in, Represents the dimensionality of the cross-modal common semantic space;

[0058] S1.1.5: Use MLP as the visual mapping layer to map the visual semantic feature representation into the cross-modal common semantic space to obtain the modality-aligned query statement semantic feature representation .

[0059] S1.2: The visual feature semantic representation and the text feature semantic representation are interacted with each other through the cross-modal interaction module to fuse multimodal information.

[0060] S1.2.1: Compute cross-modal attention scores for visual and textual semantic feature representations via cosine similarity :

[0061] (1);

[0062] in, It indicates the relevance between each word in the query and each video unit in the video to be retrieved. is the normalized exponential function, represents the set of real numbers, Represents the modulo operation.

[0063] S1.2.2: Aggregate the corresponding text information for each video unit through cross-modal attention scores to achieve multimodal information fusion. The specific process is as follows:

[0064] (2);

[0065] in, is the semantic feature representation of the video unit that integrates text information. yes The transpose of .

[0066] S1.3: The probability of each video unit being retained is predicted through the prediction layer. The specific process is as follows:

[0067] (3);

[0068] in, is the weight matrix of the prediction layer, is the probability of each video unit being retained, is the activation function.

[0069] S1.4: Generate supervision signals based on the timestamps of the target video action segments and calculate the loss function to train the large image-text pre-training model.

[0070] S1.4.1: Calculate the timestamp of each video unit The timestamp of the target video segment Intersection of Union (IoU), the specific process is as follows:

[0071] (4);

[0072] in, It is The IoU between the video unit and the target video action segment, and It is The start and end times of each video unit. and are the start and end times of the target video behavior segment.

[0073] S1.4.2: Specify the intersection-over-union ratio threshold as 0.5, and assign labels to each video unit as follows:

[0074] (5);

[0075] in, This is for The label assigned to each video unit is 1, which means that the video unit is a positive example and its visual content matches the query text. Otherwise, the label is 0.

[0076] S1.4.3: Since the number of positive and negative video units is extremely unbalanced, balanced cross entropy loss is used to calculate the model loss size.

[0077] S1.4.3.1: First define the probability median as follows:

[0078] (6);

[0079] in, It is the predicted The probability that a video unit is retained.

[0080] S1.4.3.2: Calculate the balanced cross entropy loss as follows:

[0081] (7);

[0082] in, is the weight of the video unit, which is inversely proportional to the number of positive and negative video units.

[0083] S2: Figure 3As shown, the video behavior segment localization task is converted into a form that can be processed by the pre-trained model to utilize the rich external knowledge therein and generate a more accurate set of video behavior segment candidates.

[0084] S2.1: Probability vector of video unit being retained , sort the video units and keep the first Video units.

[0085] S2.2: The video unit sampler obtains high-confidence video units (i.e., the retained video units), and the visual adaptation layer maps the visual semantic feature representation of the retained video units to the input space of the pre-trained model. The specific process is as follows:

[0086] (8);

[0087] in, It is the visual semantic feature representation of the retained video unit after being aligned to the input space of the large pre-trained image and text model. are the trainable weights of the visual adaptation layer.

[0088] The insertion method of the adaptation layer of this implementation is as follows Figure 4 As shown in the figure, the output of the multi-head self-attention module passes through the first adaptation layer and is superimposed with the input of the multi-head self-attention module as the input of the first normalization layer. The output of the first normalization layer is used as the input of the feedforward layer. The output of the feedforward layer is used as the input of the second adaptation layer. The output of the second adaptation layer is superimposed with the input of the feedforward layer and used as the input of the second normalization layer.

[0089] S2.3: To instruct the large pre-trained model to generate a candidate set of video action segments, construct an instruction sentence for the model input: "Please determine whether the content of each video segment provided matches the text description in the question. Please answer a <yes> or <no> for each video segment in order." Extract the semantic feature representation of the instruction as described in S1.1.1. ,in It is the length of the instruction after word segmentation.

[0090] S2.4: Splicing preset instructions , modality-aligned query semantic feature representation , the visual semantic feature representation of the retained video unit after alignment to the input space of the large image and text pre-training model As the final input of the large image and text pre-training model, the specific process is as follows:

[0091] (9);

[0092] in, It is the final input of the spliced ​​image and text pre-training large model. Represents a dimension concatenation operation.

[0093] S2.5: The image and text pre-training large model is lightly fine-tuned through the adaptation layer to adapt to the task of generating candidate sets of video behavior segments. The adapter adopts a bottleneck structure to reduce the amount of calculation. The formal process is as follows:

[0094] (10);

[0095] in, is the intermediate hidden state matrix output by the intermediate layer of the image and text, is a nonlinear activation function, and They are the downsampling and upsampling mapping matrices of the bottleneck structure, is the intermediate dimension of the bottleneck structure.

[0096] S2.6: Predict the probability of <yes> for each video unit based on the large image and text pre-training model to obtain a probability sequence Then, the video action segment candidate set is obtained through the watershed algorithm according to the probability ,in, and Respectively represent the start and end time of the first candidate video segment, and Respectively represent the start and end time of the second candidate video segment, and Respectively represent The start and end times of the candidate video segments, Indicates the number of candidate video segments finally generated.

[0097] Embodiment 2:

[0098] This implementation provides a video action segment candidate set generation system based on common sense enhancement, including:

[0099] The semantic feature representation generating unit is configured to: divide the video to be retrieved into continuous but non-overlapping video units, obtain the semantic feature representation of the query sentence and the semantic feature representation of each video unit;

[0100] A modality alignment unit is configured to: map the semantic feature representation of the query statement to a cross-modality common semantic space to obtain a modality-aligned semantic feature representation of the query statement, and map the semantic feature representation of the video unit to a cross-modality common semantic space to obtain a modality-aligned semantic feature representation of the video unit;

[0101] A feature fusion unit is configured to: interact the semantic feature representation of the modality-aligned query sentence with the semantic feature representation of the modality-aligned video unit to obtain the semantic feature representation of the video unit fused with the text information, which is used to predict the probability of each video unit being retained;

[0102] The candidate set generation unit is configured to: sort the video units according to the probability of the video units being retained, and retain the top video units, the previous The semantic feature representation of the fused text information of each video unit is mapped to the input space of the pre-trained model, and is concatenated with the semantic feature representation of the query sentence aligned with the modality and the preset instructions as the input of the pre-trained model to obtain a candidate set of video behavior segments.

[0103] The specific working methods of the above-mentioned units are described in Example 1 and will not be repeated here.

[0104] It is understandable that the above-mentioned units can be separately or completely combined into one or several other units to constitute, or some of the units can be further divided into multiple functionally smaller units to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the functions of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the video behavior segment candidate set generation system can also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0105] According to another embodiment of the present application, the system described in this embodiment and the method for generating a candidate set of video behavior segments of the embodiment of the present application can be constructed by running a computer program (including program code) capable of executing the steps involved in the corresponding method described in Example 1 on a general-purpose computing device such as a computer including processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM). The computer program can be recorded on, for example, a computer-readable recording medium, and loaded into the above-mentioned computing device through the computer-readable recording medium and run therein.

[0106] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for generating a candidate set of video action segments based on common sense enhancement, characterized in that: The process includes: The video to be retrieved is divided into continuous but non-overlapping video units, and the semantic feature representation of the query sentence and the semantic feature representation of each video unit are obtained; Mapping the semantic feature representation of the query statement to a cross-modal common semantic space to obtain a modality-aligned semantic feature representation of the query statement, and mapping the semantic feature representation of the video unit to a cross-modal common semantic space to obtain a modality-aligned semantic feature representation of the video unit; The semantic feature representation of the modality-aligned query statement is interacted with the semantic feature representation of the modality-aligned video unit to obtain the semantic feature representation of the video unit fused with text information, which is used to predict the probability of each video unit being retained. The specific process is as follows: Computing cross-modal attention scores for visual and textual semantic feature representations via cosine similarity , the corresponding text information is aggregated for each video unit through the cross-modal attention score, and multi-modal information is fused: ,in, To represent the semantic features of video units that integrate text information, The semantic feature representation of video units aligned with modality, It is the semantic feature representation of query sentences aligned with modality, is the cross-modal attention score The transpose of represents the set of real numbers, represents the dimension of the cross-modal common semantic space, Represents the number of video units; the prediction layer is used to predict the probability of each video unit being retained ,include: ,in, is the weight matrix of the prediction layer, is the activation function; According to the probability of video units being retained, the video units are sorted and the top video units, the previous The semantic feature representation of the fused text information of the video units is mapped to the input space of the pre-trained model, and the semantic feature representation of the query sentence aligned with the modality and the preset instruction are spliced ​​as the input of the pre-trained model to obtain a candidate set of video behavior segments, wherein the reserved previous The specific way to map the semantic feature representation of the fused text information of a video unit to the input space of the pre-trained model is as follows: ,in, It is the visual semantic feature representation of the retained video unit after being aligned to the input space of the large pre-trained image and text model. are the trainable weights of the visual adaptation layer, To represent the semantic features of video units that integrate text information, represents the dimension of the cross-modal common semantic space, Indicates the number of video units, Represents the set of real numbers.

2. The method for generating a candidate set of video action segments based on common sense enhancement according to any one of claim 1, characterized in that: Generate supervision signals based on the timestamps of the target video action segments and calculate the loss function to train the pre-trained model, including: Calculate the intersection and union ratio of the timestamp of each video unit and the timestamp of the target video behavior segment; When i When the IoU ratio of the video unit is greater than the specified IoU threshold, The label assigned to the video unit is 1, indicating that i If the visual content of a video unit matches the query text, it is determined to be a positive video unit. Otherwise, the assigned label is 0, which means it is a negative video unit. Balanced cross entropy loss is used to calculate the model loss size.

3. The method for generating a candidate set of video action segments based on common sense enhancement as claimed in claim 2, characterized in that: The balanced cross entropy loss is used to calculate the model loss size, including: Defining the median probability : ,in, It is the predicted The probability that a video unit is retained, This is for The labels assigned to the video units are used to calculate the balanced cross entropy loss: ,in, is the weight of the video unit, which is inversely proportional to the number of positive and negative video units.

4. The method for generating a candidate set of video action segments based on common sense enhancement as claimed in claim 1, characterized in that: The preset instruction is: determine whether the content in each provided video unit is consistent with the text description in the query sentence, and output yes or no for each video unit.

5. The method for generating a candidate set of video action segments based on common sense enhancement as claimed in claim 1, characterized in that: The adapter layer is used to fine-tune the large pre-trained image and text model to adapt to the task of generating candidate sets of video behavior segments. The adapter adopts a bottleneck structure, including: ,in, is the intermediate hidden state matrix output by the intermediate layer of the image and text, is a nonlinear activation function, and They are the downsampling and upsampling mapping matrices of the bottleneck structure, represents the intermediate dimension of the bottleneck structure, Indicates the text length of the query statement. represents the dimension of the cross-modal common semantic space, Indicates the length of the instruction after word segmentation. Represents the set of real numbers.

6. The method for generating a candidate set of video action segments based on common sense enhancement as claimed in claim 5, characterized in that: The probability that the content in each video unit prediction matches the text description in the query sentence is obtained based on the large image and text pre-training model, and the probability sequence is obtained. , according to the probability sequence, a candidate set of video behavior segments is obtained by using a watershed algorithm ,in, and Respectively represent the start and end time of the first candidate video segment, and Respectively represent the start and end time of the second candidate video segment, and Respectively represent The start and end times of the candidate video segments, Indicates the number of candidate video segments finally generated.

7. A video action segment candidate set generation system based on common sense enhancement, characterized in that: include: The semantic feature representation generating unit is configured to: divide the video to be retrieved into continuous but non-overlapping video units, obtain the semantic feature representation of the query sentence and the semantic feature representation of each video unit; A modality alignment unit is configured to: map the semantic feature representation of the query statement to a cross-modality common semantic space to obtain a modality-aligned semantic feature representation of the query statement, and map the semantic feature representation of the video unit to a cross-modality common semantic space to obtain a modality-aligned semantic feature representation of the video unit; The feature fusion unit is configured to: interact the semantic feature representation of the modality-aligned query statement with the semantic feature representation of the modality-aligned video unit to obtain the semantic feature representation of the video unit fused with the text information, which is used to predict the probability of each video unit being retained. The specific process is as follows: Computing cross-modal attention scores for visual and textual semantic feature representations via cosine similarity , the corresponding text information is aggregated for each video unit through the cross-modal attention score, and multi-modal information is fused: ,in, To represent the semantic features of video units that integrate text information, The semantic feature representation of video units aligned with modality, It is the semantic feature representation of query sentences aligned with modality, is the cross-modal attention score The transpose of represents the set of real numbers, represents the dimension of the cross-modal common semantic space, Represents the number of video units; the prediction layer is used to predict the probability of each video unit being retained ,include: ,in, is the weight matrix of the prediction layer, is the activation function; The candidate set generation unit is configured to: sort the video units according to the probability of the video units being retained, and retain the top video units, the previous The semantic feature representation of the fused text information of the video units is mapped to the input space of the pre-trained model, and the semantic feature representation of the query sentence aligned with the modality and the preset instruction are spliced ​​as the input of the pre-trained model to obtain a candidate set of video behavior segments, wherein the reserved previous The specific way to map the semantic feature representation of the fused text information of a video unit to the input space of the pre-trained model is as follows: ,in, It is the visual semantic feature representation of the retained video unit after being aligned to the input space of the large pre-trained image and text model. are the trainable weights of the visual adaptation layer, To represent the semantic features of video units that integrate text information, represents the dimension of the cross-modal common semantic space, Indicates the number of video units, Represents the set of real numbers.

Images