Video clip retrieval method and device based on image-text matching, equipment and medium

Abstract

The invention provides a video clip retrieval method and device based on image-text matching, equipment and a medium, and relates to the technical field of video retrieval. According to the method, the video data is segmented into the plurality of video blocks, and the features are extracted respectively, so that the fine-grained understanding of the video content is enhanced, and the retrieval accuracy is improved. The feature coding model is used for coding the retrieval text and the video block, and semantic association between text description and video content is deepened, so that the matching accuracy is improved. The video blocks are retrieved through the multi-head multi-scale detection algorithm, the matching degree of the video blocks and the text can be evaluated in a multi-dimensional mode, and the matching accuracy is improved. The feature similarity of the video blocks and the text is calculated through the contrast learning algorithm, and the video block with the highest similarity is selected as the target video clip, so that the accuracy of a final retrieval result can be ensured, and the accuracy and efficiency of video clip retrieval are remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of video retrieval technology, and in particular to a video clip retrieval method, device, equipment and medium based on image-text matching. Background Art

[0002] The task of video clip retrieval is to locate the interval in the video that matches the text given a piece of text and a video. Using text-based video clip retrieval methods in financial transaction monitoring can effectively speed up the efficiency of analyzing and reviewing transactions. For example, descriptive text queries can be used to retrieve video clips related to specific transaction activities for risk assessment.

[0003] Existing single-stage methods usually perform poorly on object detection tasks and have low accuracy in localizing video clips. In addition, single-stage methods usually pay less attention to the semantic association between text descriptions and video content, so the matching results are inaccurate when matching text to video frames.

[0004] Therefore, how to improve the accuracy of video clip retrieval has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present application provides a video clip retrieval method, device, equipment and medium based on image-text matching, aiming to improve the accuracy of video clip retrieval.

[0006] In a first aspect, the present application provides a video clip retrieval method based on image-text matching, the video clip retrieval method based on image-text matching comprising the following steps:

[0007] Obtain the video data to be retrieved and the retrieval text;

[0008] Based on a video processing algorithm, the video data to be retrieved is segmented to obtain at least one video block;

[0009] Based on the feature coding model, encoding the search text and each of the video blocks is performed respectively to obtain a first text feature corresponding to the search text and a first video feature corresponding to each of the video blocks;

[0010] Based on a multi-head multi-scale detection algorithm, searching the first video feature corresponding to each of the video blocks, and outputting at least one video segment interval matching the first text feature;

[0011] Based on a contrastive learning algorithm, feature similarity between a first video feature and a first text feature corresponding to at least one of the video blocks within the video segment interval is calculated, and the video block with the greatest feature similarity is determined as a target video segment.

[0012] In a second aspect, the present application further provides a video clip retrieval device based on image-text matching, the video clip retrieval device based on image-text matching comprising:

[0013] A data acquisition module is used to acquire video data to be retrieved and retrieve text;

[0014] A video processing module, configured to segment the video data to be retrieved based on a video processing algorithm to obtain at least one video block;

[0015] A feature encoding module, used for encoding the search text and each of the video blocks based on a feature encoding model to obtain a first text feature corresponding to the search text and a first video feature corresponding to each of the video blocks;

[0016] A multi-head detection module, configured to retrieve the first video feature corresponding to each of the video blocks based on a multi-head multi-scale detection algorithm, and output at least one video segment interval matching the first text feature;

[0017] The video segment retrieval module is used to calculate the feature similarity between the first video feature and the first text feature corresponding to at least one video block in the video segment interval based on a contrastive learning algorithm, and determine the video block with the largest feature similarity as the target video segment.

[0018] In a third aspect, the present application also provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the video clip retrieval method based on image-text matching as described above are implemented.

[0019] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the video clip retrieval method based on image-text matching as described above are implemented.

[0020] The present application provides a video clip retrieval method, device, computer equipment and storage medium based on image-text matching. The method of the present application enhances the fine-grained understanding of the video content by dividing the video data into multiple video blocks and extracting features respectively, thereby improving the accuracy of the retrieval. The retrieval text and video blocks are encoded using a feature coding model, which deepens the semantic association between the text description and the video content, thereby improving the accuracy of the match. By retrieving the video blocks through a multi-head multi-scale detection algorithm, the matching degree between the video blocks and the text can be evaluated in multiple dimensions, thereby improving the accuracy of the match. By calculating the feature similarity between the video block and the text through a comparative learning algorithm, and selecting the video block with the highest similarity as the target video clip, the accuracy of the final retrieval result can be ensured, thereby significantly improving the accuracy and efficiency of video clip retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 A schematic diagram of a flow chart of a first embodiment of a video clip retrieval method based on image-text matching provided by the present application;

[0023] Figure 2 A schematic diagram of a feature processing flow of video data and text data provided in an embodiment of the present application;

[0024] Figure 3 A schematic diagram of a flow chart of a second embodiment of a video clip retrieval method based on image-text matching provided by the present application;

[0025] Figure 4 It is a structural schematic diagram of a first embodiment of a video clip retrieval device based on image-text matching provided by the present application;

[0026] Figure 5 It is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application.

[0027] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0029] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0030] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0031] Please refer to Figure 1 , Figure 1 A schematic flow chart of a first embodiment of a video clip retrieval method based on image-text matching provided in the present application.

[0032] like Figure 1 As shown, the video clip retrieval method based on image-text matching includes steps S101 to S105.

[0033] S101, obtaining video data to be searched and search text;

[0034] In one embodiment, the source of the video data is determined, such as a surveillance camera, a video uploaded by a user, an online video platform, etc. The video data is obtained from the determined source. If it is a real-time monitoring system, real-time streaming media processing may be required; if it is historical data, it may need to be retrieved from a database or storage system.

[0035] In one embodiment, necessary preprocessing may be performed on the acquired video data, such as format conversion, compression, denoising, etc., to meet the needs of subsequent processing.

[0036] In one embodiment, the user is allowed to input search texts through a user interface, which describe the video content that the user wants to retrieve. The texts input by the user are preprocessed, including word segmentation, removal of stop words, stem extraction, etc., to improve the accuracy of the search.

[0037] S102, based on a video processing algorithm, segmenting the video data to be retrieved to obtain at least one video block;

[0038] In one embodiment, inspired by the anchor candidate box in the YOLO algorithm, the video V of N frames is first divided into N equally spaced frames.b video blocks (if not divisible, add frames), where the length of each video block is block_size.

[0039] Generally speaking, the core innovation of YOLO is to treat the object detection task as an end-to-end regression problem, through a single forward propagation, simultaneously predicting multiple bounding boxes and corresponding category probabilities in the image.

[0040] Furthermore, based on a clustering algorithm, at least one benchmark value in the video data is clustered to generate at least one candidate box of different lengths; based on the center point of each video block, each candidate box is configured in each video block, and a category label for each candidate box is constructed; wherein the category label includes a confidence label, a midpoint position label, and a length label.

[0041] Specifically, based on all the grounding truth (GT, real annotation box) in the data set, the Kmeans clustering algorithm is used to cluster C anchors of different lengths. Furthermore, for each video block, C anchors are preset with the midpoint of the video block as the center, that is, there are a total of N b ×C Anchors.

[0042] In one embodiment, a label is constructed for each anchor, including a confidence label P, a midpoint position label M, and a length label L.

[0043] Specifically, the confidence label is determined based on the intersection over union (IOU) between the anchor and the GT. If the IOU is greater than a certain threshold, then the anchor is considered a positive example, that is, the area containing the target. At this time, P = 1, and the remaining anchors are considered negative examples, that is, P = 0. The midpoint position label refers to the midpoint position label of each anchor, which is used to indicate the position of the anchor in the video. The length label refers to the length label of each anchor, which indicates the time length of the anchor box.

[0044] Among them, Intersection over Union (IOU) is an indicator to measure the degree of overlap between two boxes (usually the predicted box and the real box), and the calculation formula is the ratio of the intersection area of ​​the two boxes to the union area.

[0045] Generally, in the YOLO algorithm, positive examples refer to those anchors whose IOU with GT is greater than a certain threshold, that is, these anchor boxes contain targets. Negative examples refer to those anchors whose IOU with GT is less than the threshold, that is, these anchor boxes do not contain targets.

[0046] In addition, the positive examples are expanded with reference to YOLO, and the anchors corresponding to the two previous and next video blocks are also regarded as positive examples, that is, the offset Offset is set to [-2, -1, 0, 1, 2].

[0047] S103, encoding the search text and each of the video blocks based on a feature encoding model to obtain a first text feature corresponding to the search text and a first video feature corresponding to each of the video blocks;

[0048] In one embodiment, using the CLIP model first requires feature encoding of the video and text, using ViT and BERT, which are commonly used in CLIP, as the basic Transformer models for video frames and text, respectively.

[0049] Among them, CLIP (Contrastive Language-Image Pre-training) is based on the principle of large-scale contrastive learning, which identifies the matching relationship between an image and its description by training the image encoder and the text encoder in parallel. This method uses a large number of image-text pairs as training data and trains by optimizing the similarity between image and text representations. This strategy enables CLIP to not only understand a wide range of visual concepts, but also effectively combine these concepts with natural language, thereby achieving deep semantic understanding.

[0050] In one embodiment, the image data is preprocessed by the standard CLIP preprocessing process, including resizing, normalization, etc., to ensure that it meets the input specifications of the model. The text data is processed by the CLIP tokenizer, which converts the text into a series of tokens and then into a numerical vector to adapt to the processing requirements of the model.

[0051] In one embodiment, in the feature extraction stage, the preprocessed image and text data are input into the image encoder and text encoder of CLIP respectively. These two encoders generate high-dimensional feature representations for images and texts respectively, which capture and express the deep semantic content of images and texts in multi-dimensional space.

[0052] Specifically, the video encoder: for a video containing N frames of images First, each frame image I i Send it to the pre-trained ViT model to get the embedded representation of the frame

[0053]

[0054] Splicing to get video representation Then use a convolutional neural network with a convolution kernel size and a step length of block_size to convert X I The length is compressed to N b,Right now Further input into the stacked multi-layer Transformer Block to obtain a video representation with the length of the number of video blocks

[0055] Text encoder: For a given text containing N t The sentence S with 10 words is input into the BERT text model. In addition, since the self-attention module involves more parameters, in order to reduce the training complexity while ensuring that the network has sufficient learning ability, the BERT model parameters are frozen with reference to the idea of ​​the plug-in (Adaptor), and two layers of feedforward neural network (FFN) are stacked.

[0056]

[0057] Thereby, the trainable parameters of the text encoder are compressed into the parameters contained in the two FFNs, thus improving the training efficiency.

[0058] Furthermore, based on a plurality of preset length sizes, the first video feature is segmented to obtain video feature codes of different length sizes; based on a feedforward neural network, the video feature codes of different length sizes are processed to generate a multi-head multi-size detection head for retrieving the first video feature corresponding to each of the video blocks.

[0059] Generally, since the number of frames of the retrieved video clips may vary greatly, referring to the network architecture of YOLO, the network model of this application also uses a multi-scale multi-head detection architecture. Multi-scale can effectively locate the intervals that match the text description in videos of different scales, which helps to process video clips of different scales and resolutions; multi-head detection can simultaneously find multiple possible matching intervals, thereby improving the efficiency and accuracy of retrieval.

[0060] Specifically, taking three different block_sizes, the number of video blocks obtained is Get multi-scale video feature encoding Further output through FPN model As multiple detection heads, Each Bounding box has three label confidence labels Midpoint location label Length label So put it together A Bounding box.

[0061] S104, searching the first video feature corresponding to each of the video blocks based on a multi-head multi-scale detection algorithm, and outputting at least one video segment interval matching the first text feature;

[0062] Specifically, the multi-head detection mechanism improves retrieval efficiency and accuracy by searching multiple possible matching intervals simultaneously. This mechanism is similar to the FPN (Feature Pyramid Networks) structure used in YOLOv3, which effectively combines high-level semantic information with low-level spatial information through top-down paths and lateral connections, thereby performing more accurate object detection at different scales. In video retrieval, this means that video clips that match text descriptions can be quickly identified, regardless of the size of the target objects in these clips.

[0063] Specifically, Figure 2 As shown in the figure, the FPN model acts as multiple detection heads in the output layer and can output multiple bounding boxes. Each bounding box contains three label confidence labels, midpoint position labels, and length labels. This design enables the model to provide rich information for accurately locating and identifying target objects in videos. In video retrieval tasks, this information can help the algorithm more accurately match and retrieve video content that matches the text description.

[0064] In one embodiment, another advantage of the multi-scale multi-head detection architecture is its flexibility and scalability. By adjusting the feature encodings of different scales and the number of detection heads, it can be optimized according to the requirements of different video retrieval tasks. For example, for tasks that require high-precision retrieval, the weight of high-resolution feature encoding can be increased; and for tasks that require high real-time performance, low-resolution feature encoding can be optimized to increase processing speed.

[0065] In this embodiment, the multi-scale multi-head detection architecture provides an efficient and accurate solution for video retrieval tasks by combining feature extraction at different scales and a multi-head detection mechanism. This architecture can not only handle the diversity of video content, but also provide accurate target positioning and recognition, thereby improving the efficiency and accuracy of video retrieval.

[0066] S105 , calculating feature similarity between a first video feature and a first text feature corresponding to at least one of the video blocks in the video segment interval based on a contrastive learning algorithm, and determining the video block with the greatest feature similarity as a target video segment.

[0067] In one embodiment, CLIP trains the model through contrastive learning. Specifically, for each image-text pair in a batch, the model calculates the feature vectors of the image and text, and uses a contrastive loss function to optimize the model parameters. The goal of the contrastive loss function is to make the feature vectors of matching image-text pairs as close as possible, and the feature vectors of mismatching image-text pairs as far apart as possible.

[0068] Specifically, for each image-text pair, the model calculates losses in two directions:

[0069] Image-to-text loss: Calculate the similarity between the image feature vector and the text feature vector, and optimize to maximize the similarity of the matched image-text pairs.

[0070] Text-to-image loss: Calculates the similarity between text feature vectors and image feature vectors, and optimizes to maximize the similarity of matched text-image pairs.

[0071] Specifically, in the training of CLIP, the loss design includes two directions: from image to text and from text to image. Assume that there are n pairs of image-text pairs, the image encoding is represented as I and the text encoding is represented as T. The element s of the similarity matrix S is ij represents the similarity between the i-th image and the j-th text.

[0072] Generally, cosine similarity is calculated:

[0073]

[0074] Cross entropy loss for image to text i :

[0075] loss i =cross_entropy_loss(S,labels,axis=0)

[0076] Among them, axis=0 means calculating the loss for each row (that is, all texts corresponding to each image).

[0077] Cross entropy loss for text to image t :

[0078] loss t =cross_entropy_loss(S,labels,axis=1)

[0079] Among them, axis=1 means that the loss is calculated for each column (that is, all images corresponding to each text).

[0080] The final loss is the average of the cross entropy losses in these two directions:

[0081]

[0082] Specifically, for each image I i , calculate its similarity S with all texts i,:Using cross entropy loss, these similarities are compared with the labels (the index of the correct matching text) to ensure that the correct text description is found for each image. For each text T j , calculate its similarity S with all images :,j Using cross entropy loss, these similarities are compared to the labels (correct matching image indices), ensuring that each text finds the correct image.

[0083] In this way, the model can not only find the corresponding text description from the image, but also find the corresponding image from the text. This two-way optimization makes the CLIP model more robust and accurate in practical applications.

[0084] The present embodiment provides a video clip retrieval method based on image-text matching. The present method enhances the fine-grained understanding of the video content by dividing the video data into multiple video blocks and extracting features respectively, thereby improving the accuracy of the retrieval. The feature encoding model is used to encode the retrieval text and video blocks, which deepens the semantic association between the text description and the video content, thereby improving the accuracy of the matching. By retrieving the video blocks through a multi-head multi-scale detection algorithm, the matching degree between the video blocks and the text can be evaluated in multiple dimensions, thereby improving the accuracy of the matching. By calculating the feature similarity between the video block and the text through a comparative learning algorithm, and selecting the video block with the highest similarity as the target video clip, the accuracy of the final retrieval result can be ensured, thereby significantly improving the accuracy and efficiency of the video clip retrieval.

[0085] Please refer to Figure 3 , Figure 3 A schematic flow chart of a second embodiment of a video clip retrieval method based on image-text matching provided in the present application.

[0086] In this embodiment, Figure 3 As shown, based on the above Figure 1 In the illustrated embodiment, the S105 specifically includes:

[0087] S201, based on a feature coding optimization algorithm, performing cross attention calculation on the first video feature and the first text feature to obtain a first model loss;

[0088] In this embodiment, in the video clip retrieval task, Contrastive Content Resourcing (CCR) and Contrastive Content Swapping (CCS) modules are used to optimize feature coding.

[0089] The core idea of ​​the CCR module is to enhance the model's understanding of the correlation between query fragments and their corresponding key fragments through contrastive learning. The CCS module aims to further optimize the model's ability to suppress irrelevant features.

[0090] Furthermore, based on the feature encoding optimization algorithm, the attention features and anti-attention features of the first video feature and the first text feature are calculated; based on the attention features and the anti-attention features, the first model loss is calculated.

[0091] Specifically, the video is encoded X V The jth row of , i.e., the feature encoding of the jth video block is used as the i-th query q i , the text represents X T As the key value k, a single-layer interactive attention model is used to obtain k relative to q i The attention weight ω i :

[0092]

[0093] Among them, f att is the attention function, output e i is a scalar, thus obtaining the attention feature of q:

[0094] a i =Σ i (ω i ,k)

[0095] And the anti-attention feature of q:

[0096]

[0097] The CCR loss function is defined as:

[0098]

[0099] Among them, γ1 is the margin value used to control the loss, and Sim() represents the similarity calculation function.

[0100] In this embodiment, the CCR loss function encourages the model to increase the similarity between the query fragment and its corresponding attention feature, while reducing the similarity with the anti-attention feature.

[0101] In one embodiment, CCS needs to first define negative queries. Given a query q i , all with GT and q i The feature encodings of non-overlapping video blocks are concatenated together as negative queries

[0102] Generally, the CCS loss function is defined as:

[0103]

[0104] In this embodiment, CCS encourages the model to focus more on features related to query fragments and pay less attention to negative query fragments.

[0105] In this example, CCR helps the model identify and enhance relevant features, while CCS suppresses irrelevant features. The combination of the two can more accurately locate the part of the video clip that is relevant to the text query. Through these two strategies, the model can establish a more precise correspondence between video and text, thereby improving the accuracy of video retrieval tasks.

[0106] S202, calculating a second model loss corresponding to each of the category labels based on the category labels corresponding to each of the video clip intervals output by the multi-head multi-scale detection algorithm;

[0107] In one embodiment, in the video clip retrieval task, in addition to the two loss functions of Contrastive Content Resourcing (CCR) and Contrastive Content Swapping (CCS), the multi-scale multi-head detection architecture outputs A Bounding box also needs to calculate the corresponding loss.

[0108] Furthermore, based on the cross entropy loss function, the first label loss of the confidence label is calculated; based on the mean square error loss function, the second label loss of the center position label and the third label loss of the boundary length label are calculated; based on the first label loss, the second label loss and the third label loss, the second model loss is obtained.

[0109] In one embodiment, the loss of the confidence label P Using cross entropy, the loss of the midpoint position label M and the loss of the boundary length label L Use mean square error.

[0110] Specifically,

[0111]

[0112] S203. Calculate an optimal solution of a loss function based on the first model loss and the second model loss;

[0113] Furthermore, based on a weighted algorithm, the first model loss and the second model loss are weightedly summed to obtain the weighted loss function; based on a multi-objective optimization algorithm, the optimal weights of the first model loss and the second model loss are adjusted to obtain an optimal solution to the weighted loss function.

[0114] In one embodiment, the total loss of the video segment retrieval task is Generally, the above multiple losses are weighted:

[0115]

[0116] Among them, α, β, γ, δ, ε are hyperparameters that control the weights.

[0117] Understandably, considering the difficulty of parameter adjustment, this problem can be transformed into a multi-objective optimization problem, looking for a Pareto optimal solution that is as close as possible to multiple loss functions. The Pareto optimal solution means that in a multi-objective optimization problem, it is impossible to find a solution that is better than this solution in all objectives, that is, there is no other solution that can improve all objective functions at the same time. In multi-objective optimization, the goal is to find a set of solutions that are balanced in all objectives, and no solution can improve any one objective without sacrificing other objectives.

[0118] S204: Calculate the feature similarity between the first video feature and the first text feature corresponding to at least one of the video blocks in the video segment interval based on the loss function of the optimal solution, and determine the video block with the largest feature similarity as the target video segment.

[0119] In one embodiment, for multi-objective optimization problems, a multi-gradient descent algorithm (MGDA) can be used, which is particularly suitable for deep learning, can effectively process large-scale, high-dimensional parameter spaces in deep neural networks, and has certain theoretical guarantees.

[0120] The MGDA algorithm calculates the gradient of each task, then solves a convex optimization problem to find the optimal convex combination of all task gradients, and finally updates the shared parameters. This method can be extended to very large models and tasks with almost no increase in computational overhead.

[0121] In this embodiment, by combining CCR, CCS loss and loss calculation of Bounding box, and using MGDA for multi-objective optimization, the performance of video clip retrieval tasks can be effectively improved while reducing the workload of manual parameter adjustment. This method can not only balance the weights between different loss functions, but also find a solution close to Pareto optimality in multi-objective optimization problems, thereby improving the overall performance of the model.

[0122] See also Figure 4 , Figure 4 It is a structural schematic diagram of a first embodiment of a video clip retrieval device based on image-text matching provided by the present application. The video clip retrieval device based on image-text matching is used to execute the aforementioned video clip retrieval method based on image-text matching.

[0123] like Figure 4 As shown, the video segment retrieval device 300 based on image-text matching includes: a data acquisition module 301, a video processing module 302, a feature encoding module 303, a multi-head detection module 304 and a video segment retrieval module 305.

[0124] The data acquisition module 301 is used to acquire the video data to be retrieved and the search text;

[0125] The video processing module 302 is used to segment the video data to be retrieved based on a video processing algorithm to obtain at least one video block;

[0126] A feature encoding module 303 is used to encode the search text and each of the video blocks based on a feature encoding model to obtain a first text feature corresponding to the search text and a first video feature corresponding to each of the video blocks;

[0127] A multi-head detection module 304 is used to retrieve the first video feature corresponding to each of the video blocks based on a multi-head multi-scale detection algorithm, and output at least one video segment interval matching the first text feature;

[0128] The video segment retrieval module 305 is used to calculate the feature similarity between the first video feature and the first text feature corresponding to at least one video block in the video segment interval based on a contrastive learning algorithm, and determine the video block with the largest feature similarity as the target video segment.

[0129] In one embodiment, the video segment retrieval device 300 based on image-text matching further includes a label construction module, including:

[0130] A data clustering unit, configured to cluster at least one reference value in the video data based on a clustering algorithm to generate at least one candidate frame of different lengths;

[0131] A label construction unit, configured to configure each candidate box in each video block based on the center point of each video block, and construct a category label for each candidate box;

[0132] The category labels include confidence labels, midpoint position labels and length labels.

[0133] In one embodiment, the video segment retrieval device 300 based on image-text matching further includes a detection header generation module, including:

[0134] A multi-scale segmentation unit, configured to segment the first video feature based on a plurality of preset length sizes to obtain video feature codes of different length sizes;

[0135] The multi-scale multi-head detection head generation unit is used to process the video feature encodings of different length sizes based on a feedforward neural network to generate a multi-head multi-size detection head for retrieving the first video feature corresponding to each of the video blocks.

[0136] In one embodiment, the video clip retrieval module 305 includes:

[0137] A first model loss calculation unit, configured to perform cross attention calculation on the first video feature and the first text feature based on a feature encoding optimization algorithm to obtain a first model loss;

[0138] A second model loss calculation unit, configured to calculate a second model loss corresponding to each of the category labels based on the category labels corresponding to each of the video clip intervals output by the multi-head multi-scale detection algorithm;

[0139] A loss function optimal solution calculation unit, used to calculate an optimal solution of the loss function based on the first model loss and the second model loss;

[0140] The feature similarity calculation unit is used to calculate the feature similarity between the first video feature and the first text feature corresponding to at least one of the video blocks in the video segment interval based on the loss function of the optimal solution, and determine the video block with the largest feature similarity as the target video segment.

[0141] In one embodiment, the first model loss calculation unit includes:

[0142] an attention feature calculation subunit, configured to calculate the attention feature and the anti-attention feature of the first video feature and the first text feature based on the feature encoding optimization algorithm;

[0143] The first model loss calculation subunit is used to calculate the first model loss based on the attention feature and the anti-attention feature.

[0144] In one embodiment, the second model loss calculation unit includes:

[0145] A first label loss calculation subunit, used for calculating a first label loss of a confidence label based on a cross entropy loss function;

[0146] The second label loss calculation subunit is used to calculate the second label loss of the center position label and the third label loss of the boundary length label based on the mean square error loss function;

[0147] The second model loss calculation subunit is used to obtain the second model loss based on the first label loss, the second label loss and the third label loss.

[0148] In one embodiment, the loss function optimal solution calculation unit includes:

[0149] A weighted calculation subunit, configured to perform a weighted summation of the first model loss and the second model loss based on a weighted algorithm to obtain the weighted loss function;

[0150] The optimal solution calculation subunit is used to adjust the optimal weights of the first model loss and the second model loss based on a multi-objective optimization algorithm to obtain the optimal solution of the weighted loss function.

[0151] It should be noted that technicians in the relevant field can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned video clip retrieval method embodiment based on image and text matching, and will not be repeated here.

[0152] The apparatus provided in the above embodiment may be implemented in the form of a computer program. The computer program may be Figure 5 Runs on the computer device shown.

[0153] See also Figure 5 , Figure 5 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server.

[0154] See also Figure 5 The computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0155] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any video clip retrieval method based on image-text matching.

[0156] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0157] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any video clip retrieval method based on image-text matching.

[0158] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0159] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0160] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0161] Obtain the video data to be retrieved and the retrieval text;

[0162] Based on a video processing algorithm, the video data to be retrieved is segmented to obtain at least one video block;

[0163] Based on the feature coding model, encoding the search text and each of the video blocks is performed respectively to obtain a first text feature corresponding to the search text and a first video feature corresponding to each of the video blocks;

[0164] Based on a multi-head multi-scale detection algorithm, searching the first video feature corresponding to each of the video blocks, and outputting at least one video segment interval matching the first text feature;

[0165] Based on a contrastive learning algorithm, feature similarity between a first video feature and a first text feature corresponding to at least one of the video blocks within the video segment interval is calculated, and the video block with the greatest feature similarity is determined as a target video segment.

[0166] In one embodiment, after implementing the video processing algorithm to segment the video data to be retrieved to obtain at least one video block, the processor is further configured to implement:

[0167] Based on a clustering algorithm, clustering at least one reference value in the video data to generate at least one candidate frame of different length;

[0168] Based on the center point of each video block, each candidate box is configured in each video block, and a category label of each candidate box is constructed;

[0169] The category labels include confidence labels, midpoint position labels and length labels.

[0170] In one embodiment, before implementing the multi-head multi-scale detection algorithm to retrieve the first video feature corresponding to each of the video blocks and outputting at least one video segment interval matching the first text feature, the processor is further configured to implement:

[0171] Based on a plurality of preset length sizes, the first video feature is segmented to obtain video feature codes of different length sizes;

[0172] Based on a feedforward neural network, the video feature codes of different lengths are processed to generate a multi-head multi-size detection head for retrieving the first video feature corresponding to each of the video blocks.

[0173] In one embodiment, when the processor implements the contrastive learning algorithm, calculates the feature similarity between the first video feature and the first text feature corresponding to at least one of the video blocks in the video segment interval, and determines that the video block with the largest feature similarity is the target video segment, it is used to implement:

[0174] Based on a feature encoding optimization algorithm, cross attention calculation is performed on the first video feature and the first text feature to obtain a first model loss;

[0175] Based on the category labels corresponding to the respective video clip intervals output by the multi-head multi-scale detection algorithm, calculating the second model loss corresponding to the respective category labels;

[0176] Calculating an optimal solution of a loss function based on the first model loss and the second model loss;

[0177] Based on the loss function of the optimal solution, the feature similarity between the first video feature and the first text feature corresponding to at least one of the video blocks in the video segment interval is calculated, and the video block with the largest feature similarity is determined as the target video segment.

[0178] In one embodiment, when the processor implements the feature-based coding optimization algorithm to perform cross-attention calculation on the first video feature and the first text feature to obtain the first model loss, it is used to implement:

[0179] Based on the feature encoding optimization algorithm, calculating the attention feature and the anti-attention feature of the first video feature and the first text feature;

[0180] Based on the attention features and the anti-attention features, the first model loss is calculated.

[0181] In one embodiment, when the processor implements the category label corresponding to each of the video clip intervals outputted by the multi-head multi-scale detection algorithm and calculates the second model loss corresponding to each of the category labels, it is used to implement:

[0182] Based on the cross entropy loss function, calculate the first label loss of the confidence label;

[0183] Based on the mean square error loss function, the second label loss of the center position label and the third label loss of the boundary length label are calculated;

[0184] The second model loss is obtained based on the first label loss, the second label loss and the third label loss.

[0185] In one embodiment, when the processor implements the step of calculating the optimal solution of the weighted loss function based on the first model loss and the second model loss, the processor is configured to implement:

[0186] Based on a weighted algorithm, performing a weighted summation on the first model loss and the second model loss to obtain the weighted loss function;

[0187] Based on a multi-objective optimization algorithm, the optimal weights of the first model loss and the second model loss are adjusted to obtain an optimal solution to the weighted loss function.

[0188] A computer-readable storage medium is also provided in an embodiment of the present application, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the processor executes the program instructions to implement any one of the video clip retrieval methods based on image-text matching provided in the embodiments of the present application.

[0189] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0190] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A video clip retrieval method based on image-text matching, characterized in that: The method comprises: Obtain the video data to be retrieved and the retrieval text; Based on a video processing algorithm, the video data to be retrieved is segmented to obtain at least one video block; Based on the feature coding model, encoding the search text and each of the video blocks is performed respectively to obtain a first text feature corresponding to the search text and a first video feature corresponding to each of the video blocks; Based on a multi-head multi-scale detection algorithm, searching the first video feature corresponding to each of the video blocks, and outputting at least one video segment interval matching the first text feature; Based on a contrastive learning algorithm, feature similarity between a first video feature and a first text feature corresponding to at least one of the video blocks within the video segment interval is calculated, and the video block with the greatest feature similarity is determined as a target video segment.

2. The video clip retrieval method based on image-text matching according to claim 1, characterized in that: After the video data to be retrieved is segmented based on the video processing algorithm to obtain at least one video block, the method further includes: Based on a clustering algorithm, clustering at least one reference value in the video data to generate at least one candidate frame of different length; Based on the center point of each video block, each candidate box is configured in each video block, and a category label of each candidate box is constructed; The category labels include confidence labels, midpoint position labels and length labels.

3. The video clip retrieval method based on image-text matching according to claim 1, characterized in that: Before searching the first video feature corresponding to each of the video blocks based on the multi-head multi-scale detection algorithm and outputting at least one video segment interval matching the first text feature, the method further includes: Based on a plurality of preset length sizes, the first video feature is segmented to obtain video feature codes of different length sizes; Based on a feedforward neural network, the video feature codes of different lengths are processed to generate a multi-head multi-size detection head for retrieving the first video feature corresponding to each of the video blocks.

4. The video clip retrieval method based on image-text matching according to claim 1, characterized in that: The step of calculating the feature similarity between the first video feature and the first text feature corresponding to at least one of the video blocks in the video segment interval based on the contrastive learning algorithm, and determining the video block with the largest feature similarity as the target video segment, comprises: Based on a feature encoding optimization algorithm, cross attention calculation is performed on the first video feature and the first text feature to obtain a first model loss; Based on the category labels corresponding to the respective video clip intervals output by the multi-head multi-scale detection algorithm, calculating the second model loss corresponding to the respective category labels; Calculating an optimal solution of a loss function based on the first model loss and the second model loss; Based on the loss function of the optimal solution, the feature similarity between the first video feature and the first text feature corresponding to at least one of the video blocks in the video segment interval is calculated, and the video block with the largest feature similarity is determined as the target video segment.

5. The video clip retrieval method based on image-text matching according to claim 4 is characterized in that: The step of performing cross attention calculation on the first video feature and the first text feature based on the feature encoding optimization algorithm to obtain a first model loss includes: Based on the feature encoding optimization algorithm, calculating the attention feature and the anti-attention feature of the first video feature and the first text feature; Based on the attention features and the anti-attention features, the first model loss is calculated.

6. The video clip retrieval method based on image-text matching according to claim 4, characterized in that: The calculating the second model loss corresponding to each category label based on the category label corresponding to each video segment interval output by the multi-head multi-scale detection algorithm includes: Based on the cross entropy loss function, calculate the first label loss of the confidence label; Based on the mean square error loss function, the second label loss of the center position label and the third label loss of the boundary length label are calculated; The second model loss is obtained based on the first label loss, the second label loss and the third label loss.

7. The video clip retrieval method based on image-text matching according to claim 4, characterized in that: The calculating an optimal solution of a weighted loss function based on the first model loss and the second model loss includes: Based on a weighted algorithm, performing a weighted summation on the first model loss and the second model loss to obtain the weighted loss function; Based on a multi-objective optimization algorithm, the optimal weights of the first model loss and the second model loss are adjusted to obtain an optimal solution to the weighted loss function.

8. A video clip retrieval device based on image-text matching, characterized in that: The video clip retrieval device based on image-text matching comprises: A data acquisition module is used to acquire video data to be retrieved and retrieve text; A video processing module, configured to segment the video data to be retrieved based on a video processing algorithm to obtain at least one video block; A feature encoding module, used for encoding the search text and each of the video blocks based on a feature encoding model to obtain a first text feature corresponding to the search text and a first video feature corresponding to each of the video blocks; A multi-head detection module, configured to retrieve the first video feature corresponding to each of the video blocks based on a multi-head multi-scale detection algorithm, and output at least one video segment interval matching the first text feature; The video segment retrieval module is used to calculate the feature similarity between the first video feature and the first text feature corresponding to at least one video block in the video segment interval based on a contrastive learning algorithm, and determine the video block with the largest feature similarity as the target video segment.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the video clip retrieval method based on image-text matching as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the video clip retrieval method based on image-text matching according to any one of claims 1 to 7 are implemented.

Images