Weak supervision video clip retrieval method and device based on multi-modal information enhancement and medium

Abstract

The invention relates to a weak supervision video clip retrieval method and device based on multi-modal information enhancement and a medium, and the method comprises the steps: inputting an unedited video and a query text into an input representation module, and obtaining video features and query text features; and inputting the non-edited video into a scene distribution detection module, and clustering video frames to obtain scene features. And inputting the video features, the query text features and the scene features into a proposal fragment generation module to obtain predicted candidate proposal fragments. And inputting the candidate proposal fragments into a proposal fragment enhancement module to obtain multi-modal enhanced fragment features. And calculating query text reconstruction loss and query text label similarity loss of the fragment characteristics in the proposed fragment evaluation module, selecting the fragment with the minimum loss as a retrieval result, and outputting starting time and ending time of the fragment. Compared with the prior art, the method has the advantages of high accuracy, high timeliness, high effectiveness and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of multimedia information retrieval, and in particular to a weakly supervised video clip retrieval method, device and medium based on multimodal information enhancement. Background Art

[0002] Video clip retrieval is a cross-modal information retrieval task that accurately locates and extracts specific clips from uncut videos based on natural language query text. Accurate video clip retrieval can greatly improve user experience, achieve efficient content management, and accurately obtain information. Traditional retrieval methods rely on time boundary annotation. The manual annotation process is complex and costly, requiring professionals to watch the video frame by frame to mark the exact start and end time points. In addition, since different people may have different understandings of the same video, the consistency and accuracy of the annotation results are difficult to guarantee. Faced with a huge video library, traditional methods cannot meet the requirements of real-time and efficiency. Therefore, studying video clip retrieval methods under weak supervision has become a focus of attention. Weakly supervised learning reduces the need for finely labeled data and uses a small number of labeled samples or indirect supervision signals to train the model, thereby reducing the annotation cost and improving the generalization ability of the model.

[0003] There are still multiple challenges in the video clip retrieval method based on weakly supervised learning. First, there is a problem of mismatching scene proposals, that is, the model fails to fully consider the characteristics of scene distribution in the video, resulting in the generated proposals not accurately reflecting the complexity of the video content. Secondly, the multimodal information is not fully utilized. Most existing methods rely on simple visual-text feature combinations, ignoring the rich scene, object and motion information in the video, which is crucial for enhancing the proposal representation. Video is not just a collection of static images, it also contains rich spatiotemporal information, such as the actions of characters, interactions between objects, and changes in the environment. In addition, the negative sample mining strategy lacks an effective mechanism, which limits the further improvement of model performance. During the training process, if all non-relevant clips are simply used as negative samples, it may cause the model to overfit the background noise and reduce its distinguishing ability. On the contrary, selecting representative negative samples can help the model better learn the semantic differences between the query text and the video. Therefore, how to combine the weakly supervised video clip retrieval method with multimodal information and improve the accuracy and effectiveness of video clip retrieval by selecting negative samples for comparative learning is a technical problem that needs to be solved. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a weakly supervised video clip retrieval method, device and medium based on multimodal information enhancement. Through the scene distribution detection module, the scene features are used as a guide to generate proposed clips that match the complexity of the video, thereby solving the mismatch problem between the scene and the proposed clip. In addition, the multimodal feature fusion of object label detection and action label detection is combined to enhance the semantic richness of the proposed clip, thereby achieving more effective and accurate video clip retrieval.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] According to one aspect of the present invention, a weakly supervised video segment retrieval method based on multimodal information enhancement is provided, and the specific steps include:

[0007] S1. Input the uncut video and the query text into the input representation module to obtain video features and query text features; at the same time, input the uncut video into the scene distribution detection module to cluster the video frames and obtain scene features;

[0008] S2, inputting the video features, query text features and scene features into a proposed segment generation module to obtain predicted candidate proposed segments;

[0009] S3, inputting the candidate proposed segment into the proposed segment enhancement module to obtain multimodal enhanced segment features;

[0010] S4. Calculate the query text reconstruction loss and the query text label similarity loss of the segment feature in the proposed segment evaluation module, select the segment with the smallest loss as the retrieval result, and output the start and end time of the segment.

[0011] In S1, the input representation module uses a pre-trained 3D convolutional neural network to extract frame features from the uncut video to obtain video features; performs word segmentation and stop word removal preprocessing on the query text, and uses the pre-trained GloVe model to obtain the word vector representation of the query text to obtain the query text features; uses a hierarchical clustering algorithm to group the frame features, divides the video into multiple continuous scene segments, constructs scene distribution information, and obtains scene features; uses the TransformerEncorder structure to perform cross-modal interaction on the video features and the query text features, and obtains video features and query text features that integrate cross-modal information and contextual semantic information.

[0012] The proposed segment generation module of S2 splices video, query text and scene features, obtains fused features through TransformerEncorder, and uses a multi-layer perceptron to predict the number, center position and length information of candidate proposed positive sample segments. Each segment uses a Flatten Gaussian mask to represent the temporal structure.

[0013] For each candidate proposed positive sample segment, a corresponding candidate proposed negative sample segment is generated in each scene segment, and the similarity score function of the distance between the central frame of the candidate proposed negative sample segment and the central frame of the corresponding candidate proposed positive sample segment in the cluster space is minimized.

[0014] The similarity score function is obtained by feature similarity and spatiotemporal distance. For each candidate proposal positive sample segment Generate candidate negative samples in each scene segment Similarity score function The expression is:

[0015]

[0016] in, To calculate feature similarity using cosine similarity, is the temporal distance between the center points of the positive and negative sample segments, α(E) is the weight parameter that is dynamically adjusted as the number of training rounds increases, and ò is a hyperparameter.

[0017] In S3, for the candidate proposed positive sample segment, the object and motion label are detected by the pre-trained object detection model yolov5s and the motion detection model S3D, and the segment features, object label features, and motion label features are fused using Transformer Encorder to obtain an enhanced proposed positive sample segment representation.

[0018] The query text reconstruction module in S4 evaluates the query text reconstruction capabilities of the candidate proposal positive sample segments and the candidate proposal negative sample segments, evaluates the degree of semantic alignment between the segments and the query text, and applies the query text label similarity metric to evaluate the text similarity between the query text and the object and action labels.

[0019] The query text reconstruction is to mask 1 / 3 of the nouns or verbs in the query text, and input the masked query text features and the proposed segment features into the Transformer Decoder. The query text reconstruction loss L rec The calculation expression is:

[0020]

[0021] Among them, wmsk is the masked word, that is, the probability that the word in the predicted vocabulary is a masked word, q msk is the masked query text feature, pro p is the proposed segment feature, f dec TransformerDecoder module for query text reconstruction.

[0022] According to a second aspect of the present invention, there is provided an electronic device, comprising a memory and a processor, wherein a computer program is stored in the memory, and the method described above is implemented when the processor executes the program.

[0023] According to a third aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein the program implements the method described above when executed by a processor.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] (1) Alleviate the mismatch problem between scenes and proposed clips: By introducing a scene distribution detection module and using a hierarchical clustering algorithm to cluster video frames, we can obtain the scene distribution information of the video, which helps the model better understand the structure of the video content, so that in the proposed clip generation stage, it can predict and generate positive sample clips that match the complexity of the video scene. This method significantly alleviates the mismatch problem between scenes and proposed clips and improves the accuracy of video clip retrieval.

[0026] (2) Improving the accuracy of cross-modal video clip retrieval: Combining multimodal information enhancement technology, by fusing video features, query text features, and scene distribution features, and using pre-trained object detection models and action detection models to identify the object and action label information of the clip, the multimodal feature representation is further enhanced, which can better understand the relationship between the query text and the video content, thereby improving the accuracy of cross-modal video clip retrieval.

[0027] (3) Improving the model’s ability to distinguish between different scenes: We use scene distribution information and feature similarity scoring functions to select negative sample segments. As training progresses, we gradually introduce more difficult negative samples. This helps the model better learn the semantic differences between query text and video, improves the model’s ability to distinguish between different scenes, and enables the model to continuously adapt to new challenges during training, thereby improving its generalization performance.

[0028] (4) Ensuring retrieval timeliness while maintaining effectiveness: By proposing the query text reconstruction loss and query text label similarity loss of the fragment, the fragment with the smallest loss is selected as the retrieval result. This not only ensures the accuracy of the retrieval, but also maintains the timeliness of the retrieval through an efficient computing process. While ensuring the timeliness of the retrieval, it can significantly improve the effectiveness and practicality of cross-modal video fragment retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a data flow diagram of each module in the weakly supervised video clip retrieval method based on multimodal information enhancement;

[0030] Figure 2 This is a training flowchart of the modules in the weakly supervised video clip retrieval method based on multimodal information enhancement;

[0031] Figure 3 The retrieval performance of the weakly supervised video clip retrieval method based on multimodal information enhancement on the Charades-STA dataset;

[0032] Figure 4 The retrieval performance of the weakly supervised video clip retrieval method based on multimodal information enhancement on the ActivityNet Captions dataset. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0034] like Figure 1 As shown in FIG. 1 , it is a data flow diagram of each module in the weakly supervised video clip retrieval method based on multimodal information enhancement. The specific steps include:

[0035] S1. Input the uncut video and the query text into the input representation module to obtain video features and query text features; at the same time, input the uncut video into the scene distribution detection module to cluster the video frames and obtain scene features;

[0036] S2, inputting the video features, query text features and scene features into the proposed segment generation module to obtain the predicted candidate proposed segments;

[0037] S3, inputting the candidate proposed segment into the proposed segment enhancement module to obtain multimodal enhanced segment features;

[0038] S4. In the proposed fragment evaluation module, the query text reconstruction loss and the query text label similarity loss of the fragment feature are calculated, the fragment with the smallest loss is selected as the retrieval result, and the start and end time of the fragment are output.

[0039] In S1, the input representation module uses a pre-trained 3D convolutional neural network to extract frame features from uncut videos to obtain video features; the query text is pre-processed by word segmentation and stop word removal, and the word vector representation of the query text is obtained using the pre-trained GloVe model to obtain query text features; the frame features are grouped using a hierarchical clustering algorithm, the video is divided into multiple continuous scene segments, and scene distribution information is constructed to obtain scene features; the video features and query text features are cross-modally interacted using the TransformerEncorder structure to obtain video features and query text features that integrate cross-modal information and contextual semantic information. The hierarchical clustering process first treats each frame as an independent cluster, and uses the Euclidean distance to calculate the visual similarity between frames. The agglomerative hierarchical clustering algorithm is applied to iteratively merge the most similar clusters according to the average chain distance until the specified distance threshold is reached, as expressed by:

[0040]

[0041] Among them, v i,k and v j,k are the kth feature components of frame i and frame j respectively, d represents the feature dimension, |C a | and |C b | respectively represent clusters C a and C b The number of clustered frames in represents the average chain distance.

[0042] Scene features i The calculation expression is:

[0043]

[0044] Among them, s i is the starting frame of the scene, e i is the end frame of the scene, N v is the total number of video frames, v j is frame j.

[0045] The proposed segment generation module of S2 splices the video, query text and scene features, obtains the fusion features through TransformerEncorder, and uses a multi-layer perceptron to predict the number, center position and length information of the candidate proposed positive sample segments. Each segment uses a Flatten Gaussian mask to represent the temporal structure. Construct an integer set I = {p min ,pmin +1,...,p max}Limit the number of proposed segments, use the MLP and Gumbel-Softmax method to calculate the number of candidate proposed segments, and the calculation expression is:

[0046] g = Gumbel-Softmax(MLP(h n ))∈| |I|

[0047]

[0048] Among them, N pos is the number of predicted proposal fragments, h n is the output feature of Transformer Encorder, and MLP is a multi-layer perceptron.

[0049] The center point and width of the candidate proposal segment are calculated as follows:

[0050]

[0051] Where σ(·) is the Sigmoid activation function, W p is a learnable weight matrix, c and w represent the center point and width of the proposed regression.

[0052] The calculation formula for using Flatten Gaussian mask to represent the timing structure is:

[0053]

[0054] Among them, β is the hyperparameter that controls the variance of the Gaussian curve, m p [i] is the mask value of fragment p at position i.

[0055] For each candidate proposal positive sample segment, a corresponding candidate proposal negative sample segment is generated in each scene segment, and the similarity score function of the distance between the center frame of the candidate proposal negative sample segment and the center frame of the corresponding candidate proposal positive sample segment in the cluster space is the smallest.

[0056] The similarity score function is obtained by feature similarity and spatiotemporal distance. For each candidate proposal positive sample segment Generate candidate negative samples in each scene segment Similarity score function The expression is:

[0057]

[0058] in, To calculate feature similarity using cosine similarity, is the temporal distance between the center points of the positive and negative sample segments, α(E) is the weight parameter that is dynamically adjusted as the number of training rounds increases, and ò is a hyperparameter.

[0059] In S3, for the candidate proposed positive sample fragments, the object and motion labels are detected by the pre-trained object detection model yolov5s and the motion detection model S3D, and the fragment features, object label features, and motion label features are fused using Transformer Encorder to obtain the enhanced proposed positive sample fragment representation.

[0060] The query text reconstruction module in S4 evaluates the query text reconstruction ability of candidate proposal positive sample fragments and candidate proposal negative sample fragments, evaluates the degree of semantic alignment between the fragments and the query text, and applies the query text label similarity measurement to evaluate the text similarity between the query text and object and action labels.

[0061] The query text reconstruction is to mask 1 / 3 of the nouns or verbs in the query text, and input the masked query text features and the proposed fragment features into the Transformer Decoder. The query text reconstruction loss L rec The calculation expression is:

[0062]

[0063] Among them, w msk is the masked word, that is, the probability that the word in the predicted vocabulary is a masked word, q msk is the masked query text feature, pro p is the proposed segment feature, f dec TransformerDecoder module for query text reconstruction.

[0064] like Figure 2 As shown, it is a training flow chart of the modules in the weakly supervised video clip retrieval method based on multimodal information enhancement.

[0065] like Figure 3As shown, it is the retrieval performance result of the method of this embodiment on the public dataset Charades-STA. The evaluation index used is R@n, IoU=m, which means the ratio of IoU ≥ m between the predicted segment and the real segment in the first n retrieval results. IoU is an indicator to measure the similarity between two segments (predicted segment and real segment), which is defined as the ratio between the intersection and union of the predicted segment and the real segment. In the experimental setting, n is set to 1 or 5 respectively; m is set to 0.3, 0.5 or 0.7. The results show that the method in this embodiment shows excellent retrieval accuracy in all indicators, indicating that the method has a strong ability to understand and retrieve short-term complex video events. As Figure 4 The figure shows the retrieval performance results on the public dataset ActivityNetCaptions. The evaluation indicators used are R@n, IoU=m. In the experimental settings, n is set to 1 or 5; m is set to 0.1, 0.3 or 0.5. The results show that the method proposed in this paper has excellent retrieval accuracy in all indicators, indicating that in long video scenes, the method has the ability to segment and retrieve complex events.

[0066] Description of the datasets used above: Charades-STA is a dataset for text-described video clip retrieval tasks based on the Charades dataset. It contains 12,408 video-query text pairs for training and 3,720 video-query text pairs for verification. The average video length is 30.6 seconds, and the average target clip length is 8.1 seconds. ActivityNet Captions is extended from the ActivityNet dataset and is specifically used for long video clip retrieval tasks. It contains 37,417 video-query text pairs for training and 17,505 video-query text pairs for verification. The average video length is 117.6 seconds, and the average length of the target clip is 36.2 seconds. The semantic range and time span of the clips vary greatly.

[0067] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0068] The electronic device of the present invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0069] A plurality of components in the device are connected to the I / O interface, including: an input unit, such as a keyboard, a mouse, etc.; an output unit, such as various types of displays, speakers, etc.; a storage unit, such as a disk, an optical disk, etc.; and a communication unit, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit allows the device to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks. The processing unit performs the various methods and processes described above, such as the method of the present invention. For example, in some embodiments, the method of the present invention may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into the RAM and executed by the CPU, one or more steps of the method of the present invention described above may be performed. Alternatively, in other embodiments, the CPU may be configured to perform the method of the present invention in any other appropriate manner (e.g., by means of firmware).

[0070] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0071] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0072] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0073] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A weakly supervised video clip retrieval method based on multimodal information enhancement, characterized in that: The specific steps include: S1. Input the uncut video and the query text into the input representation module to obtain video features and query text features; at the same time, input the uncut video into the scene distribution detection module to cluster the video frames and obtain scene features; S2, inputting the video features, query text features and scene features into a proposed segment generation module to obtain predicted candidate proposed segments; S3, inputting the candidate proposed segment into the proposed segment enhancement module to obtain multimodal enhanced segment features; S4. Calculate the query text reconstruction loss and the query text label similarity loss of the segment feature in the proposed segment evaluation module, select the segment with the smallest loss as the retrieval result, and output the start and end time of the segment.

2. According to claim 1, a weakly supervised video segment retrieval method based on multimodal information enhancement is characterized in that: In S1, the input representation module uses a pre-trained 3D convolutional neural network to extract frame features from the uncut video to obtain video features; The query text is preprocessed by segmenting and removing stop words, and the word vector representation of the query text is obtained using the pre-trained GloVe model to obtain the query text features; the hierarchical clustering algorithm is used to group the frame features, the video is divided into multiple continuous scene segments, the scene distribution information is constructed, and the scene features are obtained; The Transformer Encorder structure is used to perform cross-modal interaction on video features and query text features to obtain video features and query text features that integrate cross-modal information and contextual semantic information.

3. The weakly supervised video segment retrieval method based on multimodal information enhancement according to claim 1, characterized in that: The proposed segment generation module of S2 splices video, query text and scene features, obtains fused features through TransformerEncorder, and uses a multi-layer perceptron to predict the number, center position and length information of candidate proposed positive sample segments. Each segment uses a Flatten Gaussian mask to represent the temporal structure.

4. The weakly supervised video segment retrieval method based on multimodal information enhancement according to claim 3 is characterized in that: For each candidate proposed positive sample segment, a corresponding candidate proposed negative sample segment is generated in each scene segment, and the similarity score function of the distance between the central frame of the candidate proposed negative sample segment and the central frame of the corresponding candidate proposed positive sample segment in the cluster space is minimized.

5. The weakly supervised video segment retrieval method based on multimodal information enhancement according to claim 4, characterized in that: The similarity score function is obtained by feature similarity and spatiotemporal distance. For each candidate proposal positive sample segment Generate candidate negative samples in each scene segment Similarity score function The expression is: in, To calculate feature similarity using cosine similarity, is the temporal distance between the center points of the positive and negative sample segments, α(E) is the weight parameter that is dynamically adjusted as the number of training rounds increases, and ò is a hyperparameter.

6. The weakly supervised video segment retrieval method based on multimodal information enhancement according to claim 1, characterized in that: In S3, for the candidate proposed positive sample segment, the object and motion label are detected by the pre-trained object detection model yolov5s and the motion detection model S3D, and the segment features, object label features, and motion label features are fused using Transformer Encorder to obtain an enhanced proposed positive sample segment representation.

7. The weakly supervised video segment retrieval method based on multimodal information enhancement according to claim 1, characterized in that: The query text reconstruction module in S4 evaluates the query text reconstruction capabilities of the candidate proposal positive sample segments and the candidate proposal negative sample segments, evaluates the degree of semantic alignment between the segments and the query text, and applies the query text label similarity metric to evaluate the text similarity between the query text and the object and action labels.

8. The weakly supervised video segment retrieval method based on multimodal information enhancement according to claim 7, characterized in that: The query text reconstruction is to mask 1 / 3 of the nouns or verbs in the query text, and input the masked query text features and the proposed segment features into the Transformer Decoder. The query text reconstruction loss L rec The calculation expression is: Among them, w msk is the masked word, that is, the probability that the word in the predicted vocabulary is a masked word, q msk is the masked query text feature, pro p is the proposed segment feature, f dec Transformer Decoder module for query text reconstruction.

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images