Long video retrieval method and device based on multi-scale multi-example similarity learning

Abstract

The application discloses a long video retrieval method and device based on multi-scale multi-example similarity learning. The method acquires video and text preliminary features; uses coarse-to-fine coding mode to extract information of different time granularities from video segment scale and frame scale; based on video representation of two scales, uses segment scale similarity learning branch to filter out video segments most relevant to the text and obtain segment scale similarity; uses frame scale similarity learning branch to aggregate video features guided by the filtered most relevant video segments to obtain more detailed video information, and after similarity calculation with the text, frame scale similarity is obtained; a common space learning algorithm is used to learn multi-scale similarity between long videos and texts, and a model is trained in an end-to-end manner to realize text-to-long video retrieval. The application uses the idea of multi-scale multi-example learning, and can effectively solve the text-to-long video retrieval task.

Description

Technical Field

[0001] This invention relates to the field of video cross-modal retrieval technology, and in particular to a long video retrieval method and apparatus based on multi-scale multi-instance similarity learning. Background Technology

[0002] With the advent of the big data era, millions of videos are uploaded to the internet every day. Users' demand for retrieving videos from big data is increasing daily. Since users generally express their information needs using natural language queries, research on text-to-video retrieval is crucial. Given a query in the form of a natural language sentence, the traditional task of text-to-video retrieval is to retrieve videos from a video library that are semantically relevant to the given query.

[0003] Traditional text-to-video retrieval methods are primarily trained on datasets generated from video descriptions. In these datasets, videos are pre-edited, and the provided text effectively describes the key points of the video content. Therefore, the pre-edited short video content and corresponding text are fully relevant. However, in reality, since the user's text query is not prior, the target video is unedited and may be quite long. Only a portion of the content in such long videos is relevant to the user's text query. This leads to poor performance of traditional text-to-video retrieval methods in real-world applications. Based on these conclusions, this invention proposes a long-video retrieval task that is more practically oriented than traditional text-to-video retrieval tasks. This task aims to retrieve target videos that are partially relevant to the query text from a large number of unedited, long-duration videos. Since traditional text-to-video retrieval tasks do not require consideration of the partial correlation between text and video, their model typically involves pre-encoding the video and text and mapping them to a common space for cross-modal similarity calculation. Therefore, traditional video retrieval models focus on the design of video and text encoders and cross-modal similarity learning algorithms, while long video retrieval tasks require models to focus more on the mining and measurement of partial correlations between query text and corresponding long videos. Summary of the Invention

[0004] The purpose of this invention is to address the limitations of traditional text-to-video retrieval tasks in reality by proposing a long video retrieval method and apparatus based on multi-scale and multi-instance similarity learning. The long video to be retrieved contains both specific moments related to the corresponding text and a large amount of content unrelated to the corresponding text.

[0005] The objective of this invention is achieved through the following technical solution: a long video retrieval method based on multi-scale, multi-instance similarity learning, comprising the following steps:

[0006] (1) Perform feature pre-extraction on the query text and the video to be retrieved to obtain initial text features and initial video features;

[0007] (2) Encode the initial text features obtained in step (1) to obtain the text feature representation;

[0008] (3) The initial video features obtained in step (1) are encoded into segment scale features and frame scale features respectively to obtain video segment scale feature representation and video frame scale feature representation;

[0009] (4) Construct a multi-scale similarity learning network model, which includes a similarity learning branch based on segment-scale video representation and a similarity learning branch based on frame-scale video representation;

[0010] (5) Input the video segment scale feature representation obtained in step (3) and the text feature representation obtained in step (2) into the similarity learning branch based on the segment scale video representation, perform similarity calculation, obtain the segment scale similarity of video and text, and detect key video segments containing text content.

[0011] (6) Input the video frame scale feature representation obtained in step (3), the key video segment obtained in step (5), and the text feature representation obtained in step (2) into the similarity learning branch based on the frame scale video representation, aggregate the video frame scale feature representation and the key video segment to obtain the aggregated frame scale feature representation, calculate the similarity with the text feature representation, and obtain the frame scale similarity between the video and the text.

[0012] (7) The video and text segments obtained in steps (5) and (6) are weighted and summed with the frame scale similarity to obtain the final text and video similarity, and a multi-scale similarity learning network model is trained.

[0013] (8) A trained multi-scale similarity learning network model is obtained through step (7). Videos and texts are input into the trained model to achieve cross-modal retrieval of texts to their partially related videos.

[0014] Furthermore, step (1) uses different pre-trained models to extract initial features from the text and video, including the following steps:

[0015] (1-1) Use pre-trained 2D and 3D deep convolutional networks to extract initial video features;

[0016] (1-2) Use the pre-trained large text feature extractor RoBERTa model to extract initial text features.

[0017] Furthermore, the method for encoding the initial text features obtained in step (1) in step (2) includes the following steps:

[0018] (2-1) The initial text features of the input are reduced in dimensionality using a fully connected layer, and then the position embedding encoding is performed before being input into the transformer for encoding.

[0019] (2-2) The text features encoded in step (2-1) are aggregated using an attention module to obtain the final encoded text features.

[0020] Furthermore, the method for obtaining the scale feature representation of the video segment in step (3) includes the following steps:

[0021] (3-1) After downsampling the initial video features obtained in step (1) to a fixed size, the dimensionality is reduced using a fully connected layer, and the location embedding is encoded and then input into the transformer for encoding;

[0022] (3-2) For the encoded video features in step (3-1), feature selection for different video segment sizes is performed using a sliding window to obtain the video segment scale feature representation.

[0023] Furthermore, the method for obtaining the video frame scale feature representation in step (3) is as follows: the initial video features obtained in step (1) are reduced in dimensionality using a fully connected layer, and their position embedding encoding is then input into a transformer for encoding to obtain the video frame scale feature representation.

[0024] Furthermore, the method for obtaining the segment scale similarity between the video and the text in step (5) is as follows:

[0025] The similarity between the video segment scale feature representation obtained in step (3) and the text feature representation obtained in step (2) is calculated to obtain the similarity between each video segment and the text. The maximum value is taken as the segment scale similarity between the video and the text, and the corresponding segment with the highest similarity is selected as the key video segment.

[0026] Furthermore, the method for obtaining the frame scale similarity between the video and the text in step (6) is as follows:

[0027] (6-1) The video frame scale feature representation obtained in step (3) is mapped using two different fully connected layers to obtain two sets of mapped frame scale feature representations.

[0028] (6-2) Calculate the similarity between one set of frame scale feature representations and the key video segments obtained in step (5) to obtain the similarity between each video frame and the key video segments;

[0029] (6-3) For another set of frame scale feature representations, the similarity between each video frame obtained in step (6-2) and the key video segment is used as the weight to perform a weighted sum, resulting in the aggregated video frame scale feature representations.

[0030] (6-4) Calculate the similarity between the aggregated video frame scale feature representation obtained in step (6-3) and the text feature representation obtained in step (2) to obtain the frame scale similarity between the video and the text.

[0031] Furthermore, in step (7), a multi-scale similarity learning network model is trained using a multi-instance learning approach. The correlation between the video and text modalities is learned through ternary ranking loss and contrastive learning loss, and the multi-scale similarity learning network model is trained end-to-end, so that the model can automatically learn the correlation between the video and text modalities.

[0032] Furthermore, step (8) specifically includes:

[0033] (8-1) The query text is represented by features, and all candidate videos are represented by segment and frame scale features;

[0034] (8-2) Input the feature representations of the text and video into the trained multi-scale similarity learning network model, calculate the segment and frame scale similarity between the query text and all candidate videos, sort the candidate videos according to the weighted sum of the two similarities, and return the retrieval results.

[0035] On the other hand, the present invention provides a long video retrieval device based on multi-scale multi-instance similarity learning, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it implements the long video retrieval method based on multi-scale multi-instance similarity learning.

[0036] The beneficial effects of this invention are as follows: This invention provides a long video retrieval method and apparatus based on multi-scale, multi-instance similarity learning. In the method, the long video is represented as features at multiple segment and frame scales. After multi-scale representation of the long video, it is input into a multi-scale similarity learning network. The multi-scale similarity learning network includes a similarity learning branch based on segment-scale video representation and a similarity learning branch based on frame-scale video representation. Key segments of the long video are detected in the segment-scale video representation similarity learning branch. Then, segment-scale similarity is calculated based on the similarity between the key segments of the long video and the query text. Furthermore, the key segments serve as encoding guides in the frame-scale video representation branch, measuring the importance of each frame at fine time scales and weighting the feature representations of all frames into a single feature representation. The similarity between the calculated single feature representation and the query text is used as the frame-scale similarity. Finally, segment-scale similarity and frame-scale similarity are used to jointly measure the similarity between the long video and the query text. This invention utilizes the idea of ​​multi-scale, multi-instance learning; multi-scale feature representation helps handle related segments of different lengths between long videos and corresponding text. Meanwhile, the frame-scale similarity learning branch and the segment-scale similarity learning branch in the model network of this invention have a mutually reinforcing effect on long video representation. When the segment-scale similarity learning branch may have insufficient understanding of the video, the frame-scale similarity learning branch can help the segment-scale similarity learning branch supplement the missing information. The network model proposed in this invention utilizes the above branch modules to deeply model the partial correlation between text and corresponding long videos, thereby effectively solving the text-to-long video retrieval task. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of a long video retrieval method based on multi-scale, multi-instance similarity learning provided by the present invention.

[0038] Figure 2 This is a schematic diagram of the multi-scale similarity learning module structure of the present invention.

[0039] Figure 3 This is a schematic diagram illustrating a retrieval example according to an embodiment of the present invention.

[0040] Figure 4 This is a schematic diagram of a long video retrieval device based on multi-scale, multi-instance similarity learning provided by the present invention. Detailed Implementation

[0041] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0042] To address the task of long video retrieval in practical applications, this invention proposes a long video retrieval method based on multi-scale, multi-instance similarity learning. This method effectively mines partial relevance between long videos and their corresponding text. In this method, long videos are represented as features at multiple segment and frame scales. This multi-scale feature representation helps handle related segments of different lengths between the long video and the corresponding text. After multi-scale representation of the long video, it is input into a multi-scale similarity learning network. In this network, the segment scale, which typically represents a longer duration in the video, is considered a coarse-grained temporal representation. Correspondingly, the frame scale reflects more detailed video content and is considered a fine-grained temporal representation. The multi-scale similarity learning network includes a similarity learning branch based on segment-scale video representation and a similarity learning branch based on frame-scale video representation. They learn video representations together in a coarse-to-fine manner, and the two similarity learning branches interact. Key segments of the long video are detected in the segment-scale video representation similarity learning branch. Then, segment-scale similarity is calculated based on the similarity between the key segments of the long video and the query text. Furthermore, key segments are used as encoding guides for frame-scale video representation branches to measure the importance of each frame at fine temporal scales, and the feature representations of all frames are weighted and summed into a single feature representation. The similarity between the calculated single feature representation and the query text is used as the frame-scale similarity. Finally, segment-scale similarity and frame-scale similarity are used to jointly measure the similarity between the long video and the query text. Figure 1 and Figure 2 As shown, the specific steps of this invention are as follows:

[0043] (1) Use different feature extraction methods to extract features from video and text modalities respectively.

[0044] (1-1) Given a sentence by n q For sentences composed of 100 words, a pre-trained RoBERTa model is used to extract the feature vector set of each word. As the initial feature of the text, Indicates the nth q Feature vectors of each word.

[0045] (1-2) Given a video, first specify a sequence of video frames with a pre-defined interval of 1.5 seconds to obtain n v The video frame sequence was analyzed using ResNet152, a 2D deep convolutional network pre-trained on ImageNet, and I3D, a 3D deep convolutional network pre-trained on Kinetics. As the initial set of feature vectors for the video, Indicates the nth v The initial feature vector of each video frame.

[0046] Through the feature extraction steps described above, the initial features of the video and text were obtained respectively. Next, they need to be represented in a deeper level.

[0047] (2) First, input the initial text features obtained in step (1) into the sentence feature representation module for encoding. The specific steps are as follows:

[0048] (2-1) The initial 768-dimensional text features are reduced to 384-dimensional by a fully connected layer (FC) and the ReLU activation function, and the reduced text features are then encoded by position embedding.

[0049] (2-2) The text features obtained in step (2-1) are input into the transformer to capture its contextual information. In the transformer, the text features pass through a multi-head self-attention layer and a feedforward layer in sequence. Both of these encoding layers are accompanied by residual connections and layer normalization operations, i.e.:

[0050] Q′=Transformer(ReLu(FC(Q))+PE)

[0051] In the above formula, Transformer represents the standard transformer module, and PE represents the position embedding coding module.

[0052] (2-3) The text features obtained in step (2-2) are processed through an attention mechanism. Aggregate the single-dimensional text feature vector. Perform a dot product between the weight vector w and Q′, and then pass it through a softmax layer to obtain n. q Each weight αq is used. The calculated weights are then used to weight and sum the text features Q′ to obtain the final text feature q, i.e.:

[0053] α q =Softmax(w T Q′)

[0054] (3) Input the initial video features obtained in step (1) into the segment-scale video representation module for encoding. The specific steps are as follows:

[0055] (3-1) Before constructing the video segment, the input is first downsampled in the temporal domain to reduce the length of the initial video feature sequence and to help reduce the computational complexity of the encoding module. For the initial video features... Downsample it to a length of n u eigenvectors Indicates the nth u Each video feature vector.

[0056] (3-2) The downsampled video feature vector U obtained in step (3-1) is dimensionality reduced using a fully connected layer and a ReLU activation function, while position embedding encoding is performed. Further, it is input into a transformer with the same structure as in step (2-2) to capture its contextual information, resulting in the encoded video feature vector U′, i.e.:

[0057] U′=Transformer(ReLu(FC(U))+PE)

[0058] (3-3) Use sliding windows of different sizes to traverse along the time dimension with a step size of 1. During the traversal, average pooling is performed on the features falling within the sliding window to obtain the video segment feature sequence φ of the corresponding size. k Its visualization process is as follows: Figure 1 The middle segment construction module is shown. This is achieved by simultaneously using segments of size from 1 to n. u A sliding window is used to obtain a set of feature sequences for video segments. This indicates the use of size n u The set of video segment feature sequences obtained by traversing the video segment features using a sliding window, and the final video segment feature sequence obtained by expanding the set of video segment feature sequences. n c =n u (n u +1) / 2.

[0059] (4) Input the initial video features obtained in step (1) into the frame-scale video representation module for encoding. The specific steps are as follows:

[0060] (4-1) The initial 3072-dimensional video features are reduced to 384-dimensional by a fully connected layer (FC) and the ReLU activation function, and the reduced video features are then encoded by position embedding (PE).

[0061] (4-2) Input it into a transformer with the same structure as in step (2-2) to capture its contextual information, and obtain the frame-scale feature sequence F of the video, i.e.:

[0062] F=Transformer(ReLu(FC(V))+PE)

[0063] In the above steps, the fully connected layer (FC), position embedding coding (PE), and transformer coding modules used in the sentence feature representation coding module, video segment scale feature representation coding module, and video frame scale feature representation coding module all have the same structure, but do not share parameters.

[0064] Through the above steps, the feature representation of the text is obtained. Video segment scale feature sequence and the frame-scale feature sequence of the video Since the specific location of the text within its corresponding long video is not provided during training, directly calculating similarity at a fine-grained scale is challenging. Therefore, a multi-scale similarity learning network model is constructed, including branches based on segment-scale video representation and frame-scale video representation. This multi-scale similarity learning method measures the similarity between text and corresponding long videos in a coarse-to-fine calculation manner. This calculation method is based on the assumption that if the model knows the rough correlation between the long video and the corresponding text, it can help the model accurately find more relevant content at a finer granular scale. First, the key video segments most relevant to the text in the long video are detected. Using these key video segments as further guidance, the importance of each frame in the long video is measured at a fine-grained temporal scale. Finally, the segment-scale and frame-scale similarity between the query text and the long video are jointly considered as the final similarity score.

[0065] (5) Input the video segment feature sequence C obtained in step (3-3) and the text feature representation q obtained in step (2-3) into the similarity learning branch based on segment-scale video representation, and perform similarity calculation. For each video segment feature c in C... i The cosine similarity is calculated with q, and the maximum value is taken as the scale similarity S of the video segment. c (v, q), and extract the corresponding video segment features. As key video segment features. That is:

[0066]

[0067] (6) Input the video frame scale feature representation, key video segment features, and text feature representation into the similarity learning branch based on the frame scale video representation, and calculate the video frame scale similarity. The specific steps are as follows:

[0068] (6-1) The feature sequence F at the video frame scale obtained in step (4-2) is processed through a learnable mapping matrix. and These are mapped to the key feature sequence K and the value feature sequence Z, respectively:

[0069] K = W k F, Z = W v F

[0070] (6-2) The key video segment features obtained in step (5) The query features are multiplied by the key feature sequence obtained in step (6-1) and then input into the Softmax layer to obtain the aggregated weights of the video frame scale features. The dot product measures the similarity between the video frame and the key video segment; therefore, video frames more similar to the key video segment will have a larger aggregated weight. Finally, the aggregated weights are used to aggregate the value feature sequence obtained in step (6-1) to obtain the video frame scale feature representation vector. Calculate the cosine similarity between r and the text feature representation q as the video frame scale similarity S. f (v, q), that is:

[0071]

[0072] S f (v, q) = cos(r, q)

[0073] Through the above steps, the segment scale similarity S between the text and the video was obtained. c (v, q) and frame scale similarity S f (v, q), then the common space learning algorithm is used to learn the correlation between the two text and video modalities and train a multi-scale similarity learning network model. The specific steps are as follows:

[0074] (7-1) In multiple instance learning, a sample is considered a bag consisting of a large number of examples. If one or more examples in the bag are positive samples, then the bag is a positive sample; otherwise, the bag is a negative sample. A long video as a whole can be considered a bag, and each frame or segment composed of frames of different sizes in the video can be considered a different example. If the text is related to a frame or segment of a long video, then the text is considered to be related to the long video. Therefore, long video retrieval tasks and multiple instance learning are highly correlated. According to the definition of multiple instance learning, if a video contains a segment that corresponds to the query text, then such a video-text pair is considered a positive sample pair; otherwise, it is considered a negative sample pair. Based on the above definition, the triplet ranking loss and contrastive learning loss, which are widely used in retrieval tasks, are used to jointly constrain the model. The triplet ranking loss... The formula is:

[0075]

[0076] Where m is a boundary constant with a value of 0.1, and S(·) is a similarity function, which can be expressed as fragment-scale similarity S. c (·) or frame scale similarity S f (·). q - and v - The negative text samples are for video v and the negative video samples are for text q, respectively. The negative samples are generated from the small batch data at the beginning of training. n data points are randomly selected from the dataset, and a mini-batch of data is taken after 20 training epochs. The most difficult negative sample in the dataset.

[0077] Secondly, compare the learning loss The formula is:

[0078]

[0079] in Represents small batches of data The set of all negative text samples corresponding to the mid-video v. This represents the i-th negative text sample. Represents small batches of data The set of all negative samples corresponding to the mid-video q. Let i represent the i-th negative sample from the video.

[0080] The final loss used when training the model for:

[0081]

[0082] in and These represent the triplet ranking losses using fragment scale similarity and frame scale similarity, respectively. and λ1 and λ2 represent the contrastive learning losses using fragment scale similarity and frame scale similarity, respectively. λ1 = 0.02 and λ2 = 0.04 are set to balance the initial weights of each loss at the start of training.

[0083] (8) Through the above steps, a well-trained multi-scale similarity learning network model was obtained. The specific steps for it to achieve cross-modal retrieval from text to its partially related videos are as follows:

[0084] (8-1) Input text and a set of candidate videos. For a given text and a candidate video, calculate the scale similarity S between the text and its segments. c (v, q) and frame scale similarity S f (v, q) are weighted and summed to obtain the final text similarity S(v, q), i.e.:

[0085] S(v, q) = α*S c (v, q) + (1-α)*S f (v, q)

[0086] After experimenting with α from 0 to 1, the results showed that the network model achieved optimal performance when α was 0.7.

[0087] (8-2) Sort the similarity between the text and all candidate videos, and take the result with the best similarity as the final returned retrieval result, so as to realize the cross-modal retrieval task from text to its partially related videos.

[0088] Implementation Examples

[0089] This invention selects a widely used TV show retrieval dataset to train the proposed network model, thereby demonstrating the effectiveness of the invention in practical applications. The TV show retrieval dataset extracts approximately 22,000 long videos from six major TV programs, each video corresponding to five specific descriptive statements. These descriptions contain only a portion of the corresponding long video, aligning with real-world long video retrieval scenarios. Figure 3 This is a retrieval example of the network model proposed in this invention on a TV program video retrieval dataset after training.

[0090] Given query text 1: Monica is wearing a gray plaid shirt. The network model proposed in this invention first encodes the query text into a separate feature representation. Then, it performs segment-scale feature representation on all candidate videos in the video library, meaning that each video will generate multiple candidate segments. After calculating the similarity between the text's feature representation and the multiple candidate segments of each video, for each video, the candidate segment with the highest similarity is selected as the key segment of that video. Simultaneously, the corresponding similarity is retained as the segment-scale similarity between the text and the video. Next, the key segments of each video are used to perform frame-scale feature representation on each video and calculate the similarity with the text features, obtaining the frame-scale similarity between the text and each video. Finally, the segment-scale similarity and frame-scale similarity of the query text and each video are weighted and summed to obtain the similarity between the query text and all candidate videos in the video library. After sorting, the final query result is obtained. It can be seen that... Figure 3 In the example, for query text 1, the key segment returned by the network model of this invention completely overlaps with the correct segment corresponding to the query text in the video. Furthermore, the video corresponding to query text 1 ranks second in final similarity score among all candidate videos. The same applies to query text 2.

[0091] Specifically, for query text 3: Chandler and Monica are comforting their child in their arms, the network model provides only two frames as the key segment. However, the correct segment corresponding to the query text in the video consists of the first 10 frames, with only a small overlap. At this point, most frames in the correct segment show relatively high similarity. Therefore, the video corresponding to query text 3 ranks first in final similarity score among all candidate videos. This indicates that the frame-scale similarity learning branch in the model network of this invention can help the segment-scale similarity learning branch supplement missing information. The same applies to query text 4 and query text 3.

[0092] Corresponding to the aforementioned embodiments of the long video retrieval method based on multi-scale multi-instance similarity learning, the present invention also provides embodiments of a long video retrieval device based on multi-scale multi-instance similarity learning.

[0093] See Figure 4 The present invention provides a long video retrieval device based on multi-scale multi-instance similarity learning, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement the long video retrieval method based on multi-scale multi-instance similarity learning in the above embodiments.

[0094] The embodiments of the long video retrieval device based on multi-scale, multi-instance similarity learning of this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 4 The diagram shown is a hardware structure diagram of any device with data processing capabilities, which is the long video retrieval device based on multi-scale multi-instance similarity learning according to the present invention. (Except for...) Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0095] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0096] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0097] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the long video retrieval method based on multi-scale multi-instance similarity learning described in the above embodiments.

[0098] The computer-readable storage medium can be an internal storage unit of any data processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0099] The above embodiments are used to explain and illustrate the present invention, but not to limit the present invention. Any modifications and changes made to the present invention within the spirit and scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A long video retrieval method based on multi-scale, multi-instance similarity learning, characterized in that, Includes the following steps: (1) Perform feature pre-extraction on the query text and the video to be retrieved to obtain initial text features and initial video features; (2) Encode the initial text features obtained in step (1) to obtain text feature representations; (3) The initial video features obtained in step (1) are encoded into segment scale features and frame scale features respectively to obtain video segment scale feature representation and video frame scale feature representation; (4) Construct a multi-scale similarity learning network model, which includes a similarity learning branch based on segment-scale video representation and a similarity learning branch based on frame-scale video representation; This method employs a multi-scale similarity learning approach to measure the similarity between text and corresponding long videos using a coarse-to-fine calculation method. First, it detects the key video segments in the long video that are most relevant to the text. Then, using these key video segments as further guidance, it measures the importance of each frame in the long video at a fine-grained temporal scale. Finally, it considers the segment-scale and frame-scale similarity between the query text and the long video as the final similarity score. (5) Input the video segment scale feature representation obtained in step (3) and the text feature representation obtained in step (2) into the similarity learning branch based on the segment scale video representation, perform similarity calculation, obtain the segment scale similarity of video and text, and detect key video segments containing text content. (6) Input the video frame scale feature representation obtained in step (3), the key video segment obtained in step (5), and the text feature representation obtained in step (2) into the similarity learning branch based on the frame scale video representation, aggregate the video frame scale feature representation and the key video segment to obtain the aggregated frame scale feature representation, calculate the similarity with the text feature representation, and obtain the frame scale similarity between the video and the text. (7) The video and text segments obtained in steps (5) and (6) are weighted and summed with the frame-scale similarity to obtain the final text-video similarity, and a multi-scale similarity learning network model is trained. (8) A trained multi-scale similarity learning network model is obtained through step (7). The video and text are input into the trained model to realize cross-modal retrieval from the text to its partially related video.

2. The long video retrieval method based on multi-scale, multi-instance similarity learning according to claim 1, characterized in that, Step (1) uses different pre-trained models to extract initial features from text and video, including the following steps: (1-1) Use pre-trained 2D and 3D deep convolutional networks to extract initial video features; (1-2) Use the pre-trained large text feature extractor RoBERTa model to extract initial text features.

3. The long video retrieval method based on multi-scale, multi-instance similarity learning according to claim 1, characterized in that, The method for encoding the initial text features obtained in step (1) in step (2) includes the following steps: (2-1) The initial text features of the input are reduced in dimensionality using a fully connected layer, and then the position embedding encoding is performed before being input into the transformer for encoding; (2-2) The text features encoded in step (2-1) are aggregated using an attention module to obtain the final encoded text features.

4. The long video retrieval method based on multi-scale multi-instance similarity learning according to claim 1, characterized in that, The method for obtaining the scale feature representation of the video segment in step (3) includes the following steps: (3-1) After downsampling the initial video features obtained in step (1) to a fixed size, use a fully connected layer to reduce the dimensionality, and then input the position embedding encoding into the transformer for encoding. (3-2) For the encoded video features in step (3-1), feature selection for different video segment sizes is performed using a sliding window to obtain the video segment scale feature representation.

5. The long video retrieval method based on multi-scale, multi-instance similarity learning according to claim 1, characterized in that, The method for obtaining the video frame scale feature representation in step (3) is as follows: the initial video features obtained in step (1) are reduced in dimensionality using a fully connected layer, and their position embedding encoding is then input into a transformer for encoding to obtain the video frame scale feature representation.

6. The long video retrieval method based on multi-scale multi-instance similarity learning according to claim 1, characterized in that, The method for obtaining the segment scale similarity between video and text in step (5) is as follows: The similarity between the video segment scale feature representation obtained in step (3) and the text feature representation obtained in step (2) is calculated to obtain the similarity between each video segment and the text. The maximum value is taken as the segment scale similarity between the video and the text, and the corresponding segment with the highest similarity is selected as the key video segment.

7. The long video retrieval method based on multi-scale multi-instance similarity learning according to claim 1, characterized in that, The method for obtaining the frame scale similarity between the video and the text in step (6) is as follows: (6-1) The video frame scale feature representation obtained in step (3) is mapped using two different fully connected layers to obtain two sets of mapped frame scale feature representations; (6-2) Calculate the similarity between one set of frame scale feature representations and the key video segments obtained in step (5) to obtain the similarity between each frame of the video and the key video segments; (6-3) For another set of frame scale feature representations, the similarity between each video frame obtained in step (6-2) and the key video segment is used as the weight to perform a weighted sum of the feature representations of each video frame to obtain the aggregated video frame scale feature representations. (6-4) Calculate the similarity between the aggregated video frame scale feature representation obtained in step (6-3) and the text feature representation obtained in step (2) to obtain the frame scale similarity between the video and the text.

8. The long video retrieval method based on multi-scale multi-instance similarity learning according to claim 1, characterized in that, In step (7), a multi-scale similarity learning network model is trained using a multi-instance learning approach. The correlation between the video and text modalities is learned through ternary ranking loss and contrastive learning loss. The multi-scale similarity learning network model is trained end-to-end, enabling the model to automatically learn the correlation between the video and text modalities.

9. The long video retrieval method based on multi-scale multi-instance similarity learning according to claim 1, characterized in that, The specific steps (8) are as follows: (8-1) The query text is represented by features, and all candidate videos are represented by segment and frame scale features; (8-2) Input the feature representations of the text and video into the trained multi-scale similarity learning network model, calculate the segment and frame scale similarity between the query text and all candidate videos, sort the candidate videos according to the weighted sum of the two similarities, and return the retrieval results.

10. A long video retrieval device based on multi-scale, multi-instance similarity learning, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that... When the processor executes the executable code, it implements the long video retrieval method based on multi-scale multi-instance similarity learning as described in any one of claims 1-9.

Images