Video-text cross-modal retrieval method based on dual contrast learning and related device

Abstract

The application provides a video-text cross-modal retrieval method based on double contrast learning and related equipment. The method comprises: obtaining a query sentence and at least one video; obtaining a query feature vector corresponding to the query sentence according to the query sentence; obtaining a video feature vector corresponding to each video according to the video; the video feature vector is jointly represented by a visual feature vector and a caption feature vector of the corresponding video; obtaining the similarity between each video and the query sentence according to the query feature vector and the video feature vector; determining the video corresponding to the maximum similarity as the target video. Based on the query-aware cross-modal double contrast learning mechanism, the application improves the efficiency and accuracy of video retrieval by performing fine-grained retrieval and positioning on videos and video segments.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video retrieval, and in particular to a video-text cross-modal retrieval method based on dual contrast learning and related equipment. BACKGROUND

[0002] With the development of the Internet and the era of self-media, the amount of video data is growing explosively, and people's retrieval needs are becoming more and more refined.

[0003] Modality refers to the form of data, for example, data exists in the form of video or text. Data with the same form and describing the same transaction or event can be considered as single-modal data; data with different forms but describing the same transaction or event can be considered as cross-modal data.

[0004] In related technologies, video retrieval is performed through single-modal features; or after extracting features of multi-modal data respectively and performing cross-modal similarity measurement, target videos are determined through video matching according to the features of the multi-modal data.

[0005] However, due to the inconsistency of the representation forms of different types of data, the data are in different distribution controls, and similarity measurement is difficult. Therefore, the related technologies have the problem of low video retrieval accuracy. SUMMARY

[0006] Therefore, the purpose of the present application is to provide a video-text cross-modal retrieval method based on dual contrast learning and related equipment.

[0007] To achieve the above purpose, the present application provides a video-text cross-modal retrieval method based on dual contrast learning, which comprises:

[0008] obtaining a query sentence and at least one video;

[0009] obtaining a query feature vector corresponding to the query sentence according to the query sentence;

[0010] obtaining a video feature vector corresponding to each video according to the video; the video feature vector is obtained by jointly representing the visual feature vector and the subtitle feature vector of the corresponding video;

[0011] obtaining the similarity between each video and the query sentence according to the query feature vector and the video feature vector;

[0012] determining the video corresponding to the maximum similarity as the target video.

[0013] Optionally, after determining the target video, the method further comprises:

[0014] determine a target video feature vector corresponding to the target video;

[0015] According to the target video feature vector and the query feature vector, a start time prediction score and an end time prediction score are calculated by the following formula:

[0016]

[0017]

[0018]

[0019] wherein, the target video feature vector is represented by , the query feature vector is represented by , and the one-dimensional convolution calculation function is represented by

[0020] According to the start time prediction score and the end time prediction score, a confidence score of the predicted video segment is calculated by the following formula:

[0021]

[0022]

[0023] wherein, the prediction start time probability and the prediction end time probability determined according to the start time prediction score and the end time prediction score are represented by , the prediction start time is represented by , the prediction end time is represented by , the function maximum value calculation function is represented by , the function with as the independent variable is represented by , the function with as the independent variable is represented by , and the confidence score of the predicted segment is represented by

[0024] In response to determining that the confidence score is greater than or equal to a preset threshold, the predicted segment is determined as a target segment.

[0025] Optionally, after obtaining the video feature vector, the method further comprises:

[0026] According to the query feature vector and the video feature vector, an optimized query feature vector and an optimized video feature vector are obtained by using a trained intra-modal representation optimization model and a trained inter-modal semantic fusion model.

[0027] ​​​​​​​​​The optimized query feature vector is used as the query feature vector, and the optimized video feature vector is used as the video feature vector.

[0028] Optionally, obtaining the similarity between each video and the query statement based on the query feature vector and the video feature vector includes:

[0029] Based on the query feature vector and the video feature vector, the similarity of each video is calculated using the following formula;

[0030] ;

[0031] in, Indicates the similarity between videos. This represents the video feature vector of the video. This represents the query feature vector of the video.

[0032] Optionally, obtaining the video feature vector corresponding to each video based on the video includes:

[0033] Based on the video, visual feature vectors and subtitle feature vectors are obtained;

[0034] Based on the visual feature vector and the query feature vector, the visual feature weights are obtained using the following formula;

[0035] ;

[0036] in, Represents the weights of visual features. This represents the first weight matrix. This represents the second weight matrix; the first weight matrix and the second weight matrix are obtained based on pre-training or manual preset.

[0037] Based on the visual feature vector and the query feature vector, the subtitle feature weights are obtained using the following formula;

[0038] ;

[0039] in, Indicates the subtitle feature weights. This represents the first weight matrix. This represents the second weight matrix;

[0040] Based on the visual feature weights and the subtitle feature weights, the video feature vector is calculated using the following formula;

[0041] ;

[0042] in, Represents the video feature vector. Represents the subtitle feature vector. This represents a visual feature vector.

[0043] Optionally, obtaining the visual feature vector and subtitle feature vector based on the video includes:

[0044] Based on the video, the initial visual feature vector and the initial subtitle feature vector are obtained by using the trained first feature vector extraction model and the second feature vector extraction model, respectively.

[0045] Based on the initial visual feature vector and the initial subtitle feature vector, the visual feature vector and the subtitle feature vector are obtained through a trained first neural network based on an attention mechanism.

[0046] Optionally, obtaining the query feature vector corresponding to the query statement based on the query statement includes:

[0047] Based on the query statement, the initial query feature vector is obtained through the trained third feature vector extraction model;

[0048] Based on the initial query feature vector, the query feature vector is obtained through a trained second neural network based on an attention mechanism.

[0049] Based on the same inventive concept, this application also provides a video text cross-modal retrieval device based on dual contrast learning, comprising:

[0050] The acquisition module is configured to retrieve the query statement and at least one video.

[0051] The first extraction module is configured to obtain the query feature vector corresponding to the query statement based on the query statement;

[0052] The second extraction module is configured to obtain a video feature vector corresponding to each video based on the video; the video feature vector is obtained by jointly representing the visual feature vector and the subtitle feature vector of the corresponding video;

[0053] The calculation module is configured to obtain the similarity between each video and the query statement based on the query feature vector and the video feature vector;

[0054] The target video determination module is configured to identify the video with the highest similarity as the target video.

[0055] Based on the same inventive concept, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the video text cross-modal retrieval method based on dual contrast learning as described in any of the above claims.

[0056] Based on the same inventive concept, this application also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute any of the above-described video text cross-modal retrieval methods based on dual contrast learning.

[0057] As can be understood from the above, the video-text cross-modal retrieval method and related device based on dual contrastive learning provided in this application obtain a jointly represented video feature vector by fusing the visual feature vector and text feature vector of the video, and then calculate the similarity between the video feature vector and the query feature vector according to a similarity function. In other words, this application, based on a query-aware cross-modal dual contrastive learning mechanism, improves the efficiency and accuracy of video retrieval by performing fine-grained retrieval and localization of videos and video segments. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is a schematic diagram of the video text cross-modal retrieval method based on dual contrast learning according to one or more embodiments of this application;

[0060] Figure 2 This is a schematic diagram of the structure of a video text cross-modal retrieval device based on dual contrast learning, according to one or more embodiments of this application.

[0061] Figure 3 This is a schematic diagram of the structure of a video text cross-modal retrieval system based on dual contrast learning, according to one or more embodiments of this application.

[0062] Figure 4 This is a schematic diagram of the hardware structure of an electronic device according to one or more embodiments of this application. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0064] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0065] As described in the background section, with the development of the Internet and the era of self-media, the amount of video data has exploded, and people's search needs have become increasingly refined.

[0066] Some related technologies employ unimodal retrieval methods for video retrieval. However, retrieval based on a single semantic meaning cannot fully utilize diverse data formats. For example, when retrieving videos related to "cats," these technologies typically select videos whose visual content includes cats based on image features. However, for videos where cats are mentioned in subtitles or audio but not visually, these methods fail to achieve effective retrieval. Therefore, video retrieval using unimodal retrieval methods suffers from low retrieval accuracy.

[0067] Some related technologies propose using multimodal retrieval methods for video retrieval. That is, multiple features are considered simultaneously during video retrieval. For example, when retrieving videos related to "cats," both image features and subtitle text features are considered. Although these methods take into account different types of features, due to the heterogeneity gap between different feature types, it is necessary to unify the distribution space of multimodal features and perform similarity measurements as the basis for video retrieval. Considering the difficulty of similarity measurement, the above methods still suffer from relatively low retrieval accuracy.

[0068] Therefore, this application proposes a video text cross-modal retrieval method based on dual contrast learning. By fusing multimodal video data to obtain aligned feature information, it solves the problem of difficulty in similarity measurement in related technologies, realizes video retrieval based on multimodal data, and further improves retrieval accuracy.

[0069] The technical solutions of one or more embodiments of this application will be described in detail below through specific examples.

[0070] refer to Figure 1 The video text cross-modal retrieval method based on dual contrast learning according to one or more embodiments of this application includes the following steps:

[0071] Step S101: Obtain the query statement and at least one video.

[0072] In this step, we first obtain the query statement and the video. The video data includes subtitle text.

[0073] Step S102: Obtain the query feature vector corresponding to the query statement based on the query statement.

[0074] In some embodiments, the query statement can be extracted using a feature vector extraction model to obtain the query feature vector corresponding to the query statement. In some implementations, an initial query feature vector can be obtained first using a feature vector extraction model, and then the initial query feature vector can be optimized using a neural network model in conjunction with the context of the query statement to obtain the final query feature vector.

[0075] In some embodiments, the feature vector extraction model described above can be a text feature extractor. In some embodiments, the neural network model described above can be a transformer neural network based on an attention mechanism. A transformer neural network based on an attention mechanism can be used to give more consideration to the contextual content during the process of obtaining the query feature vector.

[0076] In some embodiments, after obtaining the initial query feature vector through the feature vector extraction model, the obtained initial query feature vector can be projected into a special space of a preset dimension through a feedforward neural network, so that the final query feature vector also has a preset dimension.

[0077] Step S103: Based on the video, obtain the video feature vector corresponding to each video; the video feature vector is jointly represented by the visual feature vector and the subtitle feature vector of the corresponding video.

[0078] In this step, both the visual content features of the video and the text features of the subtitles are considered simultaneously. That is, multimodal joint representation of the video is achieved. In implementing this application, to fully learn the semantic similarity constraints within and between modalities, further enhance the semantic association learning ability of data from different modalities, and narrow the cross-modal semantic gap, the applicant proposes a cross-modal dual contrastive learning mechanism for video-text cross-modal retrieval. This mechanism combines inter-modal and intra-modal contrastive learning techniques to optimize intra-modal representations and fuse semantics between modalities, thereby better maintaining semantic similarity within and between modalities. This enhances the semantic alignment and fusion of data from different modalities, and improves the discriminativeness and semantic consistency of data representations from different modalities.

[0079] In some embodiments, for the video with subtitles obtained in step S101, two unimodal features, visual feature vector and subtitle feature vector, can be obtained respectively through a feature vector extraction model. In some embodiments, an initial visual feature vector and an initial subtitle feature vector can be obtained first through a feature vector extraction model, and then the visual feature vector and subtitle feature vector can be obtained through a neural network model.

[0080] In some embodiments, the feature vector extraction model described above can be a visual feature extractor and a text feature extractor. In some embodiments, the neural network model described above can be a transformer neural network based on an attention mechanism. In some embodiments, in addition to connecting each of the two feature vector extraction models to a transformer neural network based on an attention mechanism, a transformer neural network based on joint attention is also connected after the two transformer neural networks. This is to capture the correlation between multimodal data and achieve multimodal joint representation.

[0081] In some embodiments, after obtaining the initial query feature vector through the feature vector extraction model, the obtained initial visual feature vector and initial subtitle feature vector can be projected into a special space of a preset dimension through a feedforward neural network, so that the final query feature vector is also of the preset dimension.

[0082] In some embodiments, after obtaining the visual feature vector and the subtitle feature vector, the video feature vector can be obtained through feature fusion. This step is for multimodal feature alignment to ensure that multimodal features can be unified into the same semantic space, thereby achieving comprehensive consideration of multimodal features.

[0083] In some embodiments, the video feature vectors and subtitle feature vectors described above can be encoded into fixed-length vector representations. and In some embodiments, , , Represents the subtitle feature vector. Represents visual feature vectors. This represents the spatial dimension of the subtitle feature vector. Represents the spatial dimension of visual feature vectors. This represents the spatial dimension of the video feature vector. Vector representations have better expressive power and discriminability than the original visual features, and can better reflect the semantic information in the video, thus providing better semantic features for subsequent query-aware cross-modal fusion.

[0084] In some embodiments, obtaining a video feature vector corresponding to each of the above videos includes: obtaining a visual feature vector and a subtitle feature vector based on the above videos; and obtaining a visual feature weight based on the visual feature vector and the query feature vector using the following formula. ;in, Represents the weights of visual features. This represents the first weight matrix. The second weight matrix is ​​represented; the first and second weight matrices are obtained based on pre-training or manual preset; the subtitle feature weights are obtained using the following formula based on the visual feature vector and the query feature vector. ;in, Indicates the subtitle feature weights. This represents the first weight matrix. Let represent the second weight matrix; based on the above visual feature weights and the above subtitle feature weights, the video feature vector is calculated using the following formula; ;in, Represents the video feature vector. Represents the subtitle feature vector. This represents a visual feature vector.

[0085] In some embodiments, obtaining the visual feature vector and the subtitle feature vector based on the video includes: obtaining an initial visual feature vector and an initial subtitle feature vector respectively using a trained first feature vector extraction model and a second feature vector extraction model based on the video; and obtaining the visual feature vector and the subtitle feature vector based on the initial visual feature vector and the initial subtitle feature vector using a trained first neural network based on an attention mechanism.

[0086] In some embodiments, the first weight matrix and the second weight matrix are obtained through training or preset manually. Since the softmax layer can normalize the input vector into a probability distribution, in some embodiments, a fully connected network with a softmax layer can be constructed to obtain the query-aware visual feature weights and caption feature weights, where the first weight matrix and the second weight matrix are the weight matrices of the fully connected layer. The softmax layer is chosen because it can normalize the input vector into a probability distribution.

[0087] In some embodiments, after obtaining the video feature vector, the method further includes: obtaining an optimized query feature vector and an optimized video feature vector by using a trained intramodal representation optimization model and a trained intermodal semantic fusion model based on the query feature vector and the video feature vector; and using the optimized query feature vector as the query feature vector and the optimized video feature vector as the video feature vector.

[0088] In some embodiments, the intra-modal representation optimization model optimizes intra-modal representation by maximizing the mutual information between the start and end times of the target video segment and the target segment. In some embodiments, the inter-modal semantic fusion model achieves inter-modal semantic fusion by reducing the distance between positive examples and increasing the distance between negative examples.

[0089] Step S104: Based on the query feature vector and the video feature vector, obtain the similarity between each video and the query statement.

[0090] In some embodiments, the similarity of each video can be calculated using the following formula based on the query feature vector and the video feature vector described above; ;in, Indicates the similarity between videos. This represents the video feature vector of the video. This represents the query feature vector of the video.

[0091] Step S105: Determine the video with the highest similarity as the target video.

[0092] Based on the similarity value obtained for each of the above videos in step S104, the video with the largest similarity value is selected as the target video.

[0093] In some embodiments, the target video segment corresponding to the query statement may be further determined, and the target video segment is extracted from the target video.

[0094] By identifying the target video segment, fine-grained retrieval and positioning can be achieved, further improving the efficiency and accuracy of video retrieval.

[0095] In some embodiments, after determining the target video, the method further includes: determining the target video feature vector corresponding to the target video; and calculating the prediction score at the start time using the following formula based on the target video feature vector and the query feature vector. and the predicted score at the end time ; ; ; ;in, Represents the feature vector of the target video. This represents the query feature vector. This represents a one-dimensional convolution calculation function; based on the prediction scores at the start and end times mentioned above, the confidence score of the predicted video segment is calculated using the following formula; ; Among them, the above This represents the predicted start time probability and predicted end time probability determined based on the predicted start time score and the predicted end time score, respectively. Indicates the start time of the prediction. Indicates the end time of the prediction. This represents a function that calculates the maximum value of a function. Indicates A function of the independent variable. Indicates A function of the independent variable. The confidence score of the predicted segment is indicated; in response to determining that the confidence score is greater than or equal to a preset threshold, the predicted segment is determined to be the target segment.

[0096] It should be noted that the method in this embodiment can be executed by a single device, such as a computer or server. The method can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this embodiment, and the multiple devices will interact with each other to complete the method described.

[0097] It should be noted that the above description describes some embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0098] Based on the same inventive concept, and corresponding to any of the above embodiments, this application also provides a video text cross-modal retrieval device based on dual contrast learning.

[0099] refer to Figure 2 The video text cross-modal retrieval device based on dual contrast learning includes:

[0100] Module 11 is configured to retrieve a query statement and at least one video.

[0101] The first extraction module 12 is configured to obtain the query feature vector corresponding to the query statement based on the above query statement;

[0102] The second extraction module 13 is configured to obtain a video feature vector corresponding to each of the above videos; the video feature vector is obtained by jointly representing the visual feature vector and the subtitle feature vector of the corresponding video.

[0103] The calculation module 14 is configured to obtain the similarity between each of the above videos and the above query statement based on the above query feature vector and the above video feature vector;

[0104] The target video determination module 15 is configured to determine the video with the highest similarity as the target video.

[0105] In some embodiments, the apparatus further includes a target video segment determination module 16, configured to determine a target video feature vector corresponding to the target video; and to calculate the prediction score at the start time using the following formula based on the target video feature vector and the query feature vector. and the predicted score at the end time ; ; ;

[0106] ;in, Represents the feature vector of the target video. This represents the query feature vector. This represents a one-dimensional convolution calculation function; based on the prediction scores at the start and end times mentioned above, the confidence score of the predicted video segment is calculated using the following formula; ; Among them, the above This represents the predicted start time probability and predicted end time probability determined based on the predicted start time score and the predicted end time score, respectively. Indicates the start time of the prediction. Indicates the end time of the prediction. This represents a function that calculates the maximum value of a function. Indicates A function of the independent variable. Indicates A function of the independent variable. The confidence score of the predicted segment is indicated; in response to determining that the confidence score is greater than or equal to a preset threshold, the predicted segment is determined to be the target segment.

[0107] In some embodiments, the apparatus further includes an optimization module 17, configured to obtain an optimized query feature vector and an optimized video feature vector based on the query feature vector and the video feature vector, using a trained intramodal representation optimization model and a trained intermodal semantic fusion model; and to use the optimized query feature vector as the query feature vector and the optimized video feature vector as the video feature vector.

[0108] In some embodiments, the calculation module 14 is specifically configured to calculate the similarity of each video based on the query feature vector and the video feature vector using the following formula; ;in, Indicates the similarity between videos. This represents the video feature vector of the video. This represents the query feature vector of the video.

[0109] In some embodiments, the second extraction module 13 is specifically configured to obtain a visual feature vector and a subtitle feature vector based on the video; and to obtain the visual feature weights based on the visual feature vector and the query feature vector using the following formula. ;in, Represents the weights of visual features. This represents the first weight matrix. The second weight matrix is ​​represented; the first and second weight matrices are obtained based on pre-training or manual preset; the subtitle feature weights are obtained using the following formula based on the visual feature vector and the query feature vector. ;in, Indicates the subtitle feature weights. This represents the first weight matrix. Let represent the second weight matrix; based on the above visual feature weights and the above subtitle feature weights, the video feature vector is calculated using the following formula; ;in, Represents the video feature vector. Represents the subtitle feature vector. This represents a visual feature vector.

[0110] In some embodiments, the second extraction module 13 is specifically configured to obtain an initial visual feature vector and an initial subtitle feature vector respectively from the video using a trained first feature vector extraction model and a second feature vector extraction model; and to obtain the visual feature vector and the subtitle feature vector from the video using a trained first neural network based on an attention mechanism based on the initial visual feature vector and the initial subtitle feature vector.

[0111] In some embodiments, the first extraction module 12 is specifically configured to obtain an initial query feature vector based on the query statement using a trained third feature vector extraction model; and to obtain the query feature vector based on the initial query feature vector using a trained second neural network based on an attention mechanism.

[0112] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware.

[0113] The apparatus described above is used to implement the corresponding video text cross-modal retrieval method based on dual contrast learning in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0114] In some embodiments, the above-described video text cross-modal retrieval method based on dual contrast learning can be applied to, for example... Figure 3 The aforementioned video text cross-modal retrieval system based on dual contrast learning.

[0115] Specifically, the aforementioned video text cross-modal retrieval system based on dual contrast learning includes: a query encoding unit, a video encoding unit, a fusion unit, an optimization unit, a video retrieval unit, and a time-location unit.

[0116] The query encoding unit obtains the query feature vector based on the query statement through a trained text feature extractor, a feedforward neural network, and a transformer neural network based on a multi-layer attention mechanism.

[0117] The video encoding unit obtains visual feature vectors and subtitle feature vectors based on the video with subtitles through a trained text feature extractor, a trained visual feature extractor, a feedforward neural network, a transformer neural network based on a multi-layer attention mechanism, and a transformer neural network based on a joint attention mechanism.

[0118] Based on the aforementioned visual feature vectors and subtitle feature vectors, the fusion unit obtains a video feature vector representing multiple modalities through a fully connected neural network with a softmax layer and an adaptive weighted fusion algorithm.

[0119] Based on the query feature vector and video feature vector mentioned above, the optimization unit obtains the optimized query feature vector and optimized video feature vector through the trained intramodal representation optimization model and the trained intermodal semantic fusion model.

[0120] Based on the optimized query feature vector and the optimized video feature vector, the video retrieval unit uses a cosine similarity algorithm to determine the cosine similarity between the optimized video feature vector corresponding to each video and the optimized query feature vector corresponding to the query statement, and selects the video with the maximum cosine similarity as the target video.

[0121] The time-based localization unit determines the predicted video segments with confidence scores higher than a preset threshold as target video segments based on the target video feature vector and the query feature vector corresponding to the target video.

[0122] The functional implementation of each unit in the above video retrieval system is the same as steps S101 to S105 of the video text cross-modal retrieval method based on dual contrast learning described above, and will not be repeated here.

[0123] In some embodiments, to ensure that the video retrieval system can achieve high-precision video retrieval, it can be determined whether the video retrieval system meets the requirements by calculating the loss function value, and the training of the video retrieval system is determined to be completed when the loss function value is less than a preset threshold.

[0124] First, obtain multiple training queries and training videos. The training videos include positive and negative examples, where positive examples represent training videos that match the training queries, and negative examples represent training videos that do not match the training queries.

[0125] In implementing this application, the applicant discovered that by utilizing a dual contrastive learning mechanism, intramodal and intermodal contrastive learning techniques can be combined to optimize intramodal representations and fuse semantics between modalities, thereby better maintaining semantic similarity between modalities and enhancing the semantic alignment and fusion of data from different modalities, thus improving the discriminativeness and semantic consistency of data representations from different modalities.

[0126] Intramodal representation optimization is achieved by maximizing the mutual information between the start / end time and the target segment. Therefore, the following intramodal contrastive learning loss function is designed:

[0127] ;

[0128] ;

[0129] ;

[0130] ;

[0131] in, and This represents the mutual information between the start and end times of the target segment in the positive sample and other negative samples. This represents the mutual information between positive and negative samples. This represents a positive sample. Indicates a negative sample. and The video representation indicating the start and end boundaries of the target video segment in the positive sample; Discriminator It is the softplus activation function.

[0132] For inter-modal representation optimization, the corresponding inter-modal contrastive learning loss function includes the video-level inter-modal contrastive learning loss function. Frame-level modality contrastive learning loss function .

[0133] The video-level contrastive learning loss function is calculated using the following formula:

[0134] ;

[0135] ;

[0136] ;

[0137] in, , This represents a positive sample pair consisting of semantically related videos and queries. , This represents a negative sample pair consisting of unrelated videos and query videos. Representing vectors and mutual information, and This represents a parameter mapping used to project video and query features into the same semantic feature space.

[0138] The frame-level contrastive learning loss function is calculated using the following formula.

[0139] ;

[0140] .

[0141] The applicant designed loss functions for both the video retrieval unit and the time-based positioning unit.

[0142] For the video retrieval unit, the hinge loss function is used, and its value is as follows:

[0143] ;

[0144] in, This represents the value of the video retrieval loss function. This represents the similarity between the video feature vector of a positive example and the query feature vector. This represents the similarity between the video feature vector of any negative sample and the query feature vector. This represents the similarity between the video feature vector of any other negative sample and the query feature vector.

[0145] For the time-location unit, the loss function value is calculated using the following formula:

[0146] ;

[0147] in, yes The parameters representing probabilities obtained after standardization using the softmax function. and yes and One-hot encoding, and This indicates the start and end times of the target segment.

[0148] The calculation formula is as follows:

[0149] ;

[0150] ;

[0151] in, Represents the feature vector of the target video. This represents the query feature vector. This represents a one-dimensional convolution calculation function; based on the prediction scores at the start and end times mentioned above, the confidence score of the predicted video segment is calculated using the following formula;

[0152] Therefore, the loss function of the above video retrieval system can be calculated using the following formula:

[0153] ;

[0154] in, , , , and Hyperparameters are preset by humans.

[0155] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the video text cross-modal retrieval method based on dual contrast learning described in any of the above embodiments.

[0156] Figure 4 This embodiment illustrates a more specific hardware structure of an electronic device. The device may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0157] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0158] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0159] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0160] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0161] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0162] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0163] The electronic devices described above are used to implement the corresponding video text cross-modal retrieval methods based on dual contrast learning in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0164] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the video text cross-modal retrieval method based on dual contrast learning as described in any of the above embodiments.

[0165] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0166] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the video text cross-modal retrieval method based on dual contrast learning as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0167] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in the details for the sake of brevity.

[0168] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0169] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0170] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. A video-text cross-modal retrieval method based on dual contrast learning, characterized in that, The method comprises the following steps: obtaining a query statement and at least one video; obtaining a query feature vector corresponding to the query statement according to the query statement; obtaining a video feature vector corresponding to each video according to the video, and obtaining an optimized query feature vector and an optimized video feature vector by using a trained intra-modal representation optimization model and a trained inter-modal semantic fusion model according to the query feature vector and the video feature vector, taking the optimized query feature vector as the query feature vector and the optimized video feature vector as the video feature vector, wherein the intra-modal representation optimization model optimizes intra-modal representation by maximizing mutual information between a start time and an end time of a target video segment and the target segment, and the inter-modal semantic fusion model realizes inter-modal semantic fusion by reducing the distance between positive samples and increasing the distance between negative samples; the video feature vector is obtained by jointly representing the visual feature vector and the subtitle feature vector of the corresponding video; obtaining the similarity between each video and the query statement according to the query feature vector and the video feature vector; determining the video corresponding to the maximum similarity as a target video.

2. The dual contrast learning based video-text cross-modal retrieval method according to claim 1, characterized in that, After determining the target video, the method further comprises the following steps: determining a target video feature vector corresponding to the target video; According to the target video feature vector and the query feature vector, a start time prediction score is calculated by the following formula and an end time prediction score ; ； ； wherein, denotes a target video feature vector, denotes a query feature vector, denotes a one-dimensional convolution computation function, denotes a spatial dimension of the video feature vector; calculating a confidence score of a predicted video segment according to the start time prediction score and the end time prediction score by using the following formula: ； ； wherein the denote a predicted start time probability and a predicted end time probability determined from the start time prediction score and the end time prediction score, denote a predicted start time, denote a predicted end time, denote a function maximum calculation function, denote a function with as argument, denote a function with as argument, denote a confidence score of a predicted segment; determining the predicted segment as a target segment in response to determining that the confidence score is greater than or equal to a preset threshold.

3. The dual contrast learning based video-text cross-modal retrieval method according to any one of claims 1 or 2, characterized in that, The step of obtaining the similarity between each video and the query statement according to the query feature vector and the video feature vector comprises the following steps: calculating the similarity of each video according to the query feature vector and the video feature vector by using the following formula: ； wherein, represents a similarity of a video, represents a video feature vector of the video, represents a query feature vector of the video.

4. The dual contrast learning based video-text cross-modal retrieval method according to any one of claims 1 or 2, characterized in that, The step of obtaining a video feature vector corresponding to each video according to the video comprises the following steps: obtaining a visual feature vector and a subtitle feature vector according to the video; obtaining a visual feature weight according to the visual feature vector and the query feature vector by using the following formula: ； wherein, represents a visual feature weight, represents a first weight matrix, represents a second weight matrix; the first weight matrix and the second weight matrix are obtained according to pre-training or manual presetting; obtaining a subtitle feature weight according to the visual feature vector and the query feature vector by using the following formula: ； wherein, denotes a subtitle feature weight, denotes a first weight matrix, denotes a second weight matrix; calculating a video feature vector according to the visual feature weight and the subtitle feature weight by using the following formula: ； wherein, represents a video feature vector, represents a subtitle feature vector, represents a visual feature vector.

5. The dual contrast learning based video-text cross-modal retrieval method according to claim 4, characterized in that, The step of obtaining a visual feature vector and a subtitle feature vector according to the video comprises the following steps: obtaining an initial visual feature vector and an initial subtitle feature vector by using a trained first feature vector extraction model and a trained second feature vector extraction model according to the video; obtaining the visual feature vector and the subtitle feature vector by using a trained first neural network based on an attention mechanism according to the initial visual feature vector and the initial subtitle feature vector.

6. The dual contrast learning based video-text cross-modal retrieval method according to claim 1, characterized in that, The step of obtaining a query feature vector corresponding to the query statement according to the query statement comprises the following steps: obtaining an initial query feature vector by using a trained third feature vector extraction model according to the query statement; obtaining the query feature vector by using a trained second neural network based on an attention mechanism according to the initial query feature vector.

7. A dual contrast learning based video-text cross-modal retrieval apparatus, characterized in that, The method comprises the following steps: An acquisition module is configured to acquire a query statement and at least one video; A first extraction module is configured to obtain a query feature vector corresponding to the query statement according to the query statement; A second extraction module is configured to obtain a video feature vector corresponding to each of the videos according to the videos, to obtain an optimized query feature vector and an optimized video feature vector by using a trained intra-modal representation optimization model and a trained inter-modal semantic fusion model according to the query feature vector and the video feature vector, and to use the optimized query feature vector as the query feature vector and the optimized video feature vector as the video feature vector, wherein the intra-modal representation optimization model is used to optimize intra-modal representation by maximizing mutual information between a start time and an end time of a target video segment and the target segment, and the inter-modal semantic fusion model is used to realize inter-modal semantic fusion by reducing distance between positive samples and increasing distance between negative samples, and the video feature vector is obtained by jointly representing a visual feature vector and a subtitle feature vector of a corresponding video; A calculation module is configured to obtain similarity between each of the videos and the query statement according to the query feature vector and the video feature vector; A target video determination module is configured to determine a video corresponding to maximum similarity as a target video.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the method of any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to make the computer execute the method of any one of claims 1 to 6.

Images