Video text retrieval model training method and device, equipment and storage medium

Abstract

The invention discloses a video text retrieval model training method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a training sample, the training sample comprises an original video, first text data and second text data, and the second text data is obtained through language conversion based on the first text data; performing feature extraction on the training sample to obtain an initial video feature, a first text feature and a second text feature; performing feature interaction and aggregation processing on the initial video features to obtain video features carrying context information; and training a video text retrieval model according to the first text feature, the second text feature and the video feature carrying the context information, and obtaining a trained video text retrieval model under the condition of determining that the training is completed. And the retrieval accuracy of the video text retrieval model is improved.

Description

Technical Field

[0001] The present application relates to the field of learning and video retrieval technology, and in particular to a training method, device, electronic device and storage medium for a video text retrieval model. Background Art

[0002] With the rise and popularity of various video platforms, the number of videos on the Internet has also shown a blowout growth, and video text retrieval has become a new demand for people to efficiently find relevant videos. Among them, the video text retrieval task aims to automatically mine the latent semantic information of the corresponding modality based on the given text or video query value, and then retrieve the single or multiple videos or texts with the highest similarity in the data set.

[0003] When performing video text retrieval, a feature extraction module is usually used to extract features from the video and text respectively, and then a similarity function is designed to calculate the similarity of the features of the video and text. However, in actual applications, different videos may have similar screen compositions or plots, resulting in inaccurate extraction of video features. At the same time, due to the existence of polysemous words or differences in expression habits, the extraction of text features in a single language is also inaccurate, which in turn leads to poor retrieval and matching results of the video text retrieval model. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a training method, device, electronic device and storage medium for a video text retrieval model to solve the technical problem in the related art that the video text retrieval effect is poor due to the influence of similar videos and single language texts.

[0005] In a first aspect, an embodiment of the present application provides a method for training a video text retrieval model, comprising:

[0006] Acquire a training sample, wherein the training sample includes an original video, first text data, and second text data, and the second text data is obtained by language conversion based on the first text data;

[0007] Extracting features from the training sample to obtain initial video features, first text features, and second text features;

[0008] Performing feature interaction and aggregation processing on the initial video features to obtain video features carrying context information;

[0009] The video text retrieval model is trained according to the first text feature, the second text feature and the video feature carrying the context information, and a trained video text retrieval model is obtained when it is determined that the training is completed.

[0010] In a second aspect, an embodiment of the present application provides a training device for a video text retrieval model, comprising:

[0011] A data acquisition module, used to acquire a training sample, wherein the training sample includes an original video, a first text data and a second text data, and the second text data is obtained by language conversion based on the first text data;

[0012] A feature extraction module, used to extract features from the training sample to obtain an initial video feature, a first text feature, and a second text feature;

[0013] A feature processing module, used for performing feature interaction and aggregation processing on the initial video features to obtain video features carrying context information;

[0014] The model training module is used to train the video text retrieval model according to the first text feature, the second text feature and the video feature carrying context information, and obtain the trained video text retrieval model when the training is determined to be completed.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the training method of the video text retrieval model described in any one of the above items when executing the computer program.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps in the training method of the video text retrieval model described in any one of the above items.

[0017] The embodiment of the present application provides a training method, device, electronic device and storage medium for a video text retrieval model. When training the constructed video text retrieval model, a training sample is obtained, wherein the training sample includes an original video, a first text data corresponding to the original video, and a second text data obtained by language conversion of the first text data. Then, feature extraction is performed on the training sample to obtain the corresponding initial video feature, the first text feature and the second text feature. At the same time, feature interaction and aggregation processing are performed on the initial video feature to obtain a video feature carrying context information. Finally, the video text retrieval model is trained according to the first text feature, the second text feature and the video feature to obtain a trained video text retrieval model when the training is completed. The method realizes adding context information to the video feature during the training process, mining the association between video frames, improving the accuracy of video feature extraction, and at the same time performing language conversion on the original text data, using multilingual text for semantic alignment, reducing the interference of polysemous words and expression habits, improving the accuracy of text features, and thus improving the retrieval accuracy of the video text retrieval model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flow chart of a method for training a video text retrieval model provided in an embodiment of the present application;

[0019] Figure 2 It is a flowchart of the steps of obtaining video features carrying context information provided by an embodiment of the present application;

[0020] Figure 3 It is a flowchart of the steps of training a video text retrieval model provided in an embodiment of the present application;

[0021] Figure 4 It is a schematic block diagram of reconstructing video features provided in an embodiment of the present application;

[0022] Figure 5 It is a schematic diagram of a block diagram of training a video text retrieval model provided in an embodiment of the present application;

[0023] Figure 6 It is a structural schematic diagram of a training device for a video text retrieval model provided in an embodiment of the present application;

[0024] Figure 7 is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0025] Figure 8 This is another structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0028] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0029] In the related art, when training a video text retrieval model, the video and text are feature extracted, and then a similarity function is designed to calculate the similarity of the features of the video and text. However, since video frames have similar images, the feature extraction may be inaccurate when extracting video features. At the same time, for a single language, there may be polysemous words or different expression habits, which may lead to inaccurate extraction of text features, and thus the use accuracy of the learned and optimized video text retrieval model is low.

[0030] In order to solve the technical problems existing in the related art, the present application embodiment provides a training method for a video text retrieval model, see Figure 1 , Figure 1 It is a flow chart of a method for training a video text retrieval model provided in an embodiment of the present application, and the method includes steps 101 to 104.

[0031] Step 101: Acquire a training sample, wherein the training sample includes an original video, first text data, and second text data, and the second text data is obtained by language conversion based on the first text data.

[0032] In one embodiment, before training, training samples for training are obtained, and the obtained training samples include original video, first text data and second text data, and the second text data is obtained by language conversion based on the first text data.

[0033] For example, when obtaining training samples, firstly, the corresponding video data and the text data corresponding to the video data are obtained, and in order to avoid the influence of polysemous words or expression habits in a single language, the first text data will be converted into another language. Generally speaking, when processing, the first text data is directly translated to obtain the second text data.

[0034] Step 102: extract features from the training samples to obtain initial video features, first text features, and second text features.

[0035] In one embodiment, after obtaining the training samples, they are input into the video text retrieval model for training, and then feature extraction is performed on the training samples. Specifically, feature extraction is performed on the original data, the first text data, and the second text data to obtain initial video features, first text features, and second text features.

[0036] Furthermore, when extracting features from the original video in the training sample, it includes: sampling video frames of the original video to obtain video key frames for feature extraction, and extracting features from the video key frames to obtain initial video features of the original video; extracting features from the first text data to obtain first text features, and extracting features from the second text data to obtain second text features.

[0037] Exemplarily, when extracting features from an original video, CLIP (Contrastive Language-Image Pre-training) can be used to process the original video to obtain video features. Specifically, when extracting features, firstly sample the video frames of the original video to obtain a number of video frames for feature extraction, and then use the ViT-B / 32 image encoder in CLIP to extract features from the sampled video frames, and obtain the features of each video frame to form the initial video features of the original video. For example, when sampling video frames, 12 video frames can be randomly sampled, and then feature extraction is performed on the 12 video frames.

[0038] In addition, when extracting features from the first text data and the second text data, it includes: performing word segmentation processing on the first text data to obtain a first text sequence; filling the first text sequence according to position information corresponding to the first text sequence; and encoding the filled first text sequence to obtain a first text feature of the first text data.

[0039] Specifically, when extracting features from the first text data, the first text data is segmented to obtain a first text sequence corresponding to the first text data, and then the first text sequence is filled according to the position information of each segmented word in the first text sequence. The filled first text sequence is then reversely encoded to obtain the first text feature.

[0040] Exemplarily, when processing the first text data, the sentences in the first text data can be segmented using Byte Pair Encoding (BPE), and then the first text sequence is obtained according to the result of the segmentation. For the first text data, after the segmentation, each segmentation corresponds to a position in the first text sequence, so the position information corresponding to the first text sequence can be obtained at this time, and then the position information is fused with the first text sequence, and a [EOS] vector is embedded at the end of the processed first text sequence as a terminator, and finally the encoding is performed to obtain the corresponding first text feature.

[0041] Furthermore, when processing the second text data, it includes: performing word segmentation processing on the second text data to obtain a second text sequence; filling the second text sequence according to position information corresponding to the second text sequence; and encoding the filled second text sequence to obtain a second text feature of the second text data.

[0042] Since the method of extracting features from the second text data is the same as the method of extracting features from the first text data, a detailed description is omitted. For details, reference may be made to the embodiment of extracting features from the first text data.

[0043] Step 103 , performing feature interaction and aggregation processing on the initial video features to obtain video features carrying context information.

[0044] In order to improve the accuracy of feature extraction from videos, during training, after completing the feature extraction of the original video, it is necessary to process it accordingly to obtain video features carrying contextual information, and then use the video features carrying contextual information to train the video text retrieval model.

[0045] In one embodiment, when video features carrying context information are obtained, feature interaction and aggregation processing are performed on the obtained initial video features, and then the video features carrying context information are obtained after completing the relevant processing.

[0046] When processing the initial video features to obtain video features carrying contextual information, you can refer to Figure 2 , Figure 2It is a flowchart diagram of the steps of obtaining video features carrying context information provided in an embodiment of the present application, wherein the steps include steps 201 to 203.

[0047] Step 201, performing feature interaction on each feature in the initial video feature based on a multi-head attention mechanism to obtain corresponding interaction features;

[0048] Step 202, performing size mapping processing on the interactive features to obtain interactive video features, wherein the interactive video features have the same feature size as the initial video features;

[0049] Step 203: Perform feature aggregation processing on the interactive video features and the initial video features to obtain video features carrying context information.

[0050] Specifically, during processing, the multi-head attention mechanism is first used to perform feature interaction processing on each feature in the initial video feature to obtain the corresponding interaction feature, and then the interaction feature is size mapped to obtain the interaction video feature with the same feature size as the initial video feature. Finally, the interaction video feature is aggregated with the initial video feature to obtain the video feature carrying contextual information.

[0051] Exemplarily, when extracting the contextual features of the original video, the initial video features obtained by feature extraction are input into the contextual information extraction module, and the initial video features are interacted and learned through the multi-head attention mechanism algorithm, and then the interacted initial video features are aggregated with the initial video features to obtain video features carrying contextual information.

[0052] The obtained initial video features include the features of each video frame in the sampled video frames. During the interactive processing, the correlation between one of the features and the other remaining features is calculated, and then each feature in the initial video features is updated according to the correlation.

[0053] Among them, when performing interactive processing, it includes: selecting the first initial video feature from the initial video features, and calculating the interactive similarity matrix of the first initial video feature and the second initial video feature based on the multi-head attention mechanism, wherein the initial video feature is composed of the first initial video feature and the second initial video feature, and the number of the first initial video feature is one; aggregating the similarity matrix with each video feature in the second initial video feature, and summarizing the obtained aggregation results to obtain an interactive feature corresponding to the first initial video feature; when each feature in the initial video features is processed, the interactive feature corresponding to the initial video feature is obtained.

[0054] That is, when processing to obtain interactive features, each feature in the initial video features is updated, and when updating, the obtained interactive features are used as updated features. Taking one of the features as an example, the first initial video feature is selected from the initial video features to calculate the similarity matrix between the first initial video feature and the second initial video feature based on the multi-head attention mechanism. The second initial video feature is a collection of all features in the initial video features except the first initial video feature. After obtaining the similarity matrix, the similarity matrix is ​​aggregated with each video feature in the second initial video feature, and each aggregation process obtains a feature. Finally, all features are summarized to obtain the interactive feature corresponding to the first initial video feature. After completing the interactive processing of each feature in the initial video feature, the interactive feature corresponding to the initial video feature is obtained.

[0055] After obtaining the interactive features, in order to ensure the consistency of feature size, the interactive features are resized to obtain interactive video features. Finally, the interactive video features are aggregated with the initial video features again, and the video features obtained after processing will carry contextual information.

[0056] In the actual processing process, when extracting and processing the context information of the original video to obtain the video features carrying the context information, one of the initial video features can be first converted into a query value Q v , key value K v and real value V v , and the specific conversion method is as follows:

[0057]

[0058] K v =LayerNorm(F v )W2,

[0059] V v =LayerNorm(F v )W3,

[0060] Among them, F v is the initial video feature, F v · are the initial video features after transposition, and W1, W2, and W3 represent weight matrices respectively.

[0061] At this time, the multi-head self-attention mechanism is used to calculate the correlation weight (similarity matrix) between each feature and other features, and then the correlation weight is used to aggregate the features to obtain the features used for replacement.

[0062] For the multi-head attention mechanism used, the number of parallel layers can be set to H, where 1 < h < H. At this time, the calculation formula for the output features of the h-th layer is as follows:

[0063]

[0064] When obtaining each processed feature, the outputs of the same feature in all parallel layers are concatenated, such as connecting the head and the tail, to obtain the concatenated feature:

[0065] attn_out = MultiHead(Q v , K v , V v ) = concat(head1, …, head1)W o ;

[0066] where W0 represents the weight matrix.

[0067] Next, the concatenated feature is mapped back to the original size to obtain the interactive video feature after interaction:

[0068] Z v = LayerNorm(FC(attn_out) + LayerNorm(attn_out)),

[0069] where FC represents a fully connected layer.

[0070] Finally, when obtaining the video feature carrying context information, the obtained interactive video feature is aggregated with the corresponding initial video feature to obtain the video frame feature carrying context information:

[0071] C v = Conv(LayerNorm(F v + Z v );

[0072] where Conv represents a convolutional layer with a kernel of 1×1.

[0073] Step 104: Train the video-text retrieval model according to the first text feature, the second text feature, and the video feature carrying context information, and obtain the trained video-text retrieval model when it is determined that the training is completed.

[0074] In one embodiment, after feature extraction and feature processing are completed to obtain the first text feature, the second text feature, and the video feature carrying context information, the constructed video text retrieval model will be trained based on the first text feature, the second text feature, and the video feature. Specifically, the first text feature, the second text feature, and the video feature are input into the video text retrieval model, and whether the training is completed is determined based on the loss value generated during the training, and then a trained video text retrieval model is obtained when the training is completed.

[0075] Specifically, when training the video text retrieval model, you can refer to Figure 3 , Figure 3 It is a flowchart of the steps for training a video text retrieval model provided in an embodiment of the present application, wherein the steps include steps 301 to 303.

[0076] Step 301, reconstructing the video feature according to the first text feature to obtain a first reconstructed video feature, and reconstructing the video feature according to the second text feature to obtain a second reconstructed video feature;

[0077] Step 302: training a video text retrieval model according to the first text feature and the first reconstructed video feature to obtain a first loss value, and training the video text retrieval model according to the second text feature and the second reconstructed video feature to obtain a second loss value;

[0078] Step 303, when it is determined that the training is completed according to the first loss value and the second loss value, a trained video text retrieval model is obtained.

[0079] In one embodiment, when training is performed based on a first text feature, a second text feature, and a video feature carrying context information, the video feature is first reconstructed to obtain a first reconstructed video feature associated with the first text feature, and a second reconstructed video feature associated with the second text feature. The first text feature and the first reconstructed video feature are then used as a group to train a video text retrieval model, and the second text feature and the second reconstructed video feature are used as a group to train the video text retrieval model. In each training stage, corresponding training losses are generated, specifically including a first loss value and a second loss value. When the training is determined to be completed based on the first loss value and the second loss value, a trained video text retrieval model is obtained.

[0080] In the actual training process, the video features need to be reconstructed to achieve alignment between the video features and the text features. When reconstructing the video, the video features need to be reconstructed for the first text features and the second text features respectively. Specifically, the processing includes: calculating the similarity weight between each feature in the video features and the first text features; updating each feature in the video features according to the similarity weight, and performing feature mapping on each updated feature to obtain the mapped video features; performing feature aggregation on the mapped video features to obtain the first reconstructed video features associated with the first text data.

[0081] That is, by inputting the video features carrying contextual information and the text features into the cross-modal interaction module together, the self-attention mechanism algorithm is used to calculate the correlation weights between the text features and the video features, and then the correlation weights are used to perform feature aggregation to obtain reconstructed video features related to the text.

[0082] For the module used to reconstruct video features, please refer to Figure 4 , based on Figure 4 When the block diagram shown in the figure is interactively processed to complete the reconstruction of the video features, the first text feature and the second text feature are converted into the corresponding query value Q, and the video feature carrying the context information is converted into the key value K and the real value V. Then the self-attention algorithm is used to calculate the similarity between the first text feature and each feature in the video feature carrying the context information. Then, according to the similarity, the video features carrying the context information are aggregated to obtain the video features related to the first text data, that is, the reconstruction of the video features based on the first text data is completed. At the same time, the reconstruction of the video features based on the second text data will also be completed. Based on the method of obtaining the first reconstructed video features, the second reconstructed video features can be obtained to complete the modal alignment.

[0083] For example, taking the first reconstructed video feature associated with the first text feature as an example, when processing, the first text feature is first converted into a query value q t , converting the video features carrying contextual information into key value k v and real value v v , the self-attention algorithm is used to calculate the similarity for feature aggregation, and finally the aggregated video features are mapped to the same dimension as the video features carrying contextual information through the linear layer processing:

[0084] r v|t =LayerNorm(Attention(q t ,k v ,v v )W t );

[0085] Among them, W t is the adaptive weight matrix.

[0086] Finally, when performing the aggregation process, the first reconstructed video feature associated with the first text feature is obtained as follows:

[0087]

[0088] Similarly, when reconstructing the video feature based on the second text feature, the second reconstructed video feature associated with the second text feature can be obtained in the same manner as the first reconstructed video feature is sampled, so a detailed description is not given here.

[0089] Furthermore, after completing the reconstruction of the video features, the video text retrieval model will be trained and optimized based on the reconstructed video features, the first text features and the second text features. During the training, the training can be divided into two stages. The first stage is training based on the first text features and the first reconstructed video features, and the second stage is training based on the second text features and the second reconstructed video features. At the same time, each training stage will generate a corresponding loss value.

[0090] In one embodiment, the process of obtaining the loss value of each stage includes: inputting the first text feature and the first reconstructed video feature into the video text retrieval model, calculating the first similarity matrix between the first text feature and the first reconstructed video feature, and calculating the first loss value of the text retrieval video trained based on the training sample and the first loss of the video retrieval text based on the first similarity matrix; inputting the second text feature and the second reconstructed video feature into the video text retrieval model, calculating the second similarity matrix between the second text feature and the second reconstructed video feature, and calculating the second loss value of the text retrieval video trained based on the training sample and the second loss of the video retrieval text based on the second similarity matrix.

[0091] From the above description, it can be seen that the training in the first stage is the same as that in the second stage. The difference lies in the data used for training. When obtaining the loss value during training at each stage, the similarity matrix of the text features and the reconstructed video features is calculated, and then the loss values ​​generated during training at different stages are calculated based on the similarity matrix.

[0092] Taking the first loss value of the first stage as an example, when training the video text retrieval model, the cosine similarity between the first reconstructed video feature related to the first text feature and the first text feature is calculated to obtain the first similarity matrix:

[0093]

[0094] in, is the first reconstructed video feature, is the first reconstructed video feature after transposition, c t is the first text feature.

[0095] Then, when calculating the loss value at this stage, the loss of the text retrieval video task and the loss of the video retrieval text task are calculated, and then the losses are summed to obtain the first loss value of the training at this stage. Specifically, when the number of training samples is B and they belong to batch Ω, the cross entropy loss function is calculated based on the first similarity matrix obtained, and the loss L of this retrieval video task can be obtained. t2v and the loss L for the video retrieval text task v2t as follows:

[0096]

[0097]

[0098] The first loss value at this time is: L1=L t2v +L v2t .

[0099] Similarly, when obtaining the second loss value L2, it will also be obtained in the same manner as the first loss value.

[0100] Furthermore, when the trained video text retrieval model is finally obtained, the first loss value and the second loss value are summed to obtain a total loss value, and then whether the training is completed is determined based on the total loss value, specifically including: adding the first loss value and the second loss value to obtain the total loss value during the training of the video text retrieval model; when the total loss value is less than a preset threshold, it is determined that the training is completed and a trained video text retrieval model is obtained.

[0101] That is, after the total loss value is obtained by summing up, the total loss value is compared with a preset threshold to determine whether the training is completed, wherein the training is determined to be completed when the total loss value is less than the preset threshold, and a trained video text retrieval model is obtained.

[0102] Further, refer to Figure 5 , Figure 5 It is a block diagram of the training of the video text retrieval model provided in the embodiment of the present application.

[0103] like Figure 5As shown, when the constructed video text retrieval model is trained, the input video data and the corresponding first text data are determined, and the first text data can be translated by using the language translation function to obtain the second text data, and then the features of the first text data, the second text data and the video data are extracted respectively to obtain the first text features, the second text features and the video features carrying the context information, and then the video features are reconstructed in two parallel processes to obtain the video features associated with the first text data / the second text data, and then semantic interaction processing is performed, and the similarity matrix at different stages is calculated, and finally the loss values ​​at different stages are determined according to the similarity matrix to obtain the total loss value in the training process to determine whether the training is completed.

[0104] Furthermore, after completing the training of the constructed video text retrieval model, when in use, the video data that needs to be queried and matched is input into the trained video text retrieval model, and the final matching process is completed by calculating the similarity between the global features of the video and the text features.

[0105] Specifically, when the video text retrieval model performs video retrieval, it includes: receiving an input video to be retrieved; performing frame sampling on the video to be retrieved to obtain a target video frame; performing feature extraction on the target video frame to obtain a target video feature of the video to be retrieved; inputting the target video feature into a trained video text retrieval model to obtain a similarity value between the target video feature and each text feature in a database, and outputting the target text data corresponding to the maximum similarity value as data matching the video to be retrieved, wherein the text data corresponds to the text feature one-to-one.

[0106] Exemplarily, when a video to be retrieved is received, frame sampling is performed on the video to be retrieved to obtain a target video frame for feature extraction, and then video features are extracted from the target video frame to obtain target video features, which are then input into a trained video text retrieval model. Through model processing, the similarity value between the target video feature and the text feature of each text data can be calculated, and the text data corresponding to the maximum similarity value is the target text that matches the video to be retrieved.

[0107] In summary, the present application discloses a training method for a video text retrieval model. When training the constructed video text retrieval model, a training sample is obtained, wherein the training sample includes an original video, a first text data corresponding to the original video, and a second text data obtained by language conversion of the first text data. Then, feature extraction is performed on the training sample to obtain the corresponding initial video feature, the first text feature, and the second text feature. At the same time, feature interaction and aggregation processing are performed on the initial video feature to obtain a video feature carrying context information. Finally, the video text retrieval model is trained according to the first text feature, the second text feature, and the video feature to obtain a trained video text retrieval model when the training is completed. The method realizes adding context information to the video features during the training process, mining the association between video frames, and improving the accuracy of video feature extraction. At the same time, the original text data is converted into a language, and semantic alignment is performed using multilingual texts to reduce the interference of polysemous words and expression habits, improve the accuracy of text features, and thus improve the retrieval accuracy of the video text retrieval model.

[0108] According to the method described in the above embodiment, this embodiment will be further described from the perspective of a training device for a video text retrieval model. The training device for the video text retrieval model can be implemented as an independent entity or integrated in an electronic device, such as a terminal. The terminal may include a mobile phone, a tablet computer, etc.

[0109] See also Figure 6 , Figure 6 is a structural diagram of a training device for a video text retrieval model provided in an embodiment of the present application, such as Figure 6 As shown, the video text retrieval model training device 600 provided in the embodiment of the present application includes:

[0110] The data acquisition module 601 is used to acquire a training sample, wherein the training sample includes an original video, first text data and second text data, and the second text data is obtained by language conversion based on the first text data;

[0111] A feature extraction module 602 is used to extract features from the training sample to obtain an initial video feature, a first text feature, and a second text feature;

[0112] A feature processing module 603 is used to perform feature interaction and aggregation processing on the initial video features to obtain video features carrying context information;

[0113] The model training module 604 is used to train the video text retrieval model according to the first text feature, the second text feature and the video feature carrying context information, and obtain the trained video text retrieval model when the training is completed.

[0114] In one embodiment, the feature extraction module 602 is further used to:

[0115] Sampling video frames of the original video to obtain video key frames for feature extraction, and extracting features from the video key frames to obtain initial video features of the original video;

[0116] Feature extraction is performed on the first text data to obtain a first text feature, and feature extraction is performed on the second text data to obtain a second text feature.

[0117] In one embodiment, the feature extraction module 602 is further used to:

[0118] Performing word segmentation processing on the first text data to obtain a first text sequence;

[0119] Filling the first text sequence according to the position information corresponding to the first text sequence;

[0120] Performing encoding processing on the padded first text sequence to obtain a first text feature of the first text data;

[0121] as well as,

[0122] Performing word segmentation processing on the second text data to obtain a second text sequence;

[0123] Filling the second text sequence according to the position information corresponding to the second text sequence;

[0124] The padded second text sequence is encoded to obtain a second text feature of the second text data.

[0125] In one embodiment, the feature processing module 603 is further used to:

[0126] Perform feature interaction on each feature in the initial video feature based on the multi-head attention mechanism to obtain the corresponding interactive features;

[0127] Performing size mapping processing on the interactive features to obtain interactive video features, wherein the interactive video features have the same feature size as the initial video features;

[0128] The interactive video features are aggregated with the initial video features to obtain video features carrying contextual information.

[0129] In one embodiment, the feature processing module 603 is further used to:

[0130] Selecting a first initial video feature from the initial video features, and calculating an interactive similarity matrix between the first initial video feature and the second initial video feature based on a multi-head attention mechanism, wherein the initial video feature consists of the first initial video feature and the second initial video feature, and the number of the first initial video feature is one;

[0131] Aggregating the similarity matrix with each video feature in the second initial video feature, and summarizing all the aggregation results to obtain an interactive feature corresponding to the first initial video feature;

[0132] After processing each feature in the initial video feature, the interactive feature corresponding to the initial video feature is obtained.

[0133] In one embodiment, the model training module 604 is further used to:

[0134] Reconstructing the video feature according to the first text feature to obtain a first reconstructed video feature, and reconstructing the video feature according to the second text feature to obtain a second reconstructed video feature;

[0135] The video text retrieval model is trained according to the first text feature and the first reconstructed video feature to obtain a first loss value, and the video text retrieval model is trained according to the second text feature and the second reconstructed video feature to obtain a second loss value;

[0136] When the training is completed according to the first loss value and the second loss value, a trained video text retrieval model is obtained.

[0137] In one embodiment, the model training module 604 is further used to:

[0138] Calculate the similarity weight between each feature in the video feature and the first text feature;

[0139] The features in the video features are updated according to the similarity weights, and the updated features are feature mapped to obtain the mapped video features;

[0140] Feature aggregation is performed on the mapped video features to obtain first reconstructed video features associated with the first text data.

[0141] In one embodiment, the model training module 604 is further used to:

[0142] Inputting the first text feature and the first reconstructed video feature into the video text retrieval model, calculating a first similarity matrix between the first text feature and the first reconstructed video feature, and calculating a first loss value of the text retrieval video trained based on the training sample and a first loss of the video retrieval text according to the first similarity matrix;

[0143] The second text feature and the second reconstructed video feature are input into the video text retrieval model, and a second similarity matrix between the second text feature and the second reconstructed video feature is calculated. Based on the second similarity matrix, a second loss value of the text retrieval video trained based on the training sample and a second loss of the video retrieval text are calculated.

[0144] In one embodiment, the model training module 604 is further used to:

[0145] Add the first loss value and the second loss value to obtain the total loss value during the video text retrieval model training;

[0146] When the total loss value is less than a preset threshold, the training is determined to be completed and a trained video text retrieval model is obtained.

[0147] In one embodiment, the video text retrieval model training device 800 further includes a retrieval matching module, which is used to:

[0148] Receive input video to be retrieved;

[0149] Perform frame sampling on the video to be retrieved to obtain the target video frame;

[0150] Extract features of the target video frame to obtain target video features of the video to be retrieved;

[0151] The target video features are input into the trained video text retrieval model to obtain the similarity values ​​between the target video features and the text features in the database, and the target text data corresponding to the maximum similarity value is output as the data matching the video to be retrieved, where the text data corresponds to the text features one by one.

[0152] Also, see Figure 7 , Figure 7 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, and the electronic device may be a mobile terminal such as a smart phone, a tablet computer, or the like. Figure 7 As shown, the electronic device 700 includes a processor 701 and a memory 702. The processor 701 is electrically connected to the memory 702.

[0153] The processor 701 is the control center of the electronic device 700. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device 700 and processes data by running or loading applications stored in the memory 702 and calling data stored in the memory 702, thereby monitoring the electronic device 700 as a whole.

[0154] In this embodiment, the processor 701 in the electronic device 700 will load the instructions corresponding to the processes of one or more applications into the memory 702 according to the following steps, and the processor 701 will run the application stored in the memory 702, thereby implementing any step in the training method of the video text retrieval model provided in the above embodiment.

[0155] The electronic device 700 can implement the steps in any embodiment of the training method for the video text retrieval model provided in the embodiments of the present application. Therefore, it can achieve the beneficial effects that can be achieved by any video text retrieval model training method provided in the embodiments of the present application. Please refer to the previous embodiments for details and will not be repeated here.

[0156] See also Figure 8 , Figure 8 is another structural schematic diagram of an electronic device provided in an embodiment of the present application, such as Figure 8 As shown, Figure 8 The specific structural block diagram of the electronic device provided in the embodiment of the present application is shown, and the electronic device can be used to implement the training method of the video text retrieval model provided in the above embodiment. The electronic device 800 can be a mobile terminal such as a smart phone or a laptop computer.

[0157] The RF circuit 810 is used to receive and send electromagnetic waves, realize the mutual conversion between electromagnetic waves and electrical signals, and thus communicate with a communication network or other devices. The RF circuit 810 may include various existing circuit elements for performing these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, user identity module (SIM) cards, memories, etc. The RF circuit 810 can communicate with various networks such as the Internet, corporate intranets, wireless networks, or communicate with other devices through wireless networks. The above-mentioned wireless networks may include cellular telephone networks, wireless local area networks, or metropolitan area networks. The above-mentioned wireless network can use various communication standards, protocols and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSMEnvironment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as Institute of Electrical and Electronics Engineers standards IEEE802.11a, IEEE 802.11b, IEEE802.11g and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging and short messages, and any other suitable communication protocols, even those that have not yet been developed.

[0158] The memory 820 can be used to store software programs and modules, such as the program instructions / modules corresponding to the training method of the video text retrieval model in the above-mentioned embodiment. The processor 880 executes various functional applications and the training method of the video text retrieval model by running the software programs and modules stored in the memory 820.

[0159] The memory 820 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 820 may further include a memory remotely arranged relative to the processor 880, and these remote memories may be connected to the electronic device 800 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0160] The input unit 830 can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. Specifically, the input unit 830 may include a touch-sensitive surface 831 and other input devices 832. The touch-sensitive surface 831, also known as a touch display screen or touchpad, can collect user touch operations on or near it (such as operations performed by users using fingers, styluses, or any other suitable objects or accessories on or near the touch-sensitive surface 831), and drive corresponding connection devices according to a pre-set program. Optionally, the touch-sensitive surface 831 may include a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into touch point coordinates, and then sends it to the processor 880, and can receive and execute commands sent by the processor 880. In addition, the touch-sensitive surface 831 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch-sensitive surface 831, the input unit 830 may further include other input devices 832. Specifically, the other input devices 832 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0161] The display unit 840 can be used to display information input by the user or information provided to the user and various graphical user interfaces of the electronic device 800, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit 840 may include a display panel 841, and optionally, the display panel 841 may be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc. Further, the touch-sensitive surface 831 may cover the display panel 841, and when the touch-sensitive surface 831 detects a touch operation on or near it, it is transmitted to the processor 880 to determine the type of the touch event, and then the processor 880 provides corresponding visual output on the display panel 841 according to the type of the touch event. Although in the figure, the touch-sensitive surface 831 and the display panel 841 are implemented as two independent components to implement input and output functions, in some embodiments, the touch-sensitive surface 831 can be integrated with the display panel 841 to implement input and output functions.

[0162] The electronic device 800 may also include at least one sensor 850, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 841 according to the brightness of the ambient light, and the proximity sensor may generate an interrupt when the flip cover is closed or closed. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc. that can also be configured in the electronic device 800, they will not be repeated here.

[0163] The audio circuit 860, the speaker 861, and the microphone 862 can provide an audio interface between the user and the electronic device 800. The audio circuit 860 can transmit the electrical signal converted from the received audio data to the speaker 861, which is converted into a sound signal for output; on the other hand, the microphone 862 converts the collected sound signal into an electrical signal, which is received by the audio circuit 860 and converted into audio data, and then the audio data is output to the processor 880 for processing, and then sent to another terminal through the RF circuit 810, or the audio data is output to the memory 820 for further processing. The audio circuit 860 may also include an earplug jack to provide communication between an external headset and the electronic device 800.

[0164] The electronic device 800 can help the user receive requests, send information, etc. through the transmission module 870 (such as a Wi-Fi module), which provides the user with wireless broadband Internet access. Although the transmission module 870 is shown in the figure, it is understandable that it is not a necessary component of the electronic device 800 and can be omitted as needed without changing the essence of the invention.

[0165] The processor 880 is the control center of the electronic device 800. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 820, and calling data stored in the memory 820, it executes various functions of the electronic device 800 and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 880 may include one or more processing cores; in some embodiments, the processor 880 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communications. It can be understood that the above-mentioned modem processor may not be integrated into the processor 880.

[0166] The electronic device 800 also includes a power supply 890 (such as a battery) for supplying power to various components. In some embodiments, the power supply can be logically connected to the processor 880 through a power management system, so that the power management system can manage charging, discharging, and power consumption management. The power supply 890 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0167] Although not shown, the electronic device 800 also includes a camera (such as a front camera, a rear camera), a Bluetooth module, etc., which will not be described in detail here. Specifically in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal also includes a memory, and one or more programs, wherein one or more programs are stored in the memory, and are configured to be executed by one or more processors to implement any step in the training method of the video text retrieval model provided in the above embodiment.

[0168] In specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined and implemented as the same or several entities. The specific implementation of the above modules can be found in the previous method embodiments, which will not be repeated here.

[0169] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling related hardware through instructions, and the instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, an embodiment of the present application provides a storage medium in which a plurality of instructions are stored, and when the instructions are executed by the processor, any step in the training method of the video text retrieval model provided in the above embodiments can be implemented.

[0170] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0171] Since the instructions stored in the storage medium can execute the steps in any embodiment of the training method for the video text retrieval model provided in the embodiments of the present application, the beneficial effects that can be achieved by any video text retrieval model training method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0172] The above is a detailed introduction to the training method, device, electronic device and storage medium of a video text retrieval model provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application. Moreover, for those of ordinary skill in the art, without departing from the principles of the present application, several improvements and modifications can be made, and these improvements and modifications are also regarded as the scope of protection of the present application.

Claims

1. A training method for a video text retrieval model, characterized in that: include: Acquire a training sample, wherein the training sample includes an original video, first text data, and second text data, and the second text data is obtained by language conversion based on the first text data; Extracting features from the training sample to obtain initial video features, first text features, and second text features; Performing feature interaction and aggregation processing on the initial video features to obtain video features carrying context information; The video text retrieval model is trained according to the first text feature, the second text feature and the video feature carrying the context information, and a trained video text retrieval model is obtained when it is determined that the training is completed.

2. The method according to claim 1, characterized in that The extracting features of the training sample to obtain the initial video features, the first text features and the second text features includes: Sampling video frames of the original video to obtain video key frames for feature extraction, and extracting features from the video key frames to obtain initial video features of the original video; Feature extraction is performed on the first text data to obtain a first text feature, and feature extraction is performed on the second text data to obtain a second text feature.

3. The method according to claim 2, characterized in that The step of extracting features from the first text data to obtain first text features includes: Performing word segmentation processing on the first text data to obtain a first text sequence; Filling the first text sequence according to the position information corresponding to the first text sequence; Performing encoding processing on the padded first text sequence to obtain a first text feature of the first text data; The step of extracting features from the second text data to obtain second text features includes: Performing word segmentation processing on the second text data to obtain a second text sequence; Filling the second text sequence according to the position information corresponding to the second text sequence; The padded second text sequence is encoded to obtain a second text feature of the second text data.

4. The method according to claim 1, characterized in that The performing feature interaction and aggregation processing on the initial video features to obtain video features carrying context information includes: Performing feature interaction on each feature in the initial video feature based on a multi-head attention mechanism to obtain corresponding interaction features; Performing size mapping processing on the interactive feature to obtain an interactive video feature, wherein the interactive video feature has the same feature size as the initial video feature; The interactive video features and the initial video features are subjected to feature aggregation processing to obtain video features carrying context information.

5. The method according to claim 4, characterized in that The feature interaction of each feature in the initial video feature based on the multi-head attention mechanism to obtain corresponding interaction features includes: Selecting a first initial video feature from the initial video features, and calculating an interactive similarity matrix between the first initial video feature and the second initial video feature based on a multi-head attention mechanism, wherein the initial video feature consists of the first initial video feature and the second initial video feature, and the number of the first initial video feature is one; Aggregating the similarity matrix with each video feature in the second initial video feature, and summarizing the obtained aggregation results to obtain an interaction feature corresponding to the first initial video feature; When each feature of the initial video features is processed, the interactive feature corresponding to the initial video feature is obtained.

6. The method according to claim 1, characterized in that The step of training the video text retrieval model according to the first text feature, the second text feature and the video feature, and obtaining the trained video text retrieval model when the training is completed, comprises: Reconstructing the video feature according to the first text feature to obtain a first reconstructed video feature, and reconstructing the video feature according to the second text feature to obtain a second reconstructed video feature; The video text retrieval model is trained according to the first text feature and the first reconstructed video feature to obtain a first loss value, and the video text retrieval model is trained according to the second text feature and the second reconstructed video feature to obtain a second loss value; When the training is completed according to the first loss value and the second loss value, a trained video text retrieval model is obtained.

7. The method according to claim 6, characterized in that The step of reconstructing the video feature according to the first text feature to obtain a first reconstructed video feature includes: Calculating a similarity weight between each feature in the video feature and the first text feature; updating each feature in the video features according to the similarity weight, and performing feature mapping on each updated feature to obtain a mapped video feature; Feature aggregation is performed on the mapped video features to obtain first reconstructed video features associated with the first text data.

8. The method according to claim 6, characterized in that The step of training a video text retrieval model according to the first text feature and the first reconstructed video feature to obtain a first loss value, and training a video text retrieval model according to the second text feature and the second reconstructed video feature to obtain a second loss value includes: Inputting the first text feature and the first reconstructed video feature into a video text retrieval model, calculating a first similarity matrix between the first text feature and the first reconstructed video feature, and calculating a first loss value of a text retrieval video trained based on the training sample and a first loss of a video retrieval text based on the first similarity matrix; The second text feature and the second reconstructed video feature are input into a video text retrieval model, a second similarity matrix between the second text feature and the second reconstructed video feature is calculated, and based on the second similarity matrix, a second loss value of the text retrieval video trained based on the training sample and a second loss of the video retrieval text are calculated.

9. The method according to claim 6, characterized in that The step of obtaining a trained video text retrieval model when the training is completed according to the first loss value and the second loss value includes: Adding the first loss value and the second loss value to obtain a total loss value during training of the video text retrieval model; When the total loss value is less than a preset threshold, the training is determined to be completed and a trained video text retrieval model is obtained.

10. The method according to claim 1, characterized in that After the trained video text retrieval model is obtained when the training is determined to be completed, the method further includes: Receive input video to be retrieved; Performing frame sampling on the video to be retrieved to obtain a target video frame; Extracting features of the target video frame to obtain target video features of the video to be retrieved; The target video feature is input into a trained video text retrieval model to obtain the similarity value between the target video feature and each text feature in the database, and the target text data corresponding to the maximum similarity value is output as the data matching the video to be retrieved, wherein each text data in the database corresponds to a text feature.

11. A training device for a video text retrieval model, characterized in that: include: A data acquisition module, used to acquire a training sample, wherein the training sample includes an original video, a first text data and a second text data, and the second text data is obtained by language conversion based on the first text data; A feature extraction module, used to extract features from the training sample to obtain an initial video feature, a first text feature, and a second text feature; A feature processing module, used for performing feature interaction and aggregation processing on the initial video features to obtain video features carrying context information; The model training module is used to train the video text retrieval model according to the first text feature, the second text feature and the video feature carrying context information, and obtain the trained video text retrieval model when the training is determined to be completed.

12. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the steps in the method according to any one of claims 1 to 10 when executing the computer program.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the method according to any one of claims 1 to 10 are implemented.

Images