Video text retrieval model training method and device, equipment and storage medium

Abstract

The invention discloses a training method and device of a video text retrieval model, electronic equipment and a storage medium. The method comprises the steps of obtaining a training sample; feature extraction is carried out on the original video and the text data to obtain video frame features and text features, and the video frame features comprise global video frame features and local video frame features; a video text retrieval model is trained according to the video features and the text features, a loss function during training is obtained, the loss function is determined based on a first similarity value and a second similarity value, the first similarity value is obtained through training based on the local video features and the text features, and the second similarity value is obtained through training based on the local text features; the second similarity value is obtained by training on the basis of global video features and text features; and obtaining a trained video text retrieval model under the condition of determining that the training is completed according to the loss function. And the video retrieval matching accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of machine learning and video retrieval technology, and in particular to a training method, device, equipment and storage medium for a video text retrieval model. Background Art

[0002] With the advent of the big data era, video media software (such as YouTube, TikTok, and Instagram) has become the main way for humans to learn and entertain. Faced with the explosive growth of massive video information, people urgently need to find an efficient way to quickly search for target videos that meet user needs. At the same time, with the development of mobile photography technology and the expansion of storage devices, more and more users are paying attention to short or long video content, and a large amount of dormant video data in album products needs to be understood and managed. This has led to a way to search for videos through language and text, video text retrieval.

[0003] Generally speaking, the video text retrieval task can be divided into four parts: video feature extraction, text feature extraction, feature alignment, and video text modality similarity calculation. For the more important feature alignment part, the video and text are usually globally aligned, while the local features of the video, that is, the underlying semantic features such as contour, edge, color, texture and shape features are ignored, which makes the retrieval matching inaccurate when performing video text retrieval. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a training method, device, electronic device and storage medium for a video text retrieval model to solve the technical problem of low video text retrieval matching accuracy in related technologies.

[0005] In a first aspect, an embodiment of the present application provides a method for training a video text retrieval model, comprising:

[0006] Acquire a training sample, wherein the training sample includes an original video and text data, and the original video corresponds to the text data one by one;

[0007] Performing feature extraction on the original video and the text data to obtain video features and text features, wherein the video features include global features, local features, and original features of the original video;

[0008] The video text retrieval model is trained according to the video features and the text features, and a loss value during the training is obtained, wherein the loss value is determined based on a first similarity value and a second similarity value, the first similarity value is obtained by training based on the local features and the text features, and the second similarity value is obtained by training based on the global features and the text features;

[0009] When the training is completed according to the loss value, a trained video text retrieval model is obtained.

[0010] In a second aspect, an embodiment of the present application provides a training device for a video text retrieval model, comprising:

[0011] A sample acquisition module, used to acquire training samples, wherein the training samples include original video and text data, and the original video corresponds to the text data one by one;

[0012] A feature extraction module, used for performing feature extraction on the original video and the text data to obtain video features and text features, wherein the video features include global features, local features and original features of the original video;

[0013] A training processing module, used for training a video text retrieval model according to the video features and the text features, and obtaining a loss value during training, wherein the loss value is determined based on a first similarity value and a second similarity value, the first similarity value is obtained by training based on the local features and the text features, and the second similarity value is obtained by training based on the global features and the text features;

[0014] The training judgment module is used to obtain a trained video text retrieval model when the training is determined to be completed according to the loss value.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the training method of the video text retrieval model described in any one of the above items when executing the computer program.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps in the training method of the video text retrieval model described in any one of the above items.

[0017] The embodiment of the present application provides a training method, device, electronic device and storage medium for a video text retrieval model. When training the constructed video text retrieval model, a training sample containing original video and text data is obtained, and the original video corresponds to the text data one by one. Then, feature extraction is performed on the original video and text data respectively to obtain video frame features and text features containing global video frame features and local video frame features. Then, when training, a dual alignment stage method is used to train the model, and local video frame features and text features are aligned and trained respectively, and global video frame features and text features are aligned and trained respectively. Then, according to the loss values ​​obtained from the training in the two different stages, it is determined whether the model training is completed, and then a trained video text retrieval model is obtained when it is determined that the training is completed. In the process of model training, a dual alignment method is used to perform alignment training of local video frame features and text features and global video frame features and text features respectively, so as to reduce the influence between the two training stages, improve the accuracy of model optimization and training, and thus improve the accuracy of video retrieval matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flow chart of a training method of a video text retrieval model provided in an embodiment of the present application;

[0019] Figure 2 It is a flowchart of the steps of extracting features from an original video provided in an embodiment of the present application;

[0020] Figure 3 is a flowchart of steps for obtaining video features provided in an embodiment of the present application;

[0021] Figure 4 It is a schematic diagram of a block diagram for feature fusion processing provided by the present application;

[0022] Figure 5 It is a flowchart of the steps of training a video text retrieval model provided in an embodiment of the present application;

[0023] Figure 6 It is a flowchart of the steps of pooling local features provided by the present application;

[0024] Figure 7 It is a schematic diagram of a block diagram of training a video text retrieval model provided in an embodiment of the present application;

[0025] Figure 8 It is a structural schematic diagram of a training device for a video text retrieval model provided in an embodiment of the present application;

[0026] Fig. 9 is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0027] Fig.10 This is another structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0029] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0030] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0031] In the related art, video information has richer local information than text data. Therefore, during the model training process, when aligning the features of video and text, the local information and global information of the video need to be aligned with the text respectively. However, in the existing training, the two stages are usually placed in the same training process, so that the training of the two stages affects each other, making it difficult to optimize the model to the best performance, resulting in low accuracy when performing retrieval matching based on the obtained model.

[0032] In order to solve the technical problems existing in the related art, the present application embodiment provides a training method for a video text retrieval model, see Figure 1 , Figure 1 It is a flow chart of a method for training a video text retrieval model provided in an embodiment of the present application, and the method includes steps 101 to 104.

[0033] Step 101: Obtain training samples, wherein the training samples include original video and text data, and the original video and the text data correspond one to one.

[0034] In one embodiment, during training, a training sample is obtained for training, and the training sample includes a number of original videos and a number of text data, and each original video corresponds to a text data, that is, an original video in the training sample matches a text data.

[0035] Step 102, extracting features from the original video and text data to obtain video features and text features, wherein the video features include global features, local features, and original features of the original video.

[0036] In one embodiment, after obtaining the training samples, they are input into the video text retrieval model for training, and then feature extraction is performed on the training samples. Specifically, feature extraction is performed on the original data and text data respectively to obtain corresponding video features and text features, and the obtained video features include the original features, local features and global features of the original video, wherein the original features are features obtained by extracting features from the original video, and the global features and local features are vectors initially randomly added. By combining features with the original features, such as splicing, the video features corresponding to the original video are obtained.

[0037] Furthermore, when extracting features from original video and text data, it includes: sampling video frames of the original video to obtain video key frames for feature extraction; extracting and adding features to the video key frames to obtain video features, and extracting features from text data to obtain text features.

[0038] Specifically, when obtaining video features and text features, CLIP-ViP (An video-text pre-training model on the basis of CLIP (Contrastive Language-Image Pre-training), a video text pre-training model based on CLIP) can be used to process the original video and text data to obtain video features and text features.

[0039] When extracting text features based on CLIP-ViP, the text feature extractor in CLIP-ViP is used to extract features from the text using its own vocabulary to obtain text features. When extracting features from the original video, the video encoder in CLIP-ViP is used to encode the original video, and then feature extraction is performed on the encoded original video.

[0040] For example, when extracting features from an original video, firstly, sample and obtain video frames from the original video, and then extract features from the sampled video frames to obtain key video frames for feature extraction. When sampling video frames, video frames can be randomly sampled from the original video, such as randomly sampling 12 video frames as video key frames for subsequent feature extraction.

[0041] Among them, when extracting features from the original video, you can refer to Figure 2 , Figure 2 It is a flowchart of the steps of extracting features from an original video provided in an embodiment of the present application, wherein the steps include steps 201 to 203.

[0042] Step 201, extracting features from the video key frames to obtain original features of the video key frames;

[0043] Step 202, setting an initial feature vector, and adding the initial feature vector to the original feature to obtain an original video feature matrix, wherein the initial feature vector includes an initial global video feature and an initial local video feature;

[0044] Step 203, performing feature fusion and normalization processing on the original video feature matrix to obtain the video features of the original video.

[0045] Specifically, after sampling the video frames of the original video to obtain video key frames for feature extraction, feature extraction is performed on it to obtain original features. When obtaining the video features of the original video, an initial feature vector is set, and then the initial feature vector is added to the original features to obtain an original video feature matrix. Finally, the original video feature matrix is ​​subjected to relevant processing such as feature fusion and normalization to obtain the video features of the original video.

[0046] Exemplarily, when extracting features from the sampled video key frames, the video encoder in CLIP-ViP is used to process the video key frames. Each video frame is split to obtain a 7*7 image block, and then a convolutional neural network is used to extract the original features of each image block to obtain a feature matrix of the entire image block as the original features of a video frame, and then n+1 random vectors are added to the obtained feature matrix, where 1 vector is used as the global feature of the video and n vectors are used as the local features of the video to form the original video feature matrix of the original video. For the added random vectors, n is usually set to 3 or 4.

[0047] The obtained original video feature matrix can be expressed as F = [f0, f1, f2, ..., f n ,F0], where f0 is the global video feature, f1~fn is the local video feature, and F0 is the original feature of the video key frame.

[0048] After the processing of the video key frames is completed and the original video feature matrix of the original video is obtained, the corresponding feature fusion and normalization processing are performed, and the video features of the original video are obtained after the processing is completed. Figure 3 , Figure 3 It is a flowchart of the steps of obtaining video features provided in an embodiment of the present application, wherein the steps include steps 301 to 303.

[0049] Step 301, normalizing the original video feature matrix to obtain a first video feature matrix;

[0050] Step 302, performing feature fusion on each feature in the first video feature matrix to obtain a second video feature matrix;

[0051] Step 303: normalize the second video feature matrix to obtain video features of the original video.

[0052] Specifically, when processing the obtained original video feature matrix, normalization is first performed to obtain a first video feature matrix, and then the features in the first video feature matrix are fused to obtain a second video feature matrix. Finally, the second video feature matrix is ​​normalized again to obtain video features of the original video.

[0053] For example, the original video feature matrix F = [f0, f1, f2, ..., f n ,F0] is processed, it is input into the video feature extractor in CLIP-ViP, and after being processed by the normalization layer and multi-head attention mechanism in the feature extractor, the first video feature matrix is ​​obtained:

[0054]

[0055] Among them, F 1 The superscripts in indicate the number of times the treatment was performed.

[0056] Then, a feature fusion module is added to the 12-layer Transformer module stacked in the video feature extractor in CLIP-ViP to perform feature fusion on the first video feature matrix to obtain the second video feature matrix. The block diagram of the feature fusion process can be referred to Figure 4 ,like Figure 4As shown, when performing feature fusion, the global features and local features in the first video feature matrix are processed separately. First, the local features in the first video feature matrix are fused, and then the fusion results are fused with the global features to finally obtain the video features of the original video.

[0057] That is, when performing feature fusion processing between various features, it includes: performing feature fusion on the first local features in the first video feature matrix to obtain a first fused video feature matrix; performing feature fusion on the first fused video feature matrix with the global features in the first video feature matrix to obtain a second video feature matrix.

[0058] by Figure 4 For example, after completing the first feature fusion based on the first video feature matrix, the fused features are as follows:

[0059]

[0060] in, for Figure 4 The result after the fusion of the i-th feature in the first layer, for Figure 4 The result of fusion of the i-th feature in the second layer, λ1 and λ2 are adjustable hyperparameters, 0.5<λ1,λ2<1.

[0061] After completing the feature fusion process, the second feature matrix containing local features and global features can be obtained:

[0062]

[0063] Finally, F 2 After subsequent normalization and MLP layer processing, the video features corresponding to the original video are obtained:

[0064]

[0065] in, is a global feature, are n local features.

[0066] Step 103, train the video text retrieval model according to the video features and the text features, and obtain a loss value during training, wherein the loss value is determined based on a first similarity value and a second similarity value, the first similarity value is obtained by training based on local features and text features, and the second similarity value is obtained by training based on global features and text features.

[0067] In one embodiment, after the feature extraction of the original video and text data is completed and the video features and text features are obtained, the constructed video text retrieval model will be trained based on the video features and text features. Specifically, during the training, the video features and text features are input into the video text retrieval model to train the model, and the loss value generated during the training is obtained, wherein the loss value is obtained by training respectively in two alignment stages, and the two alignment stages are local information alignment and global information alignment.

[0068] For example, when training in the local information alignment stage, training is performed based on local features and text features, and when training in the global information alignment stage, training is performed based on global features and text features. Figure 5 , Figure 5 It is a flowchart of the steps for training a video text retrieval model provided in an embodiment of the present application, wherein the steps include steps 501 to 503.

[0069] Step 501, local features and text features are combined into a first training group, and global features and text features are combined into a second training group;

[0070] Step 502, performing a first phase training on the video text retrieval model according to the first training group, and performing a second phase training on the video text retrieval model according to the second training group after completing the first phase training based on the first training group;

[0071] Step 503, according to the loss values ​​of the first stage training and the second stage training based on the training samples, the loss value for training the video text retrieval model is obtained.

[0072] Specifically, when performing two-stage alignment training, training is performed in the local information alignment stage and the global information alignment stage. Therefore, when performing training, local features and text features can be combined to obtain a first training group, and global features and text features can be combined to obtain a second training group. Then, the video text retrieval model is trained in the first stage according to the first training group, that is, the local information alignment stage. After training based on the first training group, the video text retrieval model is trained in the second stage using the second training group, that is, the global information alignment stage. At the same time, for each stage during training, there will be a corresponding loss value to determine the loss value of the entire video text retrieval model during training based on the loss values ​​of the two stages.

[0073] Exemplarily, during training, the process includes: performing pooling processing on the third local video feature in the first training group to obtain a pooled feature corresponding to the third local video feature; performing first-stage training based on the pooled feature and the text feature input into a video text retrieval model to obtain a first similarity value between the pooled feature and the text feature; after the first-stage training, inputting the global video feature and the text feature into the video text retrieval model to perform second-stage training to obtain a second similarity value between the global video feature and the text feature.

[0074] That is, when performing training in the local information alignment stage, the first training group including local features and text features is used for training, and before training, the third local video features included in the first training group are first pooled to obtain the pooled features of the third local video features, and then the pooled features and text features are used to perform local information alignment training.

[0075] When performing pooling processing, it includes: performing linear processing on text data and local features to obtain linear text features and linear local features; performing feature fusion of linear text features and linear local features to obtain a similarity matrix between linear text features and linear local features; performing feature fusion of the similarity matrix and linear text features to obtain pooling features corresponding to the third local feature.

[0076] For example, when the third local feature is pooled, the processing process may refer to Figure 6 ,exist Figure 6 In the processing process shown, the linear layer is used to process the text features and local features respectively to obtain the corresponding linear text features and linear local features, and then the pooling features corresponding to the local features are obtained through feature fusion processing, such as matrix multiplication.

[0077] Specifically, when performing pooling, based on Figure 6 The processing box shown in the figure performs pooling on the local features. Through the pooling process, multiple local features are pooled into one feature. When performing pooling, higher weights are assigned to the local features that have high similarity with the text features:

[0078]

[0079] Among them, w is the obtained weight matrix, w∈R 1×n , n is the number of local features, W T , is the projection matrix, which maps the eigenvector to the specified dimension. D is the feature vector dimension, D p is the dimension of the vector after mapping.

[0080] Using w as the weight parameter, we can get the pooling features corresponding to the local features after pooling:

[0081]

[0082] in, is the projection matrix.

[0083] At the same time, when the training of this stage is completed, the cosine similarity between the pooled feature and the text feature can be calculated, that is, the first similarity value:

[0084]

[0085] In addition, when training in the global information alignment stage, the second training group consisting of global features and text features is directly input into the video text retrieval model for training. At the same time, when the training of this stage is completed, the cosine similarity between the global features and the text features can be calculated, that is, the second similarity value:

[0086]

[0087] Furthermore, when obtaining the loss value generated by training in two stages, it includes: calculating according to the first similarity value to obtain a first loss value of the text retrieval video when training is based on the training sample; calculating according to the second similarity value to obtain a second loss value of the text retrieval video when training is based on the training sample; adding the first loss value and the second loss value to obtain a single loss value when training is based on each sample in the training sample, and summarizing the single loss values ​​to obtain a loss value when training is based on the training sample.

[0088] During the training process, in order to improve the accuracy of retrieval matching, the loss of the model when completing the text retrieval video and video retrieval text tasks can be determined based on the first similarity value and the second similarity value obtained during training. The total loss value of the model is then obtained by adding the losses to determine whether the model training is complete.

[0089] For example, for B video-text pairs in the same batch, that is, the number of training samples is set to B, the cross entropy loss function is calculated using the obtained similarity to obtain the loss function L for the text retrieval video task. t2v The loss function L for the video retrieval text task v2t .

[0090] Among them, the calculated loss function is:

[0091]

[0092]

[0093] L=L t2v +L v2t ;

[0094] Moreover, sim is the sum of similarities obtained in the two stages, and L is the total loss value of the model.

[0095] Step 104, when the training is completed according to the loss value, a trained video text retrieval model is obtained.

[0096] In one embodiment, whether the training is completed can be determined based on the obtained loss value. During the continuous training and optimization process, the loss value will continue to decrease until the conditions for the completion of the training are met, such as the loss value is less than the set threshold. When the training is completed, a trained video text retrieval model is obtained.

[0097] Further, refer to Figure 7 , Figure 7 It is a block diagram of the training of the video text retrieval model provided in the embodiment of the present application.

[0098] like Figure 7 As shown, during training, firstly, feature extraction is performed on the video data (original video) and text data to obtain video features and text features containing local features and global features. Then, the local features and text features are used to perform training in the local information alignment stage, and the global features and text features are used to perform training in the global information alignment stage. Finally, a trained video text retrieval model is obtained when the training is completed.

[0099] When performing feature extraction, when processing text data, the text encoder is used to process it, and then the text feature extractor is used to extract the text features. When processing video data, the video encoder is used to encode the data, and then the video feature extractor is used to extract the features. When performing feature extraction, the encoded result is processed by the normalization layer and multi-head attention to obtain the video feature matrix, and then processed by the feature fusion module to complete the fusion of each feature in the video feature matrix. Finally, the normalization layer and MLP layer are used to process the video data to obtain the video features, and the obtained video features contain local features and global features.

[0100] Furthermore, after completing the training of the constructed video text retrieval model, when in use, the video data to be queried and matched is input into the trained video text retrieval model, and the final matching process is completed by calculating the similarity between the global features of the video and the text features. Specifically, it includes: receiving the input video to be retrieved; sampling the frames of the video to be retrieved to obtain the target video frame; extracting the features of the target video frame to obtain the target video frame features of the video to be retrieved; inputting the target video frame features into the trained video text retrieval model to obtain the similarity values ​​between the target video frame features and each text feature in the database, and outputting the target text data corresponding to the maximum similarity value as the data matching the video to be retrieved, wherein the text data corresponds to the text features one by one.

[0101] Exemplarily, when a video to be retrieved is received, frame sampling is performed on the video to be retrieved to obtain a target video frame for feature extraction, and then video features are extracted from the target video frame to obtain target video frame features, which are then input into a trained video text retrieval model. Through the processing of the model, the similarity value between the target video frame feature and the text feature of each text data can be calculated, and the text data corresponding to the maximum similarity value is the target text that matches the video to be retrieved.

[0102] In summary, the present application discloses a training method for a video text retrieval model. When training the constructed video text retrieval model, a training sample containing original video and text data is obtained, and the original video corresponds to the text data one by one. Then, feature extraction is performed on the original video and text data respectively to obtain video frame features and text features containing global video frame features and local video frame features. Then, during training, a dual alignment stage method is used to train the model, and local video frame features and text features are aligned and trained respectively, and global video frame features and text features are aligned and trained respectively. Then, according to the loss values ​​obtained from the training in the two different stages, it is determined whether the model training is completed, and then a trained video text retrieval model is obtained when it is determined that the training is completed. In the process of model training, a dual alignment method is used to perform alignment training of local video frame features and text features and global video frame features and text features respectively, so as to reduce the influence between the two training stages, improve the accuracy of model optimization and training, and thus improve the accuracy of video retrieval matching.

[0103] According to the method described in the above embodiment, this embodiment will be further described from the perspective of a training device for a video text retrieval model. The training device for the video text retrieval model can be implemented as an independent entity or integrated in an electronic device, such as a terminal. The terminal may include a mobile phone, a tablet computer, etc.

[0104] See also Figure 8 , Figure 8 is a structural diagram of a training device for a video text retrieval model provided in an embodiment of the present application, such as Figure 8 As shown, the video text retrieval model training device 800 provided in the embodiment of the present application includes:

[0105] The sample acquisition module 801 is used to acquire training samples, wherein the training samples include original video and text data, and the original video and the text data correspond one to one;

[0106] A feature extraction module 802 is used to extract features from the original video and text data to obtain video features and text features, wherein the video features include global features, local features and original features of the original video;

[0107] A training processing module 803 is used to train the video text retrieval model according to the video features and the text features, and obtain a loss value during the training, wherein the loss value is determined based on a first similarity value and a second similarity value, the first similarity value is obtained by training based on local features and text features, and the second similarity value is obtained by training based on global features and text features;

[0108] The training judgment module 804 is used to obtain a trained video text retrieval model when the training is determined to be completed according to the loss value.

[0109] In one embodiment, the feature extraction module 802 is further used to:

[0110] Sampling video frames of the original video to obtain video key frames for feature extraction;

[0111] Feature extraction and feature addition are performed on video key frames to obtain video features, and feature extraction is performed on text data to obtain text features.

[0112] In one embodiment, the feature extraction module 802 is further used to:

[0113] Extract features from video key frames to obtain original features of video key frames;

[0114] Setting an initial feature vector, and adding the initial feature vector to the original feature to obtain an original video feature matrix, wherein the initial feature vector includes an initial global video feature and an initial local video feature;

[0115] The feature fusion and normalization processing are performed on the original video feature matrix to obtain the video features of the original video.

[0116] In one embodiment, the feature extraction module 802 is further used to:

[0117] Normalizing the original video feature matrix to obtain a first video feature matrix;

[0118] Performing feature fusion on each feature in the first video feature matrix to obtain a second video feature matrix;

[0119] The second video feature matrix is ​​normalized to obtain the video features of the original video.

[0120] In one embodiment, the feature extraction module 802 is further used to:

[0121] Performing feature fusion on the first local features in the first video feature matrix to obtain a first fused video feature matrix;

[0122] The first fused video feature matrix is ​​fused with the global features in the first video feature matrix to obtain a second video feature matrix.

[0123] In one embodiment, the training processing module 803 is further used to:

[0124] The local features and the text features are combined into a first training group, and the global features and the text features are combined into a second training group;

[0125] Performing a first-stage training on the video text retrieval model according to the first training group, and performing a second-stage training on the video text retrieval model according to the second training group after completing the first-stage training based on the first training group;

[0126] According to the loss values ​​of the first stage training and the second stage training based on the training samples, the loss value of training the video text retrieval model is obtained.

[0127] In one embodiment, the training processing module 803 is further used to:

[0128] Performing pooling processing on the third local feature in the first training group to obtain a pooling feature corresponding to the third local feature;

[0129] The pooled features and text features are input into the video text retrieval model for the first stage of training to obtain the first similarity value between the pooled features and the text features;

[0130] After the first stage of training, the global features and text features are input into the video text retrieval model for the second stage of training to obtain the second similarity value between the global features and the text features.

[0131] In one embodiment, the training processing module 803 is further used to:

[0132] Perform linear processing on text data and local features to obtain linear text features and linear local features;

[0133] The linear text features and the linear local features are fused to obtain a similarity matrix between the linear text features and the linear local features;

[0134] The similarity matrix is ​​fused with the linear text feature to obtain the pooled feature corresponding to the third local feature.

[0135] In one embodiment, the training processing module 803 is further used to:

[0136] Calculating according to the first similarity value, obtaining a first loss value of the text retrieval video when training is performed based on the training sample;

[0137] Calculating according to the second similarity value to obtain a second loss value of the text retrieval video when training is performed based on the training sample;

[0138] The first loss value and the second loss value are added to obtain a single loss value when training is performed based on each sample in the training samples, and the single loss values ​​are summarized to obtain a loss value when training is performed based on the training samples.

[0139] In one embodiment, the video text retrieval model training device 800 further includes a retrieval matching module, which is used to:

[0140] Receive input video to be retrieved;

[0141] Perform frame sampling on the video to be retrieved to obtain the target video frame;

[0142] Extract features of the target video frame to obtain the target video frame features of the video to be retrieved;

[0143] The target video frame features are input into the trained video text retrieval model to obtain the similarity values ​​between the target video frame features and the text features in the database, and the target text data corresponding to the maximum similarity value is output as the data matching the video to be retrieved, where the text data corresponds to the text features one by one.

[0144] Also, see Fig. 9 , Fig. 9 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, and the electronic device may be a mobile terminal such as a smart phone, a tablet computer, or the like. Fig. 9 As shown, the electronic device 900 includes a processor 901 and a memory 902. The processor 901 is electrically connected to the memory 902.

[0145] The processor 901 is the control center of the electronic device 900. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device 900 and processes data by running or loading applications stored in the memory 902 and calling data stored in the memory 902, thereby monitoring the electronic device 900 as a whole.

[0146] In this embodiment, the processor 901 in the electronic device 900 will load the instructions corresponding to the processes of one or more applications into the memory 902 according to the following steps, and the processor 901 will run the application stored in the memory 902, thereby implementing any step in the training method of the video text retrieval model provided in the above embodiment.

[0147] The electronic device 900 can implement the steps in any embodiment of the training method for the video text retrieval model provided in the embodiments of the present application. Therefore, it can achieve the beneficial effects that can be achieved by any video text retrieval model training method provided in the embodiments of the present application. Please refer to the previous embodiments for details and will not be repeated here.

[0148] See also Fig.10 , Fig.10 is another structural schematic diagram of an electronic device provided in an embodiment of the present application, such as Fig.10 As shown, Fig.10 The specific structural block diagram of the electronic device provided in the embodiment of the present application is shown, and the electronic device can be used to implement the training method of the video text retrieval model provided in the above embodiment. The electronic device 1000 can be a mobile terminal such as a smart phone or a laptop computer.

[0149] The RF circuit 1010 is used to receive and send electromagnetic waves, realize the mutual conversion between electromagnetic waves and electrical signals, and thus communicate with a communication network or other devices. The RF circuit 1010 may include various existing circuit elements for performing these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, user identity module (SIM) cards, memories, etc. The RF circuit 1010 can communicate with various networks such as the Internet, corporate intranets, wireless networks, or communicate with other devices through wireless networks. The above-mentioned wireless networks may include cellular telephone networks, wireless local area networks, or metropolitan area networks. The above-mentioned wireless networks can use various communication standards, protocols and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as Institute of Electrical and Electronics Engineers standards IEEE 802.11a, IEEE 802.11b, IEEE802.11g and / or IEEE802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging and short messages, and any other suitable communication protocols, even those that have not yet been developed.

[0150] The memory 1020 can be used to store software programs and modules, such as the program instructions / modules corresponding to the training method of the video text retrieval model in the above-mentioned embodiment. The processor 1080 executes various functional applications and the training method of the video text retrieval model by running the software programs and modules stored in the memory 1020.

[0151] The memory 1020 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 1020 may further include a memory remotely arranged relative to the processor 1080, and these remote memories may be connected to the electronic device 1000 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0152] The input unit 1030 can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. Specifically, the input unit 1030 may include a touch-sensitive surface 1031 and other input devices 1032. The touch-sensitive surface 1031, also known as a touch display screen or a touch pad, can collect user touch operations on or near it (such as operations performed by the user using any suitable object or accessory such as a finger, stylus, etc. on or near the touch-sensitive surface 1031), and drive the corresponding connection device according to a pre-set program. Optionally, the touch-sensitive surface 1031 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch direction, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into touch point coordinates, and then sends it to the processor 1080, and can receive and execute commands sent by the processor 1080. In addition, the touch-sensitive surface 1031 may be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch-sensitive surface 1031, the input unit 1030 may further include other input devices 1032. Specifically, the other input devices 1032 may include, but are not limited to, one or more of a physical keyboard, a function key (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0153] The display unit 1040 can be used to display information input by the user or information provided to the user and various graphical user interfaces of the electronic device 1000, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit 1040 may include a display panel 1041, and optionally, the display panel 1041 may be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc. Further, the touch-sensitive surface 1031 may cover the display panel 1041, and when the touch-sensitive surface 1031 detects a touch operation on or near it, it is transmitted to the processor 1080 to determine the type of the touch event, and then the processor 1080 provides corresponding visual output on the display panel 1041 according to the type of the touch event. Although in the figure, the touch-sensitive surface 1031 and the display panel 1041 are implemented as two independent components to implement input and output functions, in some embodiments, the touch-sensitive surface 1031 and the display panel 1041 can be integrated to implement input and output functions.

[0154] The electronic device 1000 may also include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1041 according to the brightness of the ambient light, and the proximity sensor may generate an interrupt when the flip cover is closed or closed. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary, which can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc. that can also be configured in the electronic device 1000, they will not be repeated here.

[0155] The audio circuit 1060, the speaker 1061, and the microphone 1062 can provide an audio interface between the user and the electronic device 1000. The audio circuit 1060 can transmit the received audio data to the speaker 1061 after converting the received audio data into an electrical signal, which is converted into a sound signal for output by the speaker 1061; on the other hand, the microphone 1062 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1060 and converted into audio data, and then the audio data is processed by the output processor 1080 and sent to, for example, another terminal through the RF circuit 1010, or the audio data is output to the memory 1020 for further processing. The audio circuit 1060 may also include an earplug jack to provide communication between an external headset and the electronic device 1000.

[0156] The electronic device 1000 can help the user receive requests, send information, etc. through the transmission module 1070 (such as a Wi-Fi module), which provides the user with wireless broadband Internet access. Although the transmission module 1070 is shown in the figure, it is understandable that it is not a necessary component of the electronic device 1000 and can be omitted as needed without changing the essence of the invention.

[0157] The processor 1080 is the control center of the electronic device 1000. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 1020, and calling data stored in the memory 1020, it executes various functions of the electronic device 1000 and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 1080 may include one or more processing cores; in some embodiments, the processor 1080 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It can be understood that the above-mentioned modem processor may not be integrated into the processor 1080.

[0158] The electronic device 1000 also includes a power supply 1090 (such as a battery) for supplying power to various components. In some embodiments, the power supply can be logically connected to the processor 1080 through a power management system, so that the power management system can manage charging, discharging, and power consumption management. The power supply 1090 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0159] Although not shown, the electronic device 1000 also includes a camera (such as a front camera, a rear camera), a Bluetooth module, etc., which will not be described in detail here. Specifically in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal also includes a memory, and one or more programs, wherein one or more programs are stored in the memory, and are configured to be executed by one or more processors to implement any step in the training method of the video text retrieval model provided in the above embodiment.

[0160] In specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined and implemented as the same or several entities. The specific implementation of the above modules can be found in the previous method embodiments, which will not be repeated here.

[0161] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling related hardware through instructions, and the instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, an embodiment of the present application provides a storage medium in which a plurality of instructions are stored, and when the instructions are executed by the processor, any step in the training method of the video text retrieval model provided in the above embodiments can be implemented.

[0162] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0163] Since the instructions stored in the storage medium can execute the steps in any embodiment of the training method for the video text retrieval model provided in the embodiments of the present application, the beneficial effects that can be achieved by any video text retrieval model training method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0164] The above is a detailed introduction to the training method, device, electronic device and storage medium of a video text retrieval model provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application. Moreover, for those of ordinary skill in the art, without departing from the principles of the present application, several improvements and modifications can be made, and these improvements and modifications are also regarded as the scope of protection of the present application.

Claims

1. A training method for a video text retrieval model, characterized in that: include: Acquire a training sample, wherein the training sample includes an original video and text data, and the original video corresponds to the text data one by one; Performing feature extraction on the original video and the text data to obtain video features and text features, wherein the video features include global features, local features, and original features of the original video; The video text retrieval model is trained according to the video features and the text features, and a loss value during the training is obtained, wherein the loss value is determined based on a first similarity value and a second similarity value, the first similarity value is obtained by training based on the local features and the text features, and the second similarity value is obtained by training based on the global features and the text features; When the training is completed according to the loss value, a trained video text retrieval model is obtained.

2. The method according to claim 1, characterized in that The extracting features of the original video and the text data to obtain video features and text features includes: Sampling video frames of the original video to obtain video key frames for feature extraction; Feature extraction and feature addition are performed on the video key frames to obtain video features, and feature extraction is performed on the text data to obtain text features.

3. The method according to claim 2, characterized in that The extracting and adding features to the video key frames to obtain video features includes: Extracting features of the video key frames to obtain original features of the video key frames; Setting an initial feature vector, and adding the initial feature vector to the original feature to obtain an original video feature matrix, wherein the initial feature vector includes an initial global video feature and an initial local video feature; The original video feature matrix is ​​subjected to feature fusion and normalization processing to obtain the video features of the original video.

4. The method according to claim 3, characterized in that The step of performing feature fusion and normalization processing on the original video feature matrix to obtain the video features of the original video includes: Normalizing the original video feature matrix to obtain a first video feature matrix; Performing feature fusion on each feature in the first video feature matrix to obtain a second video feature matrix; The second video feature matrix is ​​normalized to obtain video features of the original video.

5. The method according to claim 4, characterized in that The step of fusing the features in the first video feature matrix to obtain a second video feature matrix includes: Performing feature fusion on the first local features in the first video feature matrix to obtain a first fused video feature matrix; The first fused video feature matrix is ​​fused with the global features in the first video feature matrix to obtain a second video feature matrix.

6. The method according to claim 1, characterized in that The step of training the video text retrieval model according to the video features and the text features and obtaining a loss value during the training includes: The local features and the text features form a first training group, and the global features and the text features form a second training group; Performing a first-stage training on the video text retrieval model according to the first training group, and performing a second-stage training on the video text retrieval model according to the second training group after completing the first-stage training based on the first training group; According to the loss values ​​when the first stage training and the second stage training are performed based on the training samples, the loss value for training the video text retrieval model is obtained.

7. The method according to claim 6, characterized in that The first stage training of the video text retrieval model is performed according to the first training group, and after the first stage training is completed based on the first training group, the second stage training of the video text retrieval model is performed according to the second training group, including: Performing pooling processing on the third local feature in the first training group to obtain a pooling feature corresponding to the third local feature; Input the pooled features and the text features into a video text retrieval model for first-stage training to obtain a first similarity value between the pooled features and the text features; After the first stage of training, the global features and the text features are input into the video text retrieval model for second stage training to obtain a second similarity value between the global features and the text features.

8. The method according to claim 7, characterized in that The performing pooling processing on the third local feature in the first training group to obtain a pooling feature corresponding to the third local feature includes: Performing linear processing on the text data and the local features to obtain linear text features and linear local features; The linear text feature and the linear local feature are subjected to feature fusion to obtain a similarity matrix between the linear text feature and the linear local feature; The similarity matrix is ​​fused with the linear text feature to obtain a pooled feature corresponding to the third local feature.

9. The method according to claim 7, characterized in that The step of obtaining the loss value for training the video text retrieval model according to the loss value obtained when performing the first stage training and the second stage training based on the training sample comprises: Calculating according to the first similarity value, obtaining a first loss value of the text retrieval video when training is performed based on the training sample; Calculating according to the second similarity value to obtain a second loss value of the text retrieval video when training is performed based on the training sample; The first loss value and the second loss value are added to obtain a single loss value when training is performed based on each sample in the training samples, and the single loss values ​​are summarized to obtain a loss value when training is performed based on the training samples.

10. The method according to claim 1, characterized in that In the case where the training is determined to be completed according to the loss value, after obtaining the trained video text retrieval model, the method further includes: Receive input video to be retrieved; Performing frame sampling on the video to be retrieved to obtain a target video frame; Extracting features of the target video frame to obtain features of the target video frame of the video to be retrieved; The target video frame features are input into a trained video text retrieval model to obtain similarity values ​​between the target video frame features and each text feature in a database, and the target text data corresponding to the maximum similarity value is output as data matching the video to be retrieved, wherein the text data corresponds to the text features one-to-one.

11. A training device for a video text retrieval model, characterized in that: include: A sample acquisition module, used to acquire training samples, wherein the training samples include original video and text data, and the original video corresponds to the text data one by one; A feature extraction module, used for performing feature extraction on the original video and the text data to obtain video features and text features, wherein the video features include global features, local features and original features of the original video; A training processing module, used for training a video text retrieval model according to the video features and the text features, and obtaining a loss value during training, wherein the loss value is determined based on a first similarity value and a second similarity value, the first similarity value is obtained by training based on the local features and the text features, and the second similarity value is obtained by training based on the global features and the text features; The training judgment module is used to obtain a trained video text retrieval model when the training is determined to be completed according to the loss value.

12. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the steps in the method according to any one of claims 1 to 10 when executing the computer program.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the method according to any one of claims 1 to 10 are implemented.

Images