Training method and device for video feature extraction model

By acquiring and fusing multimodal data of videos and using label feature information to train the video feature extraction model, the problem of inaccurate video feature extraction in the prior art is solved, and more accurate video feature extraction is achieved.

CN114626464BActive Publication Date: 2024-11-08TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202210261995.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-16
Publication Date
2024-11-08
Estimated Expiration
2042-03-16

AI Technical Summary

Technical Problem

The existing video feature extraction model fails to effectively utilize the semantic correlation information between labels during training, resulting in inaccurate video feature extraction.

Method used

By obtaining the modal information and labels of the sample video, determining the label feature information, and using the label feature information as supervision information to train the video feature extraction model, combining image, title and audio data for multimodal fusion, using triple loss Function optimization model.

Benefits of technology

The accuracy of the video feature extraction model is improved, so that the model can better predict video feature information and enhance the semantic correlation between labels.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114626464B_ABST
    Figure CN114626464B_ABST
Patent Text Reader

Abstract

The present application discloses a training method and device for a video feature extraction model, which belongs to the field of computer technology. The method comprises: obtaining the modal information of a sample video and the sample label corresponding to the sample video; determining the label feature information corresponding to the sample video based on the sample label corresponding to the sample video and the trained label feature extraction model; determining the video feature information corresponding to the sample video based on the modal information of the sample video and the video feature extraction model to be trained; training the video feature extraction model to be trained based on the label feature information and video feature information corresponding to the sample video. By adopting the present application, the prediction result output by the trained video feature extraction model can be made more accurate.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a training method and device for a video feature extraction model. Background Art

[0002] For some applications with video publishing or video playback functions, in order to facilitate subsequent management, a video feature extraction model is usually used to extract features from each video, so as to obtain video feature information corresponding to each video.

[0003] Currently, the training method of the video feature extraction model is to use the video label as the supervision information. Specifically, after the modal information of the sample video is input into the video feature extraction model to obtain the video feature information corresponding to the output sample video, the video feature information of the sample video is input into the label determination model to obtain the probability corresponding to each label. The probability of each label is used to indicate the probability that the sample video matches the label. Then, the video feature extraction model and the label determination model are trained based on the actual label corresponding to the sample video and the probability of each label output by the label determination model.

[0004] Since labels are fine-grained metrics, the actual labels matched to videos are also fine-grained labels. It is possible that two videos are similar to a certain extent, but they are not matched with the same actual labels. In addition, since the semantic association information between labels is not applied in the above training process, the similarity between the video feature information of the two videos may be small, that is, the video feature information may be inaccurate. For example, the actual labels of video A are "fighting games" and "stand-alone games", and the labels of video B are "fighting" and "boxing", but since the actual labels of video A and video B are different, the similarity between the feature information of the two videos obtained after feature extraction will be small, resulting in inaccuracy of the video feature extraction model. Summary of the invention

[0005] The embodiment of the present application provides a method for training a video feature extraction model, which can solve the problem of inaccurate output results of a trained video feature extraction model in the prior art.

[0006] In a first aspect, a method for training a video feature extraction model is provided, the method comprising:

[0007] Obtaining modal information of a sample video and a sample label corresponding to the sample video;

[0008] Determine label feature information corresponding to the sample video based on the sample label corresponding to the sample video and the trained label feature extraction model;

[0009] Determining video feature information corresponding to the sample video based on the modal information of the sample video and the video feature extraction model to be trained;

[0010] The video feature extraction model to be trained is trained based on the label feature information and video feature information corresponding to the sample video.

[0011] In a possible implementation, the modality information includes at least one of image data, title text data, audio data, and subtitle data.

[0012] In a possible implementation, determining the label feature information corresponding to the sample video based on the sample label corresponding to the sample video and a trained label feature extraction model includes:

[0013] Input each sample label into the trained label feature extraction model to obtain feature information corresponding to each sample label;

[0014] Based on the feature information corresponding to each sample label and the label fusion model, the label feature information corresponding to the sample video is determined.

[0015] In a possible implementation, the modality information includes image data and title text data, and the video feature extraction model includes an image feature extraction sub-model, a title feature extraction sub-model and a modality fusion sub-model;

[0016] The determining, based on the modality information of the sample video and the video feature extraction model to be trained, the video feature information corresponding to the sample video includes:

[0017] Determining image feature information corresponding to the sample video based on the image data of the sample video and the image feature extraction sub-model to be trained;

[0018] Determine the title feature information corresponding to the sample video based on the title text data corresponding to the sample video and the title feature extraction sub-model to be trained;

[0019] Based on the modality fusion sub-model to be trained, and the image feature information and title feature information corresponding to the sample video, the video feature information corresponding to the sample video is determined.

[0020] In a possible implementation, the image data of the sample video includes a plurality of image frame data, and the image feature extraction sub-model includes an image frame feature extraction module and an image fusion module;

[0021] The determining of image feature information corresponding to the sample video based on the image data of the sample video and the image feature extraction sub-model to be trained includes:

[0022] Input each image frame data into the image frame feature extraction module to be trained to obtain the image frame feature information corresponding to each image frame data;

[0023] The image frame feature information corresponding to each image frame data is input into the image fusion module to be trained to obtain the image feature information corresponding to the sample video.

[0024] In a possible implementation, the training of the video feature extraction model to be trained based on the label feature information and the video feature information corresponding to the sample video includes:

[0025] Get sample labels corresponding to other sample videos;

[0026] Determine the label feature information corresponding to the other sample videos based on the sample labels corresponding to the other sample videos and the trained label feature extraction model;

[0027] The video feature extraction model to be trained is trained based on the label feature information corresponding to the other sample videos, and the label feature information and video feature information corresponding to the sample video.

[0028] In a possible implementation, the training of the video feature extraction model to be trained based on the label feature information corresponding to the other sample videos, and the label feature information and video feature information corresponding to the sample video includes:

[0029] Taking the label feature information corresponding to the sample video as a positive sample of the video feature information corresponding to the sample video, taking the label feature information corresponding to the other sample videos as a negative sample of the video feature information corresponding to the sample video, and calculating the loss value based on the triplet loss function, the label feature information corresponding to the other sample videos, and the label feature information and video feature information corresponding to the sample video;

[0030] Based on the loss value, the video feature extraction model to be trained is trained.

[0031] In a second aspect, a training device for a video feature extraction model is provided, the device comprising:

[0032] An acquisition module, used to acquire modality information of a sample video and a sample label corresponding to the sample video;

[0033] A first determination module, configured to determine label feature information corresponding to the sample video based on a sample label corresponding to the sample video and a trained label feature extraction model;

[0034] A second determination module, configured to determine video feature information corresponding to the sample video based on the modality information of the sample video and a video feature extraction model to be trained;

[0035] The training module is used to train the video feature extraction model to be trained based on the label feature information and video feature information corresponding to the sample video.

[0036] In a possible implementation, the modality information includes at least one of image data, title text data, audio data, and subtitle data.

[0037] In a possible implementation manner, the first determining module is configured to:

[0038] Input each sample label into the trained label feature extraction model to obtain feature information corresponding to each sample label;

[0039] Based on the feature information corresponding to each sample label and the label fusion model, the label feature information corresponding to the sample video is determined.

[0040] In a possible implementation, the modality information includes image data and title text data, and the video feature extraction model includes an image feature extraction sub-model, a title feature extraction sub-model and a modality fusion sub-model;

[0041] The second determining module is used to:

[0042] Determining image feature information corresponding to the sample video based on the image data of the sample video and the image feature extraction sub-model to be trained;

[0043] Determine the title feature information corresponding to the sample video based on the title text data corresponding to the sample video and the title feature extraction sub-model to be trained;

[0044] Based on the modality fusion sub-model to be trained, and the image feature information and title feature information corresponding to the sample video, the video feature information corresponding to the sample video is determined.

[0045] In a possible implementation, the image data of the sample video includes a plurality of image frame data, and the image feature extraction sub-model includes an image frame feature extraction module and an image fusion module;

[0046] The second determining module is used to:

[0047] Input each image frame data into the image frame feature extraction module to be trained to obtain the image frame feature information corresponding to each image frame data;

[0048] The image frame feature information corresponding to each image frame data is input into the image fusion module to be trained to obtain the image feature information corresponding to the sample video.

[0049] In a possible implementation, the training module is used to:

[0050] Get sample labels corresponding to other sample videos;

[0051] Determine the label feature information corresponding to the other sample videos based on the sample labels corresponding to the other sample videos and the trained label feature extraction model;

[0052] The video feature extraction model to be trained is trained based on the label feature information corresponding to the other sample videos, and the label feature information and video feature information corresponding to the sample video.

[0053] In a possible implementation, the training module is used to:

[0054] Taking the label feature information corresponding to the sample video as a positive sample of the video feature information corresponding to the sample video, taking the label feature information corresponding to the other sample videos as a negative sample of the video feature information corresponding to the sample video, and calculating the loss value based on the triplet loss function, the label feature information corresponding to the other sample videos, and the label feature information and video feature information corresponding to the sample video;

[0055] Based on the loss value, the video feature extraction model to be trained is trained.

[0056] In a third aspect, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the operations performed by the training method of the video feature extraction model.

[0057] In a fourth aspect, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the instruction is loaded and executed by a processor to implement the operations performed by the training method of the video feature extraction model.

[0058] In a fifth aspect, a computer program product is provided, wherein the computer program product comprises at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the operations performed by the training method of the video feature extraction model.

[0059] The beneficial effects of the technical solution provided in the embodiments of the present application are as follows: the solution mentioned in the embodiments of the present application can determine the label feature information corresponding to the sample video based on the sample label corresponding to the sample video and the trained label feature extraction model, determine the video feature information corresponding to the sample video based on the modal information of the sample video and the video feature extraction model to be trained, and then train the video feature extraction model to be trained based on the label feature information and video feature information corresponding to the sample video. With the present application, since the label feature information can represent the semantic information of the sample label, different label feature information has semantic association. In this way, the video feature extraction model obtained after training the video feature extraction model to be trained using the label feature information of the sample video as supervision information can predict more accurate video feature information. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0061] Figure 1 It is a flow chart of a training method of a video feature extraction model provided in an embodiment of the present application;

[0062] Figure 2 is a schematic diagram of an association relationship between sample labels provided in an embodiment of the present application;

[0063] Figure 3 is a flow chart of a method for determining video feature information provided by an embodiment of the present application;

[0064] Figure 4 is a flow chart of a method for determining video feature information provided by an embodiment of the present application;

[0065] Figure 5 It is a flow chart of a training method of a video feature extraction model provided in an embodiment of the present application;

[0066] Figure 6 It is a schematic diagram of a metric before and after learning provided by an embodiment of the present application;

[0067] Figure 7 is a schematic diagram of an original video and a repeated video provided in an embodiment of the present application;

[0068] Figure 8 This is a flow chart of a rough sorting method provided in an embodiment of the present application;

[0069] Fig. 9 It is a structural schematic diagram of a training device for a video feature extraction model provided in an embodiment of the present application;

[0070] Fig.10 It is a structural block diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0071] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0072] An embodiment of the present application provides a method for training a video feature extraction model, and the server may be a single server or a server cluster consisting of multiple servers.

[0073] The server may include a processor, a memory, a communication component, etc., and the processor is connected to the memory and the communication component respectively.

[0074] The processor may be a CPU (Central Processing Unit). The processor may be used to read instructions and process data, for example, to obtain modal information of a sample video and a sample label corresponding to the sample video, to determine label feature information corresponding to the sample video, to determine video feature information corresponding to the sample video, to train a video feature extraction model to be trained based on the label feature information and video feature information corresponding to the sample video, and so on.

[0075] The memory may include ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic disk, optical data storage device, etc. The memory may be used for data storage, for example, data storage of modal information of acquired sample videos, data storage of sample labels corresponding to acquired sample videos, data storage of intermediate data generated in the process of determining label feature information corresponding to sample videos, data storage of label feature information corresponding to determined sample videos, data storage of intermediate data generated in the process of determining video feature information corresponding to sample videos, data storage of video feature information corresponding to determined sample videos, data storage of intermediate data generated in the process of training a video feature extraction model to be trained, and the like.

[0076] The communication component may be a wired network connector, a WiFi (Wireless Fidelity) module, a Bluetooth module, a cellular network communication module, etc. The communication component may be used to receive and send signals.

[0077] The label feature extraction model and the video feature extraction model in the embodiments of the present application belong to the field of ML (Machine Learning). Machine learning is a multi-disciplinary cross-disciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to improve their own performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include technologies such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and formulaic teaching systems.

[0078] The server may be a background server of an application with a video publishing function or a video playing function. The background server may train the video feature extraction model to be trained, thereby obtaining a trained video feature extraction model. When a new video is detected to be published in the application, the video may be feature extracted to obtain the video feature information corresponding to the video and store it accordingly. When the video is subsequently managed, the video feature information corresponding to each video may be applied. For example, when performing activities such as video recommendation, the video may be screened based on the video feature information corresponding to each video.

[0079] Figure 1 It is a flowchart of a method for training a video feature extraction model provided in an embodiment of the present application.

[0080] See also Figure 1 , the embodiment comprises:

[0081] 101. Obtain modal information of a sample video and a sample label corresponding to the sample video.

[0082] Description of the modal information of the sample video: The modal information may include at least one of image data, title text data, audio data, and subtitle data. The modal information of the sample video in the embodiment of the present application may be any type of modal information, or a combination of multiple modal information. For example, the modal information of the sample video may include the image data and title text data of the sample video, or include the image data and audio data of the sample video, or include the title text data and audio data of the sample video, and so on. It is understandable that the modal information of the sample video may also include other modal information not listed. The embodiment of the present application does not limit the specific content of the modal information of the sample video.

[0083] Explanation of sample labels corresponding to sample videos: When a new video is detected in the application, one or more labels will be matched to the video based on a preset algorithm or machine learning model, and the label will be stored in correspondence with the video. The sample labels corresponding to the sample video can be all matching labels corresponding to the sample video, or a part of the labels obtained after screening all labels.

[0084] The method for screening labels to obtain sample labels corresponding to sample videos can be: the staff can pre-set the percentage threshold. Obtain the percentage of the number of videos corresponding to each label, remove labels whose corresponding number of videos is less than the percentage threshold, and use labels whose corresponding number of videos is greater than or equal to the percentage threshold as sample labels corresponding to sample videos, wherein the percentage of the number of videos corresponding to the label is the ratio of the number of videos corresponding to the label in all exposed videos to the number of all exposed videos. In this way, labels with low frequency of occurrence can be removed, which are often niche and not generalizable.

[0085] In implementation, the staff can first set a sample set, which includes modal information and corresponding sample labels of multiple sample videos. When training the video feature extraction model to be trained, the modal information and corresponding sample labels of different sample videos can be randomly selected from the sample set each time to train the video feature extraction model respectively.

[0086] 102. Based on the sample label corresponding to the sample video and the trained label feature extraction model, determine the label feature information corresponding to the sample video.

[0087] In implementation, after obtaining one or more sample labels corresponding to the sample video, label feature information that aggregates the semantic information of all sample labels can be determined based on the sample labels corresponding to the sample video and a trained label feature extraction model.

[0088] Optionally, the method for determining the label feature information corresponding to the sample video may be as follows:

[0089] Each sample label is input into the trained label feature extraction model to obtain the feature information corresponding to each sample label. Based on the feature information corresponding to each sample label and the label fusion model, the label feature information corresponding to the sample video is determined.

[0090] In implementation, the label feature extraction model extracts features from each sample label respectively, thereby obtaining feature information corresponding to each sample label. After the feature information corresponding to the multiple sample labels is input into the label fusion model, the feature information corresponding to the multiple sample labels can be fused to obtain a fused feature information, which is the label feature information corresponding to the sample video. For example, the sample labels corresponding to sample video A are "fighting game", "PK", and "boxing". These three sample labels can be input into the label feature extraction model respectively to obtain feature information corresponding to "fighting game", feature information corresponding to "PK", and feature information corresponding to "boxing". Then these three feature information can be input into the label fusion model for fusion to obtain the label feature information corresponding to sample video A.

[0091] Optionally, if the multiple sample labels corresponding to the obtained sample videos are stored in the format of text sentences, instead of storing each sample label as a separate word, the sample labels corresponding to the sample videos can be first segmented to obtain one or more words after segmentation, and then the label feature extraction model is used to extract features from each word to obtain feature information corresponding to each word. The feature information corresponding to the one or more words is then fused through the label fusion model to obtain the label feature information corresponding to the sample video.

[0092] Among them, the label feature extraction model can be a word2vec (word to vector) model. Of course, it can also be other reasonable feature extraction models, which is not limited in the embodiments of the present application.

[0093] Before step 102, the label feature extraction model may be trained to obtain a trained label feature extraction model. For example, the word2vec model may be unsupervisedly trained to obtain a trained word2vec model.

[0094] The label fusion model can have multiple ways to fuse the feature information corresponding to multiple sample labels. For example, the dimensions of the feature information corresponding to each sample label can be the same. After the feature information corresponding to these multiple sample labels is input into the label fusion model, the label fusion model can perform a bitwise addition process on these multiple feature information, that is, add the values ​​of each dimension to obtain a new feature information, namely, the label feature information. Of course, there can be multiple fusion methods, which are not limited in the embodiments of the present application.

[0095] Through the above processing, the semantic information of the sample label corresponding to the sample video can be converted into feature information, that is, the label feature information with the semantic information of the sample label can be obtained, thereby establishing an association relationship between the label feature information corresponding to different sample videos. Figure 2 As shown in (a), after obtaining the label feature information, semantic information can be associated between the sample labels "Queen" and "King", and between "Women" and "Man". Figure 2 As shown in (b), after obtaining the label feature information, the sample labels "Big" and "Biggest", and "Small" and "Smallest" can be associated with each other in terms of grammatical information.

[0096] 103. Determine video feature information corresponding to the sample video based on the modal information of the sample video and the video feature extraction model to be trained.

[0097] In implementation, after the modal information of the sample video is obtained, it can be input into the video feature extraction model to be trained, so as to obtain the video feature information corresponding to the sample video output by the video feature extraction model.

[0098] 104. Based on the label feature information and video feature information corresponding to the sample video, the video feature extraction model to be trained is trained.

[0099] In the implementation, the label feature information corresponding to the sample video is used as the supervision information to train the video feature extraction model to be trained.

[0100] The sample labels are converted into label feature information so that the label feature information has the semantic information of the sample labels. In this way, the distance between the label feature information corresponding to different sample videos can be used as a metric between the sample labels corresponding to different sample videos to measure the semantic association strength between the sample labels corresponding to different sample videos. The closer the distance between the two label feature information, the stronger the semantic association strength between the sample labels corresponding to the two sample videos. After the video feature extraction model to be trained is trained using the label feature information as a supervisory signal, the semantic relationship between the sample labels can be added to the model, so that the video feature extraction model can learn not only the visual information, text information or audio information of the sample video itself that the label feature information has, but also the semantic information between the sample labels that the label feature information has, thereby improving the accuracy of the video feature extraction model.

[0101] The video feature extraction model to be trained is trained multiple times using the sample videos in the sample set until the preset end condition is reached, and then the training is stopped. There can be multiple preset end conditions, and the following are three of them:

[0102] The first method is to use different sample videos to train the video feature extraction model to be trained until the number of times reaches the training number threshold. In implementation, the staff can pre-set the training number threshold. When the training number reaches the training number threshold, the training can be stopped, and the video feature extraction model obtained after the last training is determined as the trained video feature extraction model. The training number threshold can be any reasonable value, for example, it can be 200, it can also be 300, etc., and the embodiment of the present application does not limit this.

[0103] The second type is that the loss values ​​obtained after a preset number of consecutive trainings are all less than the preset loss value threshold. The preset number and the preset loss value threshold can both be any reasonable values, for example, the preset number can be 3 or 5, etc., and the preset loss value threshold can be 0.05, etc., which is not limited in the embodiments of the present application.

[0104] The third type is that the number of training times reaches the training times threshold, and the loss values ​​obtained from the preset number of consecutive training times are all less than the preset loss value threshold.

[0105] The preset termination condition may be any one of the above three conditions, or other termination conditions, which is not limited in the embodiment of the present application.

[0106] Before the above step 101, the staff needs to set up a sample set first, which includes modal information of multiple sample videos and corresponding sample labels.

[0107] The staff can filter the videos in the application to obtain sample videos in the sample set. There are many specific video filtering methods, and several of them are listed below:

[0108] The first method is to filter videos by their exposure time, and select videos whose exposure time and current time have a time difference less than a preset time as sample videos. For example, videos that have been exposed online in the application in the past three months can be obtained as sample videos.

[0109] The second method is to filter videos by their content quality, that is, videos can be classified into different types, such as funny videos, eating videos, film and television videos, and junk videos. Then, videos of the junk type can be removed, and sample videos can be obtained from videos other than junk videos.

[0110] The third method is to filter videos by their clarity.

[0111] For example, the staff can set a resolution threshold in advance, and then during screening, the videos whose resolution is less than the resolution threshold can be eliminated, and sample videos can be obtained from multiple videos whose resolution is greater than or equal to the resolution threshold.

[0112] For another example, it is possible to detect whether a watermark exists in a video, remove the video with the watermark, and obtain a sample video from multiple videos without the watermark.

[0113] Fourthly, videos can be filtered by type ratio, so that the ratio of the number of videos of each type corresponding to the multiple sample videos included in the sample set is the same as the ratio of the number of videos of each type corresponding to the multiple videos exposed in the application, so that the data in the sample set is consistent with the real online data.

[0114] For example, all videos exposed by the application are divided into three types, namely funny videos, eating videos and film and television videos, and the corresponding video numbers account for 20%, 50% and 30% respectively. Then, when setting the sample set, among the multiple sample videos included in the sample set, the corresponding video numbers of funny videos, eating videos and film and television videos also account for 20%, 50% and 30% respectively.

[0115] The above-mentioned screening methods are merely examples. In an embodiment of the present application, the method for obtaining a sample set may be obtained by screening the videos in the application using the above-mentioned four screening methods, or by screening the videos using any combination of the above-mentioned screening methods, or by screening the videos using other screening methods. The embodiment of the present application does not limit this.

[0116] In the above step 103, different video feature extraction models can be set for different modal information. Figure 3 and Figure 4 As shown, the following takes the modal information including image data and title text data as an example to introduce the processing flow of the video feature extraction model in more detail:

[0117] The modality information includes image data and title text data. Correspondingly, the video feature extraction model may include an image feature extraction sub-model, a title feature extraction sub-model, and a modality fusion sub-model. Its processing flow may include:

[0118] 1031. Based on the image data of the sample video and the image feature extraction sub-model to be trained, determine the image feature information corresponding to the sample video.

[0119] In implementation, the image data of the sample video is input into the image feature extraction sub-model to be trained, thereby obtaining the feature information output by the image feature extraction sub-model, which is the image feature information corresponding to the sample video.

[0120] Optionally, the image data of the sample video may be a plurality of image frame data obtained after global and sparse sampling of the sample video. Correspondingly, the image feature extraction sub-model may include an image frame feature extraction module and an image fusion module.

[0121] The processing flow corresponding to the image feature extraction sub-model may be: inputting each image frame data into the image frame feature extraction module to be trained, and obtaining the image frame feature information corresponding to each image frame data. Inputting the image frame feature information corresponding to each image frame data into the image fusion module to be trained, and obtaining the image feature information corresponding to the sample video.

[0122] Among them, the image frame feature extraction module can be an ImageNet Backbone (a backbone network trained using a network image data set) model, for example, it can be an InceptionResNetV2 (Inception Residual NeuralNetwork V2, which uses a residual network to further improve the image classification level) model, a ResNet (Residual NeuralNetwork) model, an EfficientNet (Efficient Network) model, and other models. The image fusion module can be a NextVlad (Next Vector of Locally Aggregated Descriptors) model. Of course, the image frame feature extraction module and the image fusion module can also be other reasonable machine learning models, which are not limited to the embodiments of the present application.

[0123] Optionally, the image feature extraction sub-model may further include a feature enhancement module. During the processing, the feature information output by the image fusion module may be input into the feature enhancement module, thereby obtaining the feature information output by the feature enhancement module, which is the image feature information corresponding to the sample video.

[0124] Among them, the feature enhancement module can be a SENet (Squeeze-and-Excitation Network) model. Of course, it can also be other models, and the embodiments of the present application are not limited to this.

[0125] 1032. Determine title feature information corresponding to the sample video based on the title text data corresponding to the sample video and the title feature extraction sub-model to be trained.

[0126] During implementation, the title text data corresponding to the sample video is input into the title feature extraction sub-model to be trained. The title feature extraction sub-model extracts features from the title text data corresponding to the sample video, thereby obtaining feature information after feature extraction, which is the title feature information corresponding to the sample video.

[0127] Among them, the title feature extraction sub-model can be a BERT (Bidirectional Encoder Representations from Transformers) model. Of course, it can also be other text feature extraction models, which is not limited in the embodiments of the present application.

[0128] 1033. Determine video feature information corresponding to the sample video based on the modal fusion sub-model to be trained and the image feature information and title feature information corresponding to the sample video.

[0129] In implementation, after obtaining the image feature information and title feature information corresponding to the sample video, the image feature information and title feature information corresponding to the sample video can be input into the modal fusion sub-model to be trained, thereby obtaining feature information after fusion processing of the image feature information and the title feature information, which is the video feature information corresponding to the sample video.

[0130] The modal fusion sub-model may be a GMU (Gate Multimodal Unit, multi-channel gate unit) model, or a SENet model, and of course, may be other fusion models, which are not limited in the embodiments of the present application. Alternatively, the fusion method may be to directly splice the image feature information and the title feature information corresponding to the sample video head to tail, thereby obtaining a spliced ​​feature information, which is the video feature information corresponding to the sample video.

[0131] In the above step 104, in order to improve the accuracy of the video feature extraction model, the video feature extraction model to be trained can also be trained using triples, such as Figure 5 As shown, the corresponding processing can be as follows:

[0132] 1041. Obtain sample labels corresponding to other sample videos.

[0133] In implementation, other sample videos in the sample set may be randomly obtained, and optionally, other sample videos with labels different from the sample video in the sample set may be randomly obtained, and then sample labels corresponding to the other sample videos are obtained.

[0134] 1042. Based on the sample labels corresponding to the other sample videos and the trained label feature extraction model, determine the label feature information corresponding to the other sample videos.

[0135] In implementation, the method is consistent with the processing method in the above step 102, and features are extracted from sample labels corresponding to other sample videos based on the label feature extraction model, thereby obtaining feature information with semantic information of the sample labels as label feature information corresponding to other sample videos.

[0136] 1043. Based on the label feature information corresponding to other sample videos, and the label feature information and video feature information corresponding to the sample video, the video feature extraction model to be trained is trained.

[0137] In implementation, the triplet loss function can be used to calculate the loss value, and the corresponding processing can be as follows:

[0138] The label feature information corresponding to the sample video is used as a positive sample of the video feature information corresponding to the sample video, and the label feature information corresponding to other sample videos is used as a negative sample of the video feature information corresponding to the sample video. The label feature information corresponding to other sample videos, as well as the label feature information and video feature information corresponding to the sample video are input into the triple loss function to calculate the loss value. Based on the loss value, the video feature extraction model to be trained is trained.

[0139] In the embodiment of the present application, the metric learning method can be used to train the video feature extraction model to be trained, so that the final video feature information is closer to the positive sample and farther from the negative sample, such as Figure 6 As shown, the purpose of combining the voice information of the sample label with the video feature information is achieved, thereby improving the accuracy of the video feature extraction model.

[0140] After the label feature information is used as supervisory information to train the video feature extraction model to be trained, other information may be used as supervisory information to train the video feature extraction model again.

[0141] Optionally, in addition to labels, accounts can also be used as supervisory information. Usually, videos posted under the same account are more similar than videos posted under different accounts. Therefore, the account can be used as supervisory information to train the video feature extraction model again. The corresponding processing can be as follows:

[0142] The model obtained after training the video feature extraction model to be trained using the label feature information as the supervision information is called the label-trained video feature extraction model. Then a sample set is obtained, which includes multiple samples, each sample includes reference video feature information corresponding to the sample video, reference video feature information corresponding to the positive sample video, and reference video feature information corresponding to the negative sample video, wherein the reference video feature information is feature information output after inputting the modal information of the sample video into the label-trained video feature extraction model, the positive sample video and the sample video are videos released by the same account, and the negative sample video and the sample video are videos released by different accounts.

[0143] The video feature information corresponding to the positive sample video is used as the positive sample of the video feature information corresponding to the sample video, and the video feature information corresponding to the negative sample video is used as the negative sample of the video feature information corresponding to the sample video. The video feature information corresponding to the positive sample video, the video feature information corresponding to the negative sample video, and the video feature information corresponding to the sample video are input into the triple loss function to calculate the loss value. Based on the loss value, the video feature extraction model after label training is trained to obtain a trained video feature extraction model.

[0144] After obtaining the trained video feature extraction model, a test set can be set to evaluate the trained video feature extraction model. In the embodiment of the present application, the test set is used to evaluate Task 1 and Task 2 respectively, where Task 1 is a trained video feature extraction model obtained by directly using sample labels as supervisory information, and Task 2 is a trained video feature extraction model obtained by using label feature information as supervisory information in the embodiment of the present application. The evaluation results are shown in the following table:

[0145]

[0146] It can be seen from the above table that no matter which of the above evaluation indicators is used, the video feature extraction model in Task 2 is better than the video feature extraction model in Task 1. It can be seen that the training method using label feature information as supervision information in the embodiment of the present application can achieve better results.

[0147] After obtaining the trained video feature extraction model, the trained video feature extraction model can be used to extract features from each video in the application, that is, the modal information of the video is input into the trained video feature extraction model to obtain the video feature information corresponding to each video.

[0148] The video feature information can be applied to different scenarios of the application. The following takes two scenarios as examples for illustration:

[0149] Scenario 1

[0150] like Figure 7 As shown, a vector indexing tool (such as faiss) can be used to retrieve duplicate videos of the original video, that is, the similarity between the video feature information corresponding to other videos and the video feature information corresponding to the original video is calculated, and the video with a similarity greater than or equal to a preset similarity threshold is determined as a duplicate video of the original video. Then, it can be determined whether the account of each duplicate video is an associated account of the account of the original video (that is, multiple accounts registered by the same user), and the duplicate videos posted by the associated account are removed, and then the remaining duplicate videos are suppressed to reduce their traffic, thereby improving the user experience of the application.

[0151] Scenario 2

[0152] When recommending to a target account, you can roughly sort multiple videos. Rough sorting can reduce the number of videos to be recalled, reduce the pressure of subsequent fine sorting, and do not lose online results. Rough sorting can use a duration model to determine the similarity between each video and the videos that the target account is interested in.

[0153] like Figure 8 As shown, the corresponding processing can be: the duration model can adopt a dual-tower structure, which is divided into a user-side sub-model and an item-side sub-model. First, the reference video and the video to be recommended can be obtained. The reference video can be a video clicked or collected by the target account. Then, the video feature information corresponding to the reference video, the attribute information and historical behavior information of the target account, the video feature information corresponding to the video to be recommended, and the attribute information of the video to be recommended can be obtained. Among them, the attribute information of the target account may include information such as the account ID and user preferences of the target account, the historical behavior information of the target account may include information such as the videos clicked or collected by the target account recently or the tags corresponding to the videos, and the attribute information of the video to be recommended may include information such as click statistics, collection statistics, video type, etc. corresponding to the video to be recommended.

[0154] The user-side sub-model may include a first fully connected layer, and the item-side sub-model may include a second fully connected layer. The attribute information and historical behavior information of the target account may be input into the first fully connected layer to obtain output user-side feature information, and the attribute information of the video to be recommended may be input into the second fully connected layer to obtain output item-side feature information.

[0155] Then, the user-side feature information and the video feature information corresponding to the reference video may be spliced ​​to obtain the spliced ​​first feature information, and the item-side feature information and the video feature information corresponding to the video to be recommended may be spliced ​​to obtain the spliced ​​second feature information. The similarity between the first feature information and the second feature information may be calculated.

[0156] The similarities between the multiple videos to be recommended and the reference video can be obtained in the above manner, and the multiple videos to be recommended can be sorted in descending order according to the similarities, thereby completing a rough sorting.

[0157] Optionally, the user-side sub-model and the item-side sub-model may also include an upper-layer network (for example, the upper-layer network may be a network such as a fully connected layer), that is, the user-side sub-model also includes a first upper-layer network, and the item-side sub-model may also include a second upper-layer network. After obtaining the first feature information and the second feature information, the first feature information may be input into the first upper-layer network to obtain the output reference feature information, and the second feature information may be input into the second upper-layer network to obtain the output feature information to be recommended. Then, the similarity between the reference feature information and the feature information to be recommended is calculated. The similarity between multiple videos to be recommended and the reference video is calculated in the above manner, and then the multiple videos to be recommended are sorted in order of similarity from high to low, thereby completing a rough sorting.

[0158] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.

[0159] The scheme mentioned in the embodiments of the present application can determine the label feature information corresponding to the sample video based on the sample label corresponding to the sample video and the trained label feature extraction model, determine the video feature information corresponding to the sample video based on the modal information of the sample video and the video feature extraction model to be trained, and then train the video feature extraction model to be trained based on the label feature information and video feature information corresponding to the sample video. With the present application, since the label feature information can represent the semantic information of the sample label, different label feature information has semantic association. In this way, the video feature extraction model obtained after training the video feature extraction model to be trained using the label feature information of the sample video as supervision information can predict more accurate video feature information.

[0160] The present application embodiment provides a video feature extraction model training device, which can be the computer device in the above embodiment, such as Fig. 9 As shown, the device comprises:

[0161] An acquisition module 910 is used to acquire modality information of a sample video and a sample label corresponding to the sample video;

[0162] A first determination module 920, configured to determine label feature information corresponding to the sample video based on the sample label corresponding to the sample video and the trained label feature extraction model;

[0163] A second determination module 930, configured to determine video feature information corresponding to the sample video based on the modality information of the sample video and the video feature extraction model to be trained;

[0164] The training module 940 is used to train the video feature extraction model to be trained based on the label feature information and video feature information corresponding to the sample video.

[0165] In a possible implementation, the modality information includes at least one of image data, title text data, audio data, and subtitle data.

[0166] In a possible implementation, the first determining module 920 is configured to:

[0167] Input each sample label into the trained label feature extraction model to obtain feature information corresponding to each sample label;

[0168] Based on the feature information corresponding to each sample label and the label fusion model, the label feature information corresponding to the sample video is determined.

[0169] In a possible implementation, the modality information includes image data and title text data, and the video feature extraction model includes an image feature extraction sub-model, a title feature extraction sub-model and a modality fusion sub-model;

[0170] The second determining module 930 is configured to:

[0171] Determining image feature information corresponding to the sample video based on the image data of the sample video and the image feature extraction sub-model to be trained;

[0172] Determine the title feature information corresponding to the sample video based on the title text data corresponding to the sample video and the title feature extraction sub-model to be trained;

[0173] Based on the modality fusion sub-model to be trained, and the image feature information and title feature information corresponding to the sample video, the video feature information corresponding to the sample video is determined.

[0174] In a possible implementation, the image data of the sample video includes a plurality of image frame data, and the image feature extraction sub-model includes an image frame feature extraction module and an image fusion module;

[0175] The second determining module 930 is configured to:

[0176] Input each image frame data into the image frame feature extraction module to be trained to obtain the image frame feature information corresponding to each image frame data;

[0177] The image frame feature information corresponding to each image frame data is input into the image fusion module to be trained to obtain the image feature information corresponding to the sample video.

[0178] In a possible implementation, the training module 940 is used to:

[0179] Get sample labels corresponding to other sample videos;

[0180] Determine the label feature information corresponding to the other sample videos based on the sample labels corresponding to the other sample videos and the trained label feature extraction model;

[0181] The video feature extraction model to be trained is trained based on the label feature information corresponding to the other sample videos, and the label feature information and video feature information corresponding to the sample video.

[0182] In a possible implementation, the training module 940 is used to:

[0183] Taking the label feature information corresponding to the sample video as a positive sample of the video feature information corresponding to the sample video, taking the label feature information corresponding to the other sample videos as a negative sample of the video feature information corresponding to the sample video, and calculating the loss value based on the triplet loss function, the label feature information corresponding to the other sample videos, and the label feature information and video feature information corresponding to the sample video;

[0184] Based on the loss value, the video feature extraction model to be trained is trained.

[0185] It should be noted that: when the training device for the video feature extraction model provided in the above embodiment is training the video feature extraction model, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the training device for the video feature extraction model provided in the above embodiment and the training method embodiment of the video feature extraction model belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0186] Fig.101 is a schematic diagram of the structure of a server provided in an embodiment of the present application. The server 1000 may have relatively large differences due to different configurations or performances, and may include one or more central processors 1001 and one or more memories 1002, wherein the memory 1002 stores at least one instruction, and the at least one instruction is loaded and executed by the processor 1001 to implement the methods provided in the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output, and the server may also include other components for implementing device functions, which will not be described in detail here.

[0187] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions, and the instructions can be executed by a processor in a terminal to complete the training method of the video feature extraction model in the above embodiment. The computer-readable storage medium can be non-transitory. For example, the computer-readable storage medium can be a ROM (read-only memory), a RAM (random access memory), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0188] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0189] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals (including but not limited to signals transmitted between user terminals and other devices, etc.) involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions. For example, the modal information involved in this application is obtained with full authorization.

[0190] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A training method for a video feature extraction model, It is characterized in that The method comprises: Obtaining modal information of a sample video and a sample label corresponding to the sample video; Based on the sample label corresponding to the sample video and the trained label feature extraction model, determine the label feature information corresponding to the sample video, where the label feature information corresponding to the sample video aggregates the semantic information of the sample label corresponding to the sample video; Determining video feature information corresponding to the sample video based on the modal information of the sample video and the video feature extraction model to be trained; Based on the label feature information and video feature information corresponding to the sample video, the video feature extraction model to be trained is trained to obtain a trained video feature extraction model, wherein the trained video feature extraction model is trained using the label feature information corresponding to the sample video as supervision information.

2. The method according to claim 1, It is characterized in that The modality information includes at least one of image data, title text data, audio data, and subtitle data.

3. The method according to claim 1, It is characterized in that The modality information includes image data and title text data, and the video feature extraction model includes an image feature extraction sub-model, a title feature extraction sub-model and a modality fusion sub-model; The determining, based on the modal information of the sample video and the video feature extraction model to be trained, the video feature information corresponding to the sample video includes: Determining image feature information corresponding to the sample video based on the image data of the sample video and the image feature extraction sub-model to be trained; Determine the title feature information corresponding to the sample video based on the title text data corresponding to the sample video and the title feature extraction sub-model to be trained; Based on the modality fusion sub-model to be trained, and the image feature information and title feature information corresponding to the sample video, the video feature information corresponding to the sample video is determined.

4. The method according to claim 3, It is characterized in that The image data of the sample video includes a plurality of image frame data, and the image feature extraction sub-model includes an image frame feature extraction module and an image fusion module; The determining, based on the image data of the sample video and the image feature extraction sub-model to be trained, the image feature information corresponding to the sample video includes: Input each image frame data into the image frame feature extraction module to be trained to obtain the image frame feature information corresponding to each image frame data; The image frame feature information corresponding to each image frame data is input into the image fusion module to be trained to obtain the image feature information corresponding to the sample video.

5. The method according to claim 1, It is characterized in that The step of training the video feature extraction model to be trained based on the label feature information and the video feature information corresponding to the sample video includes: Get sample labels corresponding to other sample videos; Determine label feature information corresponding to the other sample videos based on the sample labels corresponding to the other sample videos and the trained label feature extraction model; The video feature extraction model to be trained is trained based on the label feature information corresponding to the other sample videos, and the label feature information and video feature information corresponding to the sample video.

6. The method according to claim 5, It is characterized in that The training of the video feature extraction model to be trained based on the label feature information corresponding to the other sample videos, and the label feature information and video feature information corresponding to the sample video includes: Taking the label feature information corresponding to the sample video as a positive sample of the video feature information corresponding to the sample video, taking the label feature information corresponding to the other sample videos as a negative sample of the video feature information corresponding to the sample video, and calculating the loss value based on the triplet loss function, the label feature information corresponding to the other sample videos, and the label feature information and video feature information corresponding to the sample video; Based on the loss value, the video feature extraction model to be trained is trained.

7. A training device for a video feature extraction model, It is characterized in that The device comprises: An acquisition module, used to acquire modality information of a sample video and a sample label corresponding to the sample video; A first determination module is used to determine label feature information corresponding to the sample video based on the sample label corresponding to the sample video and the trained label feature extraction model, wherein the label feature information corresponding to the sample video aggregates semantic information of the sample label corresponding to the sample video; A second determination module, configured to determine video feature information corresponding to the sample video based on the modality information of the sample video and a video feature extraction model to be trained; A training module is used to train the video feature extraction model to be trained based on the label feature information and video feature information corresponding to the sample video to obtain a trained video feature extraction model. The trained video feature extraction model is trained using the label feature information corresponding to the sample video as supervision information.

8. The device according to claim 7, It is characterized in that The modality information includes at least one of image data, title text data, audio data, and subtitle data.

9. The device according to claim 7, It is characterized in that The modality information includes image data and title text data, and the video feature extraction model includes an image feature extraction sub-model, a title feature extraction sub-model and a modality fusion sub-model; The second determining module is used to: Determining image feature information corresponding to the sample video based on the image data of the sample video and the image feature extraction sub-model to be trained; Determine the title feature information corresponding to the sample video based on the title text data corresponding to the sample video and the title feature extraction sub-model to be trained; Based on the modality fusion sub-model to be trained, and the image feature information and title feature information corresponding to the sample video, the video feature information corresponding to the sample video is determined.

10. The device according to claim 9, It is characterized in that The image data of the sample video includes a plurality of image frame data, and the image feature extraction sub-model includes an image frame feature extraction module and an image fusion module; The second determining module is used to: Input each image frame data into the image frame feature extraction module to be trained to obtain the image frame feature information corresponding to each image frame data; The image frame feature information corresponding to each image frame data is input into the image fusion module to be trained to obtain the image feature information corresponding to the sample video.

11. The device according to claim 7, It is characterized in that The training module is used to: Get sample labels corresponding to other sample videos; Determine the label feature information corresponding to the other sample videos based on the sample labels corresponding to the other sample videos and the trained label feature extraction model; The video feature extraction model to be trained is trained based on the label feature information corresponding to the other sample videos, and the label feature information and video feature information corresponding to the sample video.

12. The device according to claim 11, It is characterized in that The training module is used to: Taking the label feature information corresponding to the sample video as a positive sample of the video feature information corresponding to the sample video, taking the label feature information corresponding to the other sample videos as a negative sample of the video feature information corresponding to the sample video, and calculating the loss value based on the triplet loss function, the label feature information corresponding to the other sample videos, and the label feature information and video feature information corresponding to the sample video; Based on the loss value, the video feature extraction model to be trained is trained.

13. A computer device, It is characterized in that The computer device includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the operations performed by the training method of the video feature extraction model as described in any one of claims 1 to 6.

14. A computer-readable storage medium, It is characterized in that The storage medium stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the operations performed by the training method of the video feature extraction model as described in any one of claims 1 to 6.

15. A computer program product, It is characterized in that The computer program product includes at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the operations performed by the training method of the video feature extraction model as described in any one of claims 1 to 6.

Citation Information

Patent Citations

  • Video retrieval method and device

    CN111581437A

  • Lable feature near-duplicated video detection method based on convolutional neural network semantic classification

    CN111723692A

  • Video content evaluation method and device, storage medium and computer equipment

    CN111741330A

  • Model generation method, video screening method, related device and storage medium

    CN113392864A

  • Video feature extraction method and device, equipment and storage medium

    CN113515994A