Video processing method, device, equipment, medium and computer program product

Abstract

The present application relates to a video processing method, device, equipment, medium and computer program product, and belongs to the field of network media technology. The method includes: obtaining a set of sample video pairs; each sample video in the sample video pair carries a corresponding reference video label; obtaining the video features corresponding to each sample video through the label prediction model to be trained, and predicting the predicted video label corresponding to the sample video based on the video features; determining a first loss value based on the difference between the predicted video label and the reference video label corresponding to the same sample video; determining the video content similarity between each sample video, and the text similarity between the reference video labels of each sample video; determining a second loss value based on the video content similarity and the text similarity, and training the label prediction model based on the first loss value and the second loss value; the trained label prediction model is used to predict the label of the target video. This method can improve the efficiency of video labeling.

Description

Technical Field

[0001] The present application relates to artificial intelligence technology, and more particularly to the field of network media technology, and in particular to a video processing method, device, equipment, medium and computer program product. Background Art

[0002] With the development of computer technology, network media technology has emerged. Network media relies on the technology and equipment provided by Internet device developers to transmit, store and process video signals. In order to better understand and manage videos, video tags are important video content portrait features. Related video products, such as video recommendation and video retrieval, usually maintain a dedicated annotation team to annotate the newly generated videos every day. However, manual video tagging takes a lot of time and the speed of video tagging is slow, resulting in low video tagging efficiency. Summary of the invention

[0003] Based on this, it is necessary to provide a video processing method, device, equipment, medium and computer program product that can improve the efficiency of video tagging in response to the above technical problems.

[0004] A video processing method, the method comprising:

[0005] Obtain a set of sample video pairs; each sample video in the sample video pair carries a corresponding reference video tag;

[0006] Obtaining video features corresponding to each sample video in the sample video pair through the label prediction model to be trained, and predicting a predicted video label corresponding to the sample video based on the video features;

[0007] Determining a first loss value based on a difference between the predicted video label and the reference video label corresponding to the same sample video;

[0008] For each sample video in each sample video pair, determining the video content similarity between the sample videos and the text similarity between the reference video tags of the sample videos;

[0009] Determining a second loss value according to the video content similarity and the text similarity; the second loss value is used to characterize the label similarity relationship between the reference video labels of each sample video;

[0010] According to the first loss value and the second loss value, the label prediction model to be trained is trained to obtain a trained label prediction model; the trained label prediction model is used to perform label prediction on the target video.

[0011] A video processing device, comprising:

[0012] An acquisition module, used to acquire a set of sample video pairs; each sample video in the sample video pair carries a corresponding reference video tag;

[0013] A prediction module, used to obtain video features corresponding to each sample video in the sample video pair through a label prediction model to be trained, and predict a predicted video label corresponding to the sample video based on the video features;

[0014] A determination module, configured to determine a first loss value based on the difference between the predicted video label and the reference video label corresponding to the same sample video; determine, for each sample video in each sample video pair, the video content similarity between the sample videos and the text similarity between the reference video labels of the sample videos; determine a second loss value based on the video content similarity and the text similarity; the second loss value is used to characterize the label similarity relationship between the reference video labels of each sample video;

[0015] A training module is used to train the label prediction model to be trained according to the first loss value and the second loss value to obtain a trained label prediction model; the trained label prediction model is used to perform label prediction on the target video.

[0016] In one embodiment, the prediction module is also used to extract features of each video frame in each sample video of each sample video pair through a label prediction model to be trained to obtain image features corresponding to each video frame; and perform feature fusion on the image features corresponding to each video frame to obtain video features corresponding to each sample video in the sample video pair.

[0017] In one embodiment, the prediction module is also used to use the label prediction model to be trained to take each video frame in the sample video as the current video frame, move part of the feature channels corresponding to the current video frame to the adjacent video frame along the time dimension to exchange features with the adjacent video frame to obtain adjacent exchange features; obtain the unexchanged features in the current video frame to obtain retained features; combine the retained features and the adjacent exchange features to generate image features corresponding to the current video frame.

[0018] In one embodiment, the prediction module is also used to move part of the feature channels corresponding to the current video frame to the adjacent video frame along the time dimension, so as to obtain features corresponding to the feature channels moved to the adjacent video frame from the adjacent video frame to obtain adjacent exchange features; and obtain features located in the remaining feature channels in the current video frame to obtain retained features.

[0019] In one embodiment, the prediction module is also used to perform feature clustering on the image features corresponding to the video frames; for each cluster, determine the distance from each image feature in the cluster to the cluster center, and determine the probability value of each image feature in the cluster belonging to the cluster center; based on the distance from each image feature to the cluster center and the probability value of each image feature belonging to the cluster center, determine the features corresponding to the cluster; fuse the features corresponding to the clusters to obtain the video features corresponding to each sample video in the sample video pair.

[0020] In one embodiment, the determination module is also used to obtain offline video features of each sample video in each sample video pair, and obtain label features of the reference video labels corresponding to each sample video; perform similarity calculation on the offline video features of each sample video to obtain the video content similarity between the sample videos; perform similarity calculation on the label features corresponding to each sample video to obtain the text similarity between the reference video labels of each sample video.

[0021] In one embodiment, each sample video corresponds to a plurality of label feature vectors respectively; the determination module is also used to determine, for each sample video, a comprehensive label feature vector corresponding to the sample video according to the plurality of label feature vectors corresponding to the sample video; and determine the cosine similarity between the comprehensive label feature vectors corresponding to each sample video to obtain text similarity.

[0022] In one embodiment, the determination module is also used to determine the text difference value between the reference video labels corresponding to each sample video in the sample video pair based on the text similarity; use the video content similarity as the label similarity relationship coefficient to perform weighted adjustment on the text difference value to obtain a second loss value.

[0023] In one embodiment, the label prediction model includes a video feature extraction network and a label prediction network; the prediction module is also used to extract features from each sample video in the sample video pair through the video feature extraction network to obtain video features corresponding to each sample video in the sample video pair; and perform label prediction based on the video features through the label prediction network to obtain a predicted video label corresponding to the sample video.

[0024] In one embodiment, the training module is also used to determine a target loss value based on the first loss value and the second loss value; and adjust the network parameters of the video feature extraction network and the label prediction network in the direction of reducing the target loss value to train and obtain the trained label prediction model.

[0025] In one embodiment, the video feature extraction network includes an image feature extraction unit and an image feature fusion unit; the prediction module is also used to extract the image features corresponding to each video frame in the sample video for each sample video in each sample video pair through the image feature extraction unit; and perform feature fusion on the image features corresponding to each video frame in the sample video through the image feature fusion unit to obtain the video features corresponding to each sample video in the sample video pair.

[0026] In one embodiment, the apparatus further comprises:

[0027] The recommendation module is used to obtain a target video; predict the label of the target video through the trained label prediction model to obtain the target video label to which the target video belongs; search for videos carrying the target video label; and recommend the found videos carrying the target video label.

[0028] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0029] Obtain a set of sample video pairs; each sample video in the sample video pair carries a corresponding reference video tag;

[0030] Obtaining video features corresponding to each sample video in the sample video pair through the label prediction model to be trained, and predicting a predicted video label corresponding to the sample video based on the video features;

[0031] Determining a first loss value based on a difference between the predicted video label and the reference video label corresponding to the same sample video;

[0032] For each sample video in each sample video pair, determining the video content similarity between the sample videos and the text similarity between the reference video tags of the sample videos;

[0033] Determining a second loss value according to the video content similarity and the text similarity; the second loss value is used to characterize the label similarity relationship between the reference video labels of each sample video;

[0034] According to the first loss value and the second loss value, the label prediction model to be trained is trained to obtain a trained label prediction model; the trained label prediction model is used to perform label prediction on the target video.

[0035] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0036] Obtain a set of sample video pairs; each sample video in the sample video pair carries a corresponding reference video tag;

[0037] Obtaining video features corresponding to each sample video in the sample video pair through the label prediction model to be trained, and predicting a predicted video label corresponding to the sample video based on the video features;

[0038] Determining a first loss value based on a difference between the predicted video label and the reference video label corresponding to the same sample video;

[0039] For each sample video in each sample video pair, determining the video content similarity between the sample videos and the text similarity between the reference video tags of the sample videos;

[0040] Determining a second loss value according to the video content similarity and the text similarity; the second loss value is used to characterize the label similarity relationship between the reference video labels of each sample video;

[0041] According to the first loss value and the second loss value, the label prediction model to be trained is trained to obtain a trained label prediction model; the trained label prediction model is used to perform label prediction on the target video.

[0042] A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the following steps are implemented:

[0043] Obtain a set of sample video pairs; each sample video in the sample video pair carries a corresponding reference video tag;

[0044] Obtaining video features corresponding to each sample video in the sample video pair through the label prediction model to be trained, and predicting a predicted video label corresponding to the sample video based on the video features;

[0045] Determining a first loss value based on a difference between the predicted video label and the reference video label corresponding to the same sample video;

[0046] For each sample video in each sample video pair, determining the video content similarity between the sample videos and the text similarity between the reference video tags of the sample videos;

[0047] Determining a second loss value according to the video content similarity and the text similarity; the second loss value is used to characterize the label similarity relationship between the reference video labels of each sample video;

[0048] According to the first loss value and the second loss value, the label prediction model to be trained is trained to obtain a trained label prediction model; the trained label prediction model is used to perform label prediction on the target video.

[0049] The above-mentioned video processing method, device, equipment, medium and computer program product obtain a set of sample video pairs, each sample video in the sample video pair carries a corresponding reference video label, and through the label prediction model to be trained, the video features corresponding to each sample video in the sample video pair can be extracted, and the predicted video label corresponding to the sample video can be predicted based on the video features. Based on the difference between the predicted video label and the reference video label corresponding to the same sample video, a first loss value can be determined, and for each sample video in each sample video pair, the video content similarity between each sample video and the text similarity between the reference video labels of each sample video are determined. According to the video content similarity and the text similarity, a second loss value for characterizing the label similarity relationship between the reference video labels of each sample video can be determined. According to the first loss value and the second loss value, the label prediction model to be trained can be iteratively trained to obtain a trained label prediction model, and the trained label prediction model can be directly used to predict the label of the target video to be labeled, so as to realize automatic labeling of the label of the target video. Compared with the traditional manual labeling method, the present application directly uses the label prediction model trained based on the first loss value and the second loss value to automatically label the video to be labeled, which can greatly shorten the labeling time and improve the efficiency of video labeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A diagram showing an application environment of a video processing method in an embodiment;

[0051] Figure 2 is a schematic flow chart of a video processing method in one embodiment;

[0052] Figure 3 A schematic diagram of feature exchange in one embodiment;

[0053] Figure 4 A schematic diagram of displaying a video tag in one embodiment;

[0054] Figure 5 A schematic diagram of a search interface for performing a video search based on a keyword in one embodiment;

[0055] Figure 6 A schematic diagram of the architecture of a label prediction model in one embodiment;

[0056] Figure 7 is a flow chart of a video processing method in another embodiment;

[0057] Figure 8 is a structural block diagram of a video processing device in one embodiment;

[0058] Fig. 9 is a structural block diagram of a video processing device in another embodiment;

[0059] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0061] The video processing method provided in this application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The terminal may include a mobile phone, a tablet computer, a laptop computer, a smart TV, a car terminal, etc. It is understandable that a client may be provided on the terminal, such as a video client, a browser client, an information stream client, etc. The server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0062] The server 104 can obtain a set of sample video pairs, each sample video in the sample video pair carries a corresponding reference video tag, obtain the video features corresponding to each sample video in the sample video pair through the tag prediction model to be trained, and predict the predicted video tag corresponding to the sample video based on the video features. The server 104 can determine the first loss value based on the difference between the predicted video tag and the reference video tag corresponding to the same sample video. For each sample video in each sample video pair, the server 104 can determine the video content similarity between each sample video and the text similarity between the reference video tags of each sample video, and determine the second loss value based on the video content similarity and the text similarity. The second loss value is used to characterize the tag similarity relationship between the reference video tags of each sample video. The server 104 can train the tag prediction model to be trained based on the first loss value and the second loss value to obtain a trained tag prediction model; the trained tag prediction model is used to perform tag prediction on the target video. It can be understood that the terminal 102 can upload the target video, and the server 104 can perform tag prediction on the uploaded target video. The terminal 102 may also upload a sample video, and the server 104 may train the tag prediction model to be trained. This embodiment does not limit this, and it is understood that Figure 1 The application scenarios are only for illustration and are not limited to this.

[0063] It should be noted that the video processing method in some embodiments of the present application uses artificial intelligence (AI). For example, the video features corresponding to each sample video in the sample video pair are features extracted using artificial intelligence technology, and the predicted video labels corresponding to the sample videos are also video labels predicted using artificial intelligence technology.

[0064] In one embodiment, Figure 2 As shown, a video processing method is provided, which can be applied to a computer device, which can be a terminal or a server, and can be executed by the terminal or the server itself alone, or can be implemented through interaction between the terminal and the server. This embodiment includes the following steps:

[0065] Step 202, obtaining a set of sample video pairs; each sample video in the sample video pair carries a corresponding reference video tag.

[0066] Among them, a video pair refers to a pair of videos. It can be understood that a video pair includes two videos, for example, a video pair can include video A and video B. A sample video pair refers to a pair of sample videos. It can be understood that a sample video pair includes two sample videos. Sample videos are videos used as training data. It can be understood that sample videos can be used to train label prediction models. The label prediction model is a neural network model used to predict video labels. Video labels are text used to describe the content of a video. For example, if a video includes the video label "food", it means that the content of the video is related to food. Reference video labels are video labels marked for sample videos.

[0067] In one embodiment, a set of sample video pairs is pre-stored in the storage device. The computer device can communicate with the storage device and directly obtain the set of sample video pairs from the storage device.

[0068] In one embodiment, the computer device may obtain multiple sample videos, and generate a set of sample video pairs based on the obtained multiple sample videos. It is understood that for each sample video in the multiple sample videos, the computer device may determine the sample video and any other sample video in the multiple sample videos as a sample video pair. Furthermore, the computer device may generate a set of sample video pairs based on each sample video pair.

[0069] In one embodiment, the sample video pairs include positive sample video pairs and negative sample video pairs. The positive sample video pairs include sample video pairs with similar reference video labels, and the negative sample video pairs include sample video pairs with less similar reference video labels.

[0070] Step 204: obtain the video features corresponding to each sample video in the sample video pair through the label prediction model to be trained, and predict the predicted video label corresponding to the sample video based on the video features.

[0071] Among them, video features are features used to describe the semantic content of the video. It should be noted that the video features corresponding to the sample video are online video features. It can be understood that online video features refer to video features that can change during the training process of the label prediction model. The acquisition of video features corresponding to the sample video is closely related to the training process of the label prediction model, and the training process of the label prediction model can cause the video features corresponding to the sample video to change. The predicted video label corresponding to the sample video is the video feature predicted by the label prediction model to be trained based on the video features.

[0072] In one embodiment, a computer device may input each sample video in a sample video pair into a label prediction model to be trained, so as to perform feature extraction on the sample video through the label prediction model to be trained, obtain video features corresponding to each sample video in the sample video pair, and predict a predicted video label corresponding to the sample video based on the video features.

[0073] In one embodiment, the computer device may directly obtain the video features corresponding to each sample video in the sample video pair. Furthermore, the computer device may input the video features corresponding to each sample video in the sample video pair into the label prediction model to be trained, so as to predict the predicted video label corresponding to the sample video based on the video features through the label prediction model to be trained.

[0074] Step 206: Determine a first loss value based on the difference between the predicted video label and the reference video label corresponding to the same sample video.

[0075] The first loss value is a loss value determined based on the difference between the predicted video label and the reference video label corresponding to the same sample video.

[0076] Specifically, for the same sample video, the computer device may compare the predicted video label corresponding to the sample video with the reference video label corresponding to the sample video to obtain the difference between the predicted video label corresponding to the sample video and the reference video label. The computer device may perform loss calculation based on the difference between the predicted video label corresponding to the sample video and the reference video label to obtain a first loss value.

[0077] In one embodiment, the computer device may directly use the difference value corresponding to the difference between the predicted video label corresponding to the sample video and the reference video label as the first loss value.

[0078] Step 208 : for each sample video in each sample video pair, determine the video content similarity between the sample videos and the text similarity between the reference video tags of the sample videos.

[0079] Among them, video content similarity is the similarity between the video contents of two sample videos in a sample video pair. It can be understood that the more similar the video contents of the two sample videos in a sample video pair are, the higher the corresponding video content similarity is, and vice versa. Text similarity is the similarity between the reference video tags corresponding to the two sample videos in a sample video pair. It can be understood that the more similar the reference video tags corresponding to the two sample videos in a sample video pair are, the higher the corresponding text similarity is, and vice versa.

[0080] In one embodiment, each sample video pair includes two sample videos. For each sample video pair, the computer device may perform similarity calculation on the two sample videos included in the sample video pair to obtain the similarity between the video contents of the two sample videos. Also, for each sample video pair, the computer device may perform similarity calculation on the reference video tags corresponding to the two sample videos included in the sample video pair to obtain the similarity between the reference video tags corresponding to the two sample videos.

[0081] Step 210, determining a second loss value according to the video content similarity and the text similarity; the second loss value is used to characterize the label similarity relationship between the reference video labels of each sample video.

[0082] The second loss value is a loss value determined based on the video content similarity and the text similarity. It can be understood that each sample video pair includes two sample videos, and for each sample video pair, the label similarity relationship is used to describe whether the reference video labels corresponding to the two sample videos included in the sample video pair are similar.

[0083] In one embodiment, the computer device can calculate the second loss value based on the video content similarity and the text similarity. It can be understood that for the two sample videos in each sample video pair, the more similar the video contents of the two sample videos are, the more similar the text contents of the reference video tags corresponding to the two sample videos are. Based on this rule, the computer device can calculate the second loss value based on the video content similarity and the text similarity.

[0084] Step 212: train the label prediction model to be trained according to the first loss value and the second loss value to obtain a trained label prediction model; the trained label prediction model is used to perform label prediction on the target video.

[0085] The target video is a video to be labeled as a target. It can be understood that the target video does not carry a corresponding video label and is a video that needs label prediction.

[0086] Specifically, the computer device can iteratively train the label prediction model to be trained based on the first loss value and the second loss value until the iteration stop condition is reached, and a trained label prediction model is obtained. The computer device performs label prediction on the target video based on the trained label prediction model. It can be understood that when training the label prediction model, the second loss value used to characterize the label similarity relationship between the reference video labels of each sample video is taken into account, so that the model to be trained can know whether the two reference video labels are similar, so that the model can be trained without confusion about similar reference video labels. It can be understood that the traditional method will give up label prediction when encountering similar labels because it is confused about the labels, while the label prediction model trained by the method of the present application can no longer be confused about similar video labels, and thus can accurately predict video labels.

[0087] In one embodiment, the computer device may iteratively train the label prediction model to be trained in a direction to reduce the first loss value and the second loss value to adjust the model parameters of the label prediction model to obtain a trained label prediction model.

[0088] In one embodiment, the iteration stopping condition may specifically be that the number of iterations reaches a preset number threshold, or that both the first loss value and the second loss value are less than a preset loss value.

[0089] In the above video processing method, a set of sample video pairs is obtained, and each sample video in the sample video pair carries a corresponding reference video label. Through the label prediction model to be trained, the video features corresponding to each sample video in the sample video pair can be extracted, and the predicted video label corresponding to the sample video can be predicted based on the video features. Based on the difference between the predicted video label and the reference video label corresponding to the same sample video, a first loss value can be determined, and for each sample video in each sample video pair, the video content similarity between each sample video and the text similarity between the reference video labels of each sample video are determined. According to the video content similarity and the text similarity, a second loss value for characterizing the label similarity relationship between the reference video labels of each sample video can be determined. According to the first loss value and the second loss value, the label prediction model to be trained can be iteratively trained to obtain a trained label prediction model. The trained label prediction model can be directly used to predict the label of the target video to be labeled, so as to realize automatic labeling of the label of the target video. Compared with the traditional manual labeling method, the present application directly labels the video to be labeled automatically through the label prediction model trained based on the first loss value and the second loss value, which can greatly shorten the labeling time, improve the efficiency of video labeling, and ensure the timely release of the video. At the same time, the labor cost is also reduced by automatically labeling the video to be labeled. In addition, by automatically labeling the video to be labeled through the trained label prediction model, the interactive operations between the user and the computer device can be reduced, and the repeated response of the computer device to the interactive operations can be avoided, thereby avoiding the waste of computer resources.

[0090] Moreover, compared with the traditional manual labeling method, this application automatically labels videos with the same content through a trained label prediction model, which can avoid labeling different video labels with the same meaning but different expressions, thereby improving the uniformity of the labels.

[0091] Compared with the traditional label prediction method based on only a single video, the present application trains the label prediction model to be trained through sample videos, which can explore the relationship between the reference video labels of richer sample videos, so that the trained label prediction model has a higher label prediction accuracy.

[0092] In addition, compared with the traditional label prediction method that processes different video labels independently, the present application can determine the label similarity relationship between the reference video labels of each sample video through video content similarity and text similarity, fully considering the relationship between the reference video labels, improving the model training effect, and thus further improving the video label prediction accuracy.

[0093] In one embodiment, video features corresponding to each sample video in a sample video pair are obtained through a label prediction model to be trained, including: for each sample video in each sample video pair, feature extraction is performed on each video frame in the sample video through the label prediction model to be trained to obtain image features corresponding to each video frame; feature fusion is performed on the image features corresponding to each video frame to obtain video features corresponding to each sample video in the sample video pair.

[0094] The video frame refers to each frame of the video image. It can be understood that a sample video includes multiple frames of video images. The image feature is a feature used to describe the image content of the video frame.

[0095] Specifically, for each sample video in each sample video pair, the computer device may input each sample video in each sample video pair into the label prediction model to be trained. Furthermore, the computer device may extract features of each video frame in the input sample video through the label prediction model to be trained to obtain image features corresponding to each video frame. The computer device may perform feature fusion on the image features corresponding to each video frame to obtain video features corresponding to each sample video in the sample video pair.

[0096] In one embodiment, for each video frame in a sample video, the computer device may obtain features of the video frame and partial features of video frames adjacent to the video frame. The computer device may generate image features corresponding to the video frame based on the features of the video frame and partial features of video frames adjacent to the video frame.

[0097] In one embodiment, for each video frame in the sample video, the computer device may obtain the feature of the video frame, and further, the computer device may directly use the feature of the video frame as the image feature corresponding to the video frame.

[0098] In one embodiment, the computer device may determine the fusion weights of the image features corresponding to each video frame, and based on the determined fusion weights, perform feature fusion on the image features corresponding to each video frame to obtain the video features corresponding to each sample video in the sample video pair.

[0099] In the above embodiment, by performing feature extraction on each video frame in the sample video, rich image features corresponding to each video frame can be obtained, and by performing feature fusion on the rich image features corresponding to each video frame, video features corresponding to each sample video in the sample video pair can be obtained, thereby improving the accuracy and richness of video feature extraction.

[0100] In one embodiment, the computer device may input each sample video in the sample video pair into the label prediction model to be trained, and directly extract video features of the sample video through the label prediction model to be trained to obtain the video features corresponding to each sample video in the sample video pair.

[0101] In one embodiment, feature extraction is performed on each video frame in a sample video through a label prediction model to be trained to obtain image features corresponding to each video frame, including: using the label prediction model to be trained, taking each video frame in the sample video as a current video frame, moving part of the feature channels corresponding to the current video frame to an adjacent video frame along a time dimension to exchange features with the adjacent video frame to obtain adjacent exchange features; obtaining unexchanged features in the current video frame to obtain retained features; and combining the retained features with the adjacent exchange features to generate image features corresponding to the current video frame.

[0102] Among them, the current video frame refers to the video frame being processed among the video frames in the sample video. The feature channel is the channel for generating the features of the video frame. The adjacent video frame is the previous frame and / or the next frame of the video frame adjacent to the current video frame. Feature exchange means that the current video frame learns some features of the adjacent video frame from the adjacent video frame, and the adjacent video frame also learns some features of the current video frame from the current video frame. Adjacent exchange features are the features of the current video frame exchanged with the adjacent video frame, and the features of the adjacent video frame are learned from the adjacent video frame. The retained features are the retained features of the current video frame that have not been exchanged with the adjacent video frames.

[0103] Specifically, the computer device can use the label prediction model to be trained to use each video frame in the sample video as the current video frame. The computer device can determine the adjacent video frame for feature exchange with the current video frame, and determine the partial feature channels corresponding to the current video frame that moves during the feature exchange process, and move the partial feature channels corresponding to the current video frame to the adjacent video frame along the time dimension to exchange features with the adjacent video frame to obtain adjacent exchange features. The computer device can obtain the unexchanged features in the current video frame to obtain the retained features, and combine the retained features with the adjacent exchange features to generate the image features corresponding to the current video frame.

[0104] In the above embodiment, by moving part of the feature channels corresponding to the current video frame to the adjacent video frame along the time dimension, feature exchange with the adjacent video frame can be achieved to obtain adjacent exchange features, and the unexchanged retained features and adjacent exchange features in the current video frame are combined to generate image features corresponding to the current video frame. Since the image features corresponding to the current video frame not only consider the features of the current video frame itself, but also learn the features of the adjacent frames through feature exchange, the feature richness of the image features corresponding to the current video frame is improved.

[0105] In one embodiment, moving part of the feature channels corresponding to the current video frame to the adjacent video frame along the time dimension to exchange features with the adjacent video frame to obtain adjacent exchange features includes: moving part of the feature channels corresponding to the current video frame to the adjacent video frame along the time dimension to obtain features corresponding to the feature channels moved to the adjacent video frame from the adjacent video frame to obtain adjacent exchange features. Obtaining unexchanged features in the current video frame to obtain retained features includes: obtaining features in the remaining feature channels in the current video frame to obtain retained features.

[0106] Specifically, the computer device may determine the adjacent video frame for feature exchange with the current video frame, and determine the partial feature channels corresponding to the current video frame that moves during the feature exchange process. The computer device may move the partial feature channels corresponding to the current video frame to the adjacent video frame along the time dimension to obtain features corresponding to the feature channels moved to the adjacent video frame from the adjacent video frame to obtain adjacent exchange features. The computer device may obtain features in the remaining feature channels in the current video frame to obtain retained features, and combine the retained features with the adjacent exchange features to generate image features corresponding to the current video frame.

[0107] In the above embodiment, by moving part of the feature channels corresponding to the current video frame to the adjacent video frame along the time dimension, features corresponding to the feature channels moved to the adjacent video frame can be obtained from the adjacent video frame, thereby quickly and accurately obtaining adjacent exchange features, thereby improving the efficiency and accuracy of obtaining adjacent interaction features.

[0108] In one embodiment, Figure 3 As shown, each small square represents a feature channel, and each type of small square represents a feature channel at the same timestamp, for example, Figure 3 As shown in (c), the four types of feature channels 301, 302, 303 and 304 represent the feature channels at four timestamps, namely, t=0, t=1, t=2 and t=3, respectively. Figure 3 (a) in the figure shows the position of each feature channel before feature exchange. Figure 3(b) in the figure shows the positions of the feature channels after the 1 / 8 feature channel corresponding to the current video frame is moved to the previous adjacent video frame along the time dimension, and the 1 / 8 feature channel corresponding to the current video frame is moved to the next adjacent video frame along the time dimension. It should be noted that the position corresponding to the vacant feature channel after the movement can be directly filled with zero, which means that there is no feature channel at the position corresponding to the feature channel after the movement. Figure 3 (c) in the figure is the position of each feature channel after the 1 / 4 feature channel corresponding to the current video frame is moved to the previous adjacent video frame along the time dimension. It can be understood that by moving part of the feature channel corresponding to the current video frame to the adjacent video frame along the time dimension, the features corresponding to the feature channels moved to the adjacent video frame can be obtained from the adjacent video frame to obtain the adjacent exchange features. For example, Figure 3 The feature 305 shown in (c) belongs to the adjacent exchange feature, while Figure 3 The other features except the feature 305 in (c) are the reserved features that are not exchanged in the current video frame.

[0109] In one embodiment, feature fusion is performed on image features corresponding to each video frame to obtain video features corresponding to each sample video in the sample video pair, including: feature clustering is performed on image features corresponding to each video frame; for each cluster, the distance from each image feature in the cluster to the cluster center is determined, and the probability value of each image feature in the cluster belonging to the cluster center is determined; based on the distance from each image feature to the cluster center and the probability value of each image feature belonging to the cluster center, the feature corresponding to the cluster is determined; the features corresponding to each cluster are fused to obtain the video features corresponding to each sample video in the sample video pair.

[0110] Specifically, the computer device may perform feature clustering on the image features corresponding to each video frame based on the similarity between the image features corresponding to each video frame. It can be understood that the computer device may cluster the image features corresponding to each video frame into at least one category based on the similarity between the image features corresponding to each video frame. Furthermore, for each cluster, the computer device may respectively determine the distance from each image feature in the cluster to the cluster center, and respectively determine the probability value that each image feature in the cluster belongs to the cluster center. The computer device may determine the features corresponding to the cluster based on the distance from each image feature to the cluster center and the probability value that each image feature belongs to the cluster center, and fuse the features corresponding to each cluster to obtain the video features corresponding to each sample video in the sample video pair.

[0111] In the above embodiment, by performing feature clustering on the image features corresponding to each video frame, the distance from each image feature in the cluster to the cluster center and the probability value of each image feature in the cluster belonging to the cluster center can be determined. Based on the distance from each image feature to the cluster center and the probability value of each image feature belonging to the cluster center, the features corresponding to the cluster can be determined quickly and accurately, and then the features corresponding to each cluster are fused to obtain the video features corresponding to each sample video in the sample video pair. Compared with the traditional method of giving each image feature the same weight for fusion, the present application improves the accuracy of video features by clustering image features and automatically learning the corresponding fusion weight of each image feature based on the characteristics of each image feature itself.

[0112] In one embodiment, for each sample video in each sample video pair, the video content similarity between each sample video and the text similarity between reference video tags of each sample video are determined, including: for each sample video in each sample video pair, obtaining offline video features of each sample video and obtaining label features of reference video tags corresponding to each sample video; performing similarity calculation on the offline video features of each sample video to obtain the video content similarity between each sample video; performing similarity calculation on the label features corresponding to each sample video to obtain the text similarity between reference video tags of each sample video.

[0113] Among them, offline video features refer to video features that will not change during the training process of the label prediction model. It can be understood that the acquisition of offline video features is completely independent of the training process of the label prediction model, and the training process of the label prediction model will not cause offline video features to change. Label features are features used to describe the text semantics of the reference video label.

[0114] Specifically, for each sample video in each sample video pair, the computer device can directly extract the offline video features of each sample video, and directly obtain the label features of the reference video labels corresponding to each sample video. Furthermore, the computer device can perform similarity calculation on the offline video features of each sample video to obtain the video content similarity between each sample video, and perform similarity calculation on the label features corresponding to each sample video to obtain the text similarity between the reference video labels of each sample video.

[0115] In one embodiment, for each sample video in each sample video pair, the computer device may input each sample video into an offline video feature extraction model, so as to directly extract the offline video features of each sample video through the offline video feature extraction model. The computer device may input the reference video labels corresponding to each sample video into a label feature extraction model, so as to extract the label features of the reference video labels corresponding to each sample video through the label feature extraction model. Among them, the offline video feature extraction model is a neural network model for extracting offline video features from sample videos. The label feature extraction model is a neural network model for extracting label features from the reference video labels corresponding to each sample video.

[0116] In the above embodiment, by calculating the similarity of the offline video features of each sample video, the video content similarity between each sample video can be obtained, thereby improving the calculation accuracy of the video content similarity. By calculating the similarity of the label features corresponding to each sample video, the text similarity between the reference video labels of each sample video can be obtained, thereby improving the calculation accuracy of the text similarity.

[0117] In one embodiment, each sample video corresponds to a plurality of label feature vectors; similarity calculation is performed on the label features corresponding to each sample video to obtain text similarity between reference video labels of each sample video, including: for each sample video, based on the plurality of label feature vectors corresponding to the sample video, determining a comprehensive label feature vector corresponding to the sample video; determining the cosine similarity between the comprehensive label feature vectors corresponding to each sample video to obtain text similarity.

[0118] Among them, the label feature vector is a vector used to describe the label feature. The comprehensive label feature vector is a feature vector determined based on multiple label feature vectors. The cosine similarity is the similarity between the comprehensive label feature vectors corresponding to each sample video determined based on the cosine value.

[0119] Specifically, for each sample video, the sample video belongs to multiple reference video tags, and one reference video tag corresponds to one tag feature vector. It can be understood that the sample video corresponds to multiple tag feature vectors. The computer device can determine the comprehensive tag feature vector corresponding to the sample video based on the multiple tag feature vectors corresponding to the sample video. Furthermore, the computer device can calculate the cosine similarity between the comprehensive tag feature vectors corresponding to each sample video, and use the cosine similarity as the text similarity.

[0120] In one embodiment, the computer device may average multiple label feature vectors corresponding to the sample video to obtain an averaged label feature vector. The computer device may use the averaged label feature vector as the comprehensive label feature vector corresponding to the sample video.

[0121] In the above embodiment, based on multiple label feature vectors corresponding to the sample video, the comprehensive label feature vector corresponding to the sample video can be determined, and based on the cosine similarity between the comprehensive label feature vectors corresponding to each sample video, the text similarity can be directly obtained, further improving the calculation accuracy of the text similarity.

[0122] In one embodiment, a second loss value is determined based on video content similarity and text similarity, including: determining text difference values ​​between reference video labels corresponding to each sample video in a sample video pair based on text similarity; using video content similarity as a label similarity relationship coefficient to perform weighted adjustment on the text difference value to obtain a second loss value.

[0123] The text difference value is the difference between the text content of the reference video label corresponding to each sample video in the sample video pair. The label similarity coefficient is a coefficient used to characterize the relationship between the reference video label corresponding to each sample video in the sample video pair.

[0124] Specifically, the computer device may determine the text difference value between the reference video labels corresponding to each sample video in the sample video pair according to the text similarity. Furthermore, the computer device may use the video content similarity as the label similarity coefficient to weight the text difference value to obtain the second loss value. It can be understood that the computer device may multiply the video content similarity with the text difference value and directly use the multiplied result as the second loss value.

[0125] In the above embodiment, the text similarity can be used to quickly and accurately determine the text difference values ​​between the reference video labels corresponding to each sample video in the sample video pair, and by weighting the text difference value using the video content similarity as the label similarity relationship coefficient, the second loss value can be quickly and accurately obtained, thereby improving the efficiency and accuracy of obtaining the second loss value.

[0126] In one embodiment, the label prediction model includes a video feature extraction network and a label prediction network; through the label prediction model to be trained, the video features corresponding to each sample video in the sample video pair are obtained, including: the video feature extraction network is used to extract features of each sample video in the sample video pair to obtain the video features corresponding to each sample video in the sample video pair; based on the video features, a predicted video label corresponding to the sample video is predicted, including: the label prediction network is used to predict labels based on the video features to obtain predicted video labels corresponding to the sample video.

[0127] The video feature extraction network is a neural network used to extract video features from sample videos, and the label prediction network is a neural network used to predict the video labels of sample videos based on video features.

[0128] Specifically, the computer device may input each sample video in the sample video pair into the video feature extraction network in the label prediction model, so as to extract features from each sample video in the sample video pair through the video feature extraction network, and obtain video features corresponding to each sample video in the sample video pair. The computer device may input the video features corresponding to each sample video in the sample video pair into the label prediction network in the label prediction model, so as to perform label prediction based on the video features through the label prediction network, and obtain a predicted video label corresponding to the sample video.

[0129] In the above embodiment, through the video feature extraction network, feature extraction can be performed on each sample video in the sample video pair, and the video features corresponding to each sample video in the sample video pair can be quickly and accurately obtained. Through the label prediction network, label prediction can be performed based on the video features, and the predicted video label corresponding to the sample video can be quickly and accurately obtained, thereby improving the accuracy of the predicted video label.

[0130] In one embodiment, the label prediction model to be trained is trained according to the first loss value and the second loss value to obtain a trained label prediction model, including: determining a target loss value according to the first loss value and the second loss value; adjusting the network parameters of the video feature extraction network and the label prediction network respectively in the direction of reducing the target loss value to train the trained label prediction model.

[0131] Among them, the target loss value is the loss value used as the target loss.

[0132] Specifically, the computer device may determine a target loss value according to the first loss value and the second loss value. Furthermore, the computer device may iteratively train the video feature extraction network and the label prediction network in a direction that reduces the target loss value, so as to adjust the network parameters of the video feature extraction network and the label prediction network respectively, until a training stop condition is reached, thereby obtaining a trained label prediction model.

[0133] In one embodiment, if a sample video belongs to at least one video tag, then for each video tag, the computer device may sum the differences between the predicted video tag and the reference video tag corresponding to the sample video, and use the summed result as the first loss value.

[0134] In one embodiment, the first loss value can be calculated by the following formula:

[0135]

[0136] in, represents the first loss value. C represents the total number of reference video labels of the sample video, and c represents the number of labels in the reference video label. sc represents the score vector of the predicted sample video belonging to each reference video label, and yc represents the score vector of the pre-annotated, i.e., the real sample video belonging to each reference video label.

[0137] In one embodiment, the offline video feature may include an offline video feature vector, and the label feature may include a label feature vector. The computer device may perform dot multiplication on the offline video feature vectors corresponding to the two sample videos in the sample video pair, and perform dot multiplication on the label feature vectors corresponding to the reference video labels of the two sample videos. Furthermore, the computer device may determine the second loss value based on the offline video feature vector after dot multiplication and the label feature vector after dot multiplication.

[0138] In one embodiment, the computer device may determine the text difference value between the reference video labels corresponding to each sample video in the sample video pair according to the text similarity. The computer device may use the video content similarity as the label similarity relationship coefficient, and multiply the video content similarity with the text difference value, and use the multiplication result as the second loss value.

[0139] In one embodiment, the second loss value can be calculated by the following formula:

[0140]

[0141] in, represents the second loss value. i represents the sample video i in the sample video pair, and j represents the sample video j in the sample video pair. vi represents the offline video feature vector corresponding to the sample video i, and vj represents the offline video feature vector corresponding to the sample video j. ti represents the label feature vector corresponding to the reference video label of the sample video i, and tj represents the label feature vector corresponding to the reference video label of the sample video j.

[0142] In one embodiment, the computer device may sum the first loss values ​​corresponding to the two sample videos in the sample video pair, and use the preset adjustment parameter of the second loss value as the weighted coefficient of the second loss value. The computer device may perform a weighted summation on the summed first loss value and the weighted second loss value to obtain a target loss value.

[0143] In one embodiment, the target loss value may be calculated by the following formula:

[0144]

[0145] in, represents the target loss value, represents the first loss value corresponding to sample video i, represents the first loss value corresponding to sample video j, represents the second loss value, and λ represents the adjustment parameter of the customized second loss value.

[0146] In the above embodiment, the target loss value can be quickly determined based on the first loss value and the second loss value. In the direction of reducing the target loss value, the network parameters of the video feature extraction network and the label prediction network can be quickly adjusted to train a label prediction model, thereby improving the efficiency of model training.

[0147] In one embodiment, a video feature extraction network includes an image feature extraction unit and an image feature fusion unit; the video feature extraction network performs feature extraction on each sample video in a sample video pair to obtain video features corresponding to each sample video in the sample video pair, including: for each sample video in each sample video pair, the image feature extraction unit extracts image features corresponding to each video frame in the sample video; the image feature fusion unit performs feature fusion on image features corresponding to each video frame in the sample video to obtain video features corresponding to each sample video in the sample video pair.

[0148] The image feature extraction unit is a neural network unit for extracting image features from video frames, and the image feature fusion unit is a neural network unit for fusing image features corresponding to each video frame.

[0149] Specifically, for each sample video in each sample video pair, the computer device may input each sample video into the image feature extraction unit in the label prediction model, so as to extract the image features corresponding to each video frame in the sample video through the image feature extraction unit. The computer device may input the image features corresponding to each video frame into the image feature fusion unit in the label prediction model, so as to perform feature fusion on the image features corresponding to each video frame in the sample video through the image feature fusion unit, and obtain the video features corresponding to each sample video in the sample video pair.

[0150] In the above embodiment, the image features corresponding to each video frame in the sample video can be extracted quickly and accurately through the image feature extraction unit. The image features corresponding to each video frame in the sample video can be quickly and accurately fused through the image feature fusion unit to obtain the video features corresponding to each sample video in the sample video pair, further improving the accuracy of the video features.

[0151] In one embodiment, the video processing method also includes: obtaining a target video; predicting a label for the target video through a trained label prediction model to obtain a target video label to which the target video belongs; searching for videos carrying the target video label; and recommending the found videos carrying the target video label.

[0152] Specifically, the computer device can obtain the target video and input the target video into the trained label prediction model to predict the label of the target video through the trained label prediction model to obtain the target video label to which the target video belongs. Furthermore, the computer device can search for videos carrying the target video label in the database and recommend the found videos carrying the target video label.

[0153] In one embodiment, the trained label prediction model includes an image feature extraction unit, an image feature fusion unit, and a label prediction network. The computer device can obtain a target video and input the target video into the image feature extraction unit to extract the image features corresponding to each video frame in the target video through the image feature extraction unit. The computer device can input the image features corresponding to each video frame into the image feature fusion unit to perform feature fusion on the image features corresponding to each video frame through the image feature fusion unit to obtain the video features corresponding to the target video. The computer device can input the video features corresponding to the target video into the label prediction network to perform label prediction based on the video features corresponding to the target video through the label prediction network to obtain the target video label corresponding to the target video.

[0154] In the above embodiment, the trained label prediction model can automatically predict the label of the target video and obtain the target video label to which the target video belongs, which saves labor costs compared to the traditional manual labeling method. At the same time, by searching for videos with the target video label and recommending the found videos with the target video label, the accuracy of video recommendation can be improved.

[0155] In one embodiment, the computer device may obtain a target video, and perform label prediction on the target video through a trained label prediction model to obtain a target video label to which the target video belongs. Furthermore, the computer device may send the target video label to which the target video belongs to a video display terminal so as to display the target video label to which the target video belongs through the display terminal. The video display terminal may generate a video query instruction in response to a trigger operation for the target video label. The video display terminal may send a video query instruction to the computer device to instruct the computer device to search for a video carrying the target video label, and send the found video carrying the target video label to the video display terminal, and the video display terminal may play the video carrying the target video label sent by the computer device.

[0156] For example, Figure 4 As shown, the target video tags include 4a "fishing" and 4b "wild fishing", and the video display terminal can display the two target video tags "fishing" and "wild fishing". Figure 4 In the example, if the user is currently browsing video 401, when the user clicks 4b "Wild Fishing", the video display terminal may generate a video query instruction carrying "Wild Fishing" and send the video query instruction carrying "Wild Fishing" to the computer device. The computer device may respond to the video query instruction carrying "Wild Fishing" and search for videos related to "Wild Fishing". For example, if the video tag of video 402 includes "Wild Fishing", the computer device may recommend video 402 related to "Wild Fishing" to the video display terminal, and the video display terminal may receive and play video 402 related to "Wild Fishing" recommended by the computer device.

[0157] In one embodiment, the computer device may obtain a target video, and use the trained label prediction model to predict the label of the target video, obtain the target video label to which the target video belongs, and associate the target video label with the target video for storage. Figure 5 As shown, in the video search scenario, the user enters the video search keyword "food" in the video search box 501. After the computer device obtains the video search keyword "food", it can search for videos whose video tags include "food". For example, the video tags of videos 1-video 7 include "food". The computer device can feedback the found videos 1-video 7 to the user.

[0158] In one embodiment, Figure 6As shown, the label prediction model includes an image feature extraction unit, an image feature fusion unit, and a label prediction network. The computer device can obtain a sample video pair, wherein the sample video pair includes a sample video i and a sample video j, and both the sample video i and the sample video j carry corresponding reference video labels. The computer device samples the image frames of the sample video i and the sample video j respectively to obtain the video frames corresponding to the sample video i and the sample video j respectively. The computer device can input the video frames corresponding to the sample video i and the sample video j respectively into the image feature extraction unit, so as to extract the image features corresponding to each video frame in the sample video through the image feature extraction unit. The computer device can input the image features corresponding to each video frame into the image feature fusion unit, so as to perform feature fusion on the image features corresponding to each video frame through the image feature fusion unit, and obtain the video features corresponding to the sample video. The computer device can input the video features corresponding to the sample video into the label prediction network, so as to perform label prediction based on the video features corresponding to the sample video through the label prediction network, and obtain the predicted video label corresponding to the sample video. For the same sample video (i.e., sample video i or sample video j), the computer device can determine the first loss value based on the difference between the predicted video label and the reference video label of the sample video through the first loss function. The computer device can respectively obtain the offline video labels corresponding to the sample video i and the sample video j, and the label features corresponding to the reference video labels of the sample video i and the sample video j. The computer device can determine the second loss value based on the second loss function and the offline video labels and label features corresponding to the sample video i and the sample video j. It can be understood that based on the first loss function and the second loss function, the target loss function can be obtained, and based on the first loss value and the second loss value, the target loss value corresponding to the target loss function can be obtained. Furthermore, the computer device can iteratively train the label prediction model based on the target loss value until the iteration stop condition is reached to obtain a trained label prediction model.

[0159] like Figure 7 As shown, in one embodiment, a video processing method is provided, which is applied to a computer device, and the method specifically includes the following steps:

[0160] Step 702, obtaining a set of sample video pairs; each sample video in the sample video pair carries a corresponding reference video tag.

[0161] Step 704: for each sample video in each sample video pair, the image feature extraction unit in the label prediction model to be trained is used to extract features of each video frame in the sample video to obtain image features corresponding to each video frame.

[0162] Step 706 , the image features corresponding to each video frame are respectively fused by the image feature fusion unit in the label prediction model to be trained to obtain the video features corresponding to each sample video in the sample video pair.

[0163] Step 708: predicting a predicted video label corresponding to the sample video based on the video features through the label prediction network in the label prediction model to be trained.

[0164] Step 710: Determine a first loss value based on the difference between the predicted video label and the reference video label corresponding to the same sample video.

[0165] Step 712: for each sample video in each sample video pair, obtain the offline video features of each sample video, and obtain the label features of the reference video labels corresponding to each sample video.

[0166] Step 714: perform similarity calculation on the offline video features of each sample video to obtain the video content similarity between the sample videos.

[0167] Step 716, performing similarity calculation on the label features corresponding to each sample video to obtain the text similarity between the reference video labels of each sample video.

[0168] Step 718: Determine the text difference value between the reference video labels corresponding to each sample video in the sample video pair according to the text similarity.

[0169] Step 720, using the video content similarity as a label similarity relationship coefficient to perform weighted adjustment on the text difference value to obtain a second loss value; the second loss value is used to characterize the label similarity relationship between the reference video labels of each sample video.

[0170] Step 722: determine a target loss value according to the first loss value and the second loss value.

[0171] Step 724, in the direction of reducing the target loss value, adjust the network parameters of the video feature extraction network and the label prediction network to obtain a trained label prediction model.

[0172] Step 726, obtain the target video, and use the trained label prediction model to perform label prediction on the target video to obtain the target video label to which the target video belongs.

[0173] Step 728, searching for videos carrying the target video tag, and recommending the found videos carrying the target video tag.

[0174] The present application also provides an application scenario, which applies the above-mentioned video processing method. Specifically, the video processing method can be applied to the scenario of video recommendation. It can be understood that it can be a recommendation for short videos or long videos. Now, short video recommendation is used as an example for schematic illustration. The computer device can obtain a set of sample short video pairs; each sample short video in the sample short video pair carries a corresponding reference video label. For each sample short video in each sample short video pair, the image feature extraction unit in the label prediction model to be trained is used to extract features of each video frame in the sample short video to obtain the image features corresponding to each video frame. The image feature fusion unit in the label prediction model to be trained is used to perform feature fusion on the image features corresponding to each video frame to obtain the video features corresponding to each sample short video in the sample short video pair.

[0175] The computer device can predict the predicted video label corresponding to the sample short video based on the video features through the label prediction network in the label prediction model to be trained. Based on the difference between the predicted video label and the reference video label corresponding to the same sample short video, a first loss value is determined. For each sample short video in each sample short video pair, the offline video features of each sample short video are obtained, and the label features of the reference video label corresponding to each sample short video are obtained. The offline video features of each sample short video are similarity calculated to obtain the video content similarity between each sample short video. The label features corresponding to each sample short video are similarity calculated to obtain the text similarity between the reference video labels of each sample short video. According to the text similarity, the text difference value between the reference video labels corresponding to each sample video in the sample video pair is determined; the video content similarity is used as the label similarity relationship coefficient to weight the text difference value to obtain a second loss value, and the second loss value is used to characterize the label similarity relationship between the reference video labels of each sample short video. The target loss value is determined according to the first loss value and the second loss value. In the direction of reducing the target loss value, the network parameters of the video feature extraction network and the label prediction network are adjusted to obtain a trained label prediction model.

[0176] The computer device can obtain the target short video, and use the trained label prediction model to predict the label of the target short video to obtain the target video label to which the target short video belongs; search for short videos carrying the target video label, and recommend the found short videos carrying the target video label.

[0177] The present application also provides an application scenario, which applies the above-mentioned video processing method. Specifically, the video processing method can be applied to the scenario of video search. Similarly, it can be applied to the search for short videos or long videos. It can be understood that the label prediction model trained using the method of the present application is used to predict the label of the target video, and the target video label to which each target video belongs is obtained. When a user uses keywords to search for a video, the query keyword entered by the user can be obtained, the video corresponding to the target video label carrying the query keyword can be found, and the found video can be fed back to the user terminal as the search result.

[0178] It should be understood that, although each step in the flow chart of the above-mentioned embodiments is shown in order, these steps are not necessarily performed in order. Unless there is a clear explanation in this article, the execution of these steps does not have strict order restrictions, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the above-mentioned embodiments may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in order, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0179] In one embodiment, Figure 8 As shown, a video processing device 800 is provided. The device can adopt a software module or a hardware module, or a combination of the two to become a part of a computer device. The device specifically includes:

[0180] The acquisition module 801 is used to acquire a set of sample video pairs; each sample video in the sample video pair carries a corresponding reference video tag.

[0181] The prediction module 802 is used to obtain the video features corresponding to each sample video in the sample video pair through the label prediction model to be trained, and predict the predicted video label corresponding to the sample video based on the video features.

[0182] Determination module 803 is used to determine a first loss value based on the difference between the predicted video label and the reference video label corresponding to the same sample video; for each sample video in each sample video pair, determine the video content similarity between each sample video and the text similarity between the reference video labels of each sample video; determine a second loss value based on the video content similarity and the text similarity; the second loss value is used to characterize the label similarity relationship between the reference video labels of each sample video.

[0183] The training module 804 is used to train the label prediction model to be trained according to the first loss value and the second loss value to obtain a trained label prediction model; the trained label prediction model is used to perform label prediction on the target video.

[0184] In one embodiment, the prediction module 802 is also used to extract features of each video frame in each sample video of each sample video pair through the label prediction model to be trained to obtain image features corresponding to each video frame; perform feature fusion on the image features corresponding to each video frame to obtain video features corresponding to each sample video in the sample video pair.

[0185] In one embodiment, the prediction module 802 is also used to use the label prediction model to be trained to take each video frame in the sample video as the current video frame, move part of the feature channels corresponding to the current video frame to the adjacent video frame along the time dimension to exchange features with the adjacent video frame to obtain adjacent exchange features; obtain the unexchanged features in the current video frame to obtain the retained features; combine the retained features and the adjacent exchange features to generate the image features corresponding to the current video frame.

[0186] In one embodiment, the prediction module 802 is also used to move part of the feature channels corresponding to the current video frame to the adjacent video frame along the time dimension, so as to obtain features corresponding to the feature channels moved to the adjacent video frame from the adjacent video frame to obtain adjacent exchange features; and obtain features located in the remaining feature channels in the current video frame to obtain retained features.

[0187] In one embodiment, the prediction module 802 is also used to perform feature clustering on the image features corresponding to each video frame; for each cluster, determine the distance from each image feature in the cluster to the cluster center, and determine the probability value of each image feature in the cluster belonging to the cluster center; based on the distance from each image feature to the cluster center, and the probability value of each image feature belonging to the cluster center, determine the features corresponding to the cluster; fuse the features corresponding to each cluster to obtain the video features corresponding to each sample video in the sample video pair.

[0188] In one embodiment, the determination module 803 is also used to obtain offline video features of each sample video in each sample video pair, and obtain label features of the reference video label corresponding to each sample video; perform similarity calculation on the offline video features of each sample video to obtain the video content similarity between each sample video; perform similarity calculation on the label features corresponding to each sample video to obtain the text similarity between the reference video labels of each sample video.

[0189] In one embodiment, each sample video corresponds to a plurality of label feature vectors; the determination module 803 is also used to determine, for each sample video, a comprehensive label feature vector corresponding to the sample video according to the plurality of label feature vectors corresponding to the sample video; and determine the cosine similarity between the comprehensive label feature vectors corresponding to each sample video to obtain text similarity.

[0190] In one embodiment, the determination module 803 is also used to determine the text difference value between the reference video labels corresponding to each sample video in the sample video pair according to the text similarity; the video content similarity is used as the label similarity relationship coefficient to perform weighted adjustment on the text difference value to obtain a second loss value.

[0191] In one embodiment, the label prediction model includes a video feature extraction network and a label prediction network; the prediction module 802 is also used to extract features from each sample video in the sample video pair through the video feature extraction network to obtain video features corresponding to each sample video in the sample video pair; and perform label prediction based on the video features through the label prediction network to obtain a predicted video label corresponding to the sample video.

[0192] In one embodiment, the training module 804 is also used to determine a target loss value based on the first loss value and the second loss value; adjust the network parameters of the video feature extraction network and the label prediction network in the direction of reducing the target loss value to train a trained label prediction model.

[0193] In one embodiment, the video feature extraction network includes an image feature extraction unit and an image feature fusion unit; the prediction module 802 is also used to extract the image features corresponding to each video frame in the sample video for each sample video in each sample video pair through the image feature extraction unit; and perform feature fusion on the image features corresponding to each video frame in the sample video through the image feature fusion unit to obtain the video features corresponding to each sample video in the sample video pair.

[0194] refer to Fig. 9 In one embodiment, the video processing device 800 further includes:

[0195] The recommendation module 805 is used to obtain a target video; predict the label of the target video through the trained label prediction model to obtain the target video label to which the target video belongs; search for videos carrying the target video label; and recommend the found videos carrying the target video label.

[0196] The above-mentioned video processing device obtains a set of sample video pairs, each sample video in the sample video pair carries a corresponding reference video label. Through the label prediction model to be trained, the video features corresponding to each sample video in the sample video pair can be extracted, and the predicted video label corresponding to the sample video can be predicted based on the video features. Based on the difference between the predicted video label and the reference video label corresponding to the same sample video, a first loss value can be determined. For each sample video in each sample video pair, the video content similarity between each sample video and the text similarity between the reference video labels of each sample video can be determined. According to the video content similarity and the text similarity, a second loss value for characterizing the label similarity relationship between the reference video labels of each sample video can be determined. According to the first loss value and the second loss value, the label prediction model to be trained can be iteratively trained to obtain a trained label prediction model. The trained label prediction model can be directly used to predict the label of the target video to be labeled, so as to realize automatic labeling of the label of the target video. Compared with the traditional manual labeling method, the present application directly uses the label prediction model trained based on the first loss value and the second loss value to automatically label the video to be labeled, which can greatly shorten the labeling time and improve the efficiency of video labeling.

[0197] For the specific definition of the video processing device, please refer to the definition of the video processing method above, which will not be repeated here. Each module in the above video processing device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0198] In one embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Fig.10 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a video processing method is implemented.

[0199] Those skilled in the art will understand that Fig.10The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0200] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0201] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0202] In one embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0203] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0204] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0205] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0206] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A video processing method, characterized in that: The method comprises: Obtain a set of sample video pairs; each sample video in the sample video pair carries a corresponding reference video tag; Obtaining video features corresponding to each sample video in the sample video pair through the label prediction model to be trained, and predicting a predicted video label corresponding to the sample video based on the video features; Determining a first loss value based on a difference between the predicted video label and the reference video label corresponding to the same sample video; For each sample video in each sample video pair, determining the video content similarity between the sample videos and the text similarity between the reference video tags of the sample videos; Determining, based on the text similarity, text difference values ​​between reference video labels corresponding to each sample video in the sample video pair; The video content similarity is used as a label similarity relationship coefficient to perform weighted adjustment on the text difference value to obtain a second loss value; the second loss value is used to characterize the label similarity relationship between the reference video labels of each sample video; According to the first loss value and the second loss value, the label prediction model to be trained is trained to obtain a trained label prediction model; the trained label prediction model is used to perform label prediction on the target video.

2. The method according to claim 1, characterized in that: The step of obtaining the video features corresponding to each sample video in the sample video pair through the label prediction model to be trained includes: For each sample video in each sample video pair, extract features of each video frame in the sample video by using the label prediction model to be trained to obtain image features corresponding to each video frame; Feature fusion is performed on the image features corresponding to the video frames to obtain the video features corresponding to each sample video in the sample video pair.

3. The method according to claim 2, characterized in that The feature extraction of each video frame in the sample video is performed through the label prediction model to be trained to obtain image features corresponding to each video frame, including: Using the label prediction model to be trained, taking each video frame in the sample video as the current video frame, moving part of the feature channels corresponding to the current video frame to the adjacent video frame along the time dimension to exchange features with the adjacent video frame, and obtaining adjacent exchange features; Acquire the unexchanged features in the current video frame to obtain the retained features; The retained features and the adjacent exchange features are combined to generate image features corresponding to the current video frame.

4. The method according to claim 3, characterized in that The moving of the partial feature channels corresponding to the current video frame to the adjacent video frame along the time dimension to exchange features with the adjacent video frame to obtain adjacent exchange features includes: Moving part of the feature channels corresponding to the current video frame to the adjacent video frame along the time dimension, so as to obtain features corresponding to the feature channels moved to the adjacent video frame from the adjacent video frame, and obtain adjacent exchange features; The step of obtaining unexchanged features in the current video frame to obtain retained features includes: The features in the remaining feature channels in the current video frame are obtained to obtain retained features.

5. The method according to claim 2, characterized in that: The performing feature fusion on the image features corresponding to the respective video frames to obtain the video features corresponding to each sample video in the sample video pair includes: Performing feature clustering on the image features corresponding to each of the video frames; For each cluster, respectively determining the distance from each image feature in the cluster to the cluster center, and respectively determining the probability value of each image feature in the cluster belonging to the cluster center; Determine a feature corresponding to the cluster based on the distance from each image feature to the cluster center and the probability value that each image feature belongs to the cluster center; The features corresponding to each cluster are fused to obtain the video features corresponding to each sample video in the sample video pair.

6. The method according to claim 1, characterized in that The step of determining, for each sample video in each sample video pair, the video content similarity between the sample videos and the text similarity between the reference video tags of the sample videos comprises: For each sample video in each sample video pair, obtaining offline video features of each sample video, and obtaining label features of reference video labels corresponding to each sample video; Calculating the similarity of the offline video features of each sample video to obtain the video content similarity between the sample videos; The similarity of the label features corresponding to the sample videos is calculated to obtain the text similarity between the reference video labels of the sample videos.

7. The method according to claim 6, characterized in that Each of the sample videos corresponds to a plurality of label feature vectors; The performing similarity calculation on the label features corresponding to the sample videos to obtain the text similarity between the reference video labels of the sample videos includes: For each sample video, determining a comprehensive label feature vector corresponding to the sample video according to multiple label feature vectors corresponding to the sample video; The cosine similarity between the comprehensive label feature vectors corresponding to the sample videos is determined to obtain the text similarity.

8. The method according to claim 1, characterized in that The label prediction model includes a video feature extraction network and a label prediction network; The step of obtaining the video features corresponding to each sample video in the sample video pair through the label prediction model to be trained includes: Extracting features from each sample video in the sample video pair through the video feature extraction network to obtain video features corresponding to each sample video in the sample video pair; The predicting a predicted video label corresponding to the sample video based on the video feature includes: The label prediction network performs label prediction based on the video features to obtain a predicted video label corresponding to the sample video.

9. The method according to claim 8, characterized in that The step of training the label prediction model to be trained according to the first loss value and the second loss value to obtain a trained label prediction model includes: Determine a target loss value according to the first loss value and the second loss value; In the direction of reducing the target loss value, the network parameters of the video feature extraction network and the label prediction network are adjusted to train and obtain the trained label prediction model.

10. The method according to claim 8, characterized in that The video feature extraction network includes an image feature extraction unit and an image feature fusion unit; The extracting features of each sample video in the sample video pair by the video feature extraction network to obtain the video features corresponding to each sample video in the sample video pair includes: For each sample video in each sample video pair, extracting image features corresponding to each video frame in the sample video through the image feature extraction unit; The image feature fusion unit performs feature fusion on the image features corresponding to each video frame in the sample video to obtain the video features corresponding to each sample video in the sample video pair.

11. The method according to any one of claims 1 to 10, characterized in that The method further comprises: Get the target video; Using the trained label prediction model, label prediction is performed on the target video to obtain a target video label to which the target video belongs; Searching for videos carrying the target video tag; The found video carrying the target video tag is recommended.

12. A video processing device, characterized in that: The device comprises: An acquisition module, used to acquire a set of sample video pairs; each sample video in the sample video pair carries a corresponding reference video tag; A prediction module, used to obtain video features corresponding to each sample video in the sample video pair through a label prediction model to be trained, and predict a predicted video label corresponding to the sample video based on the video features; A determination module, used to determine a first loss value based on the difference between the predicted video label and the reference video label corresponding to the same sample video; for each sample video in each sample video pair, determine the video content similarity between the sample videos and the text similarity between the reference video labels of the sample videos; according to the text similarity, determine the text difference value between the reference video labels corresponding to each sample video in the sample video pair; use the video content similarity as a label similarity relationship coefficient to perform a weighted adjustment on the text difference value to obtain a second loss value; the second loss value is used to characterize the label similarity relationship between the reference video labels of each sample video; A training module is used to train the label prediction model to be trained according to the first loss value and the second loss value to obtain a trained label prediction model; the trained label prediction model is used to perform label prediction on the target video.

13. The device according to claim 12, characterized in that The prediction module is also used to extract features of each video frame in each sample video of each sample video pair through a label prediction model to be trained to obtain image features corresponding to each video frame; and perform feature fusion on the image features corresponding to each video frame to obtain video features corresponding to each sample video in the sample video pair.

14. The device according to claim 13, characterized in that The prediction module is also used to use the label prediction model to be trained to take each video frame in the sample video as the current video frame, and move part of the feature channels corresponding to the current video frame to the adjacent video frame along the time dimension to exchange features with the adjacent video frame to obtain adjacent exchange features; Acquire the unexchanged features in the current video frame to obtain the retained features; combine the retained features with the adjacent exchanged features to generate image features corresponding to the current video frame.

15. The device according to claim 14, characterized in that The prediction module is further used to move part of the feature channels corresponding to the current video frame to the adjacent video frame along the time dimension, so as to obtain features corresponding to the feature channels moved to the adjacent video frame from the adjacent video frame, and obtain adjacent exchange features; The features in the remaining feature channels in the current video frame are obtained to obtain retained features.

16. The device according to claim 13, characterized in that The prediction module is also used to perform feature clustering on the image features corresponding to the video frames; for each cluster, determine the distance from each image feature in the cluster to the cluster center, and determine the probability value of each image feature in the cluster belonging to the cluster center; based on the distance from each image feature to the cluster center and the probability value of each image feature belonging to the cluster center, determine the features corresponding to the cluster; and fuse the features corresponding to the clusters to obtain the video features corresponding to each sample video in the sample video pair.

17. The device according to claim 12, characterized in that The determination module is further used to obtain, for each sample video in each sample video pair, offline video features of each sample video, and obtain label features of reference video labels corresponding to each sample video; The offline video features of the sample videos are similarly calculated to obtain the video content similarity between the sample videos; the label features corresponding to the sample videos are similarly calculated to obtain the text similarity between the reference video labels of the sample videos.

18. The device according to claim 17, characterized in that Each of the sample videos corresponds to a plurality of label feature vectors; The determination module is also used to determine, for each sample video, a comprehensive label feature vector corresponding to the sample video according to multiple label feature vectors corresponding to the sample video; determine the cosine similarity between the comprehensive label feature vectors corresponding to each sample video to obtain text similarity.

19. The device according to claim 12, characterized in that The label prediction model includes a video feature extraction network and a label prediction network; The prediction module is also used to extract features from each sample video in the sample video pair through the video feature extraction network to obtain video features corresponding to each sample video in the sample video pair; and to perform label prediction based on the video features through the label prediction network to obtain a predicted video label corresponding to the sample video.

20. The device according to claim 19, characterized in that The training module is also used to determine a target loss value based on the first loss value and the second loss value; adjust the network parameters of the video feature extraction network and the label prediction network in the direction of reducing the target loss value, so as to train and obtain the trained label prediction model.

21. The device according to claim 19, characterized in that The video feature extraction network includes an image feature extraction unit and an image feature fusion unit; The prediction module is also used to extract the image features corresponding to each video frame in each sample video in each sample video pair through the image feature extraction unit; and to perform feature fusion on the image features corresponding to each video frame in the sample video through the image feature fusion unit to obtain the video features corresponding to each sample video in the sample video pair.

22. The device according to any one of claims 12 to 21, characterized in that The device also includes: The recommendation module is used to obtain a target video; predict the label of the target video through the trained label prediction model to obtain a target video label to which the target video belongs; search for videos carrying the target video label; and recommend the found videos carrying the target video label.

23. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

24. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

25. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images