Video tag processing method and device

Abstract

The embodiment of the present application provides a video label processing method and device, including: obtaining a target video; inputting the target video into a plurality of different video classification models respectively, obtaining candidate video labels output by the video classification models; correcting the candidate video labels according to the co-occurrence relationship between the candidate video labels output by two different video classification models; displaying the target video and the corrected candidate video labels in an interface. In the embodiment of the present application, the entire process is less dependent on manual labor, and the identification process of the candidate video labels is based on machine learning and does not rely on manual subjective judgment, thereby improving the accuracy of the labeling process.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video tag processing method and device, an electronic device, and a machine-readable medium. Background Art

[0002] With the rise of short video content, short video consumption has experienced explosive growth, and the demand for short video production tools is also growing.

[0003] In the related technology, video producers need to process a large amount of short video materials. In order to facilitate the management of short videos, they can watch the video content manually and add corresponding video tags to the short videos, so as to facilitate the classification and recommendation of short videos through video tags.

[0004] However, in the current solution, manual video labeling is time-consuming and labor-intensive, and is affected by subjective factors, resulting in low accuracy. Summary of the invention

[0005] The embodiment of the present application provides a video label processing method to solve the problem that the manual video labeling method in the related art is time-consuming and labor-intensive, and is affected by subjective factors and leads to low accuracy.

[0006] Correspondingly, the embodiment of the present application also provides a video tag processing device, an electronic device and a storage medium to ensure the implementation and application of the above method.

[0007] In order to solve the above problems, the embodiment of the present application discloses a video tag processing method, the method comprising:

[0008] Get the target video;

[0009] Inputting the target video into a plurality of different video classification models respectively, and obtaining candidate video labels output by the video classification models;

[0010] Correcting the candidate video tags according to the co-occurrence relationship between the candidate video tags output by two different video classification models;

[0011] The target video and the labels of the alternative videos after error correction are displayed in the interface.

[0012] The embodiment of the present application discloses a video tag processing device, the device comprising:

[0013] An acquisition module, used to acquire a target video;

[0014] A generation module, used to input the target video into a plurality of different video classification models respectively, and obtain candidate video labels output by the video classification models;

[0015] An error correction module, configured to correct errors of the candidate video tags according to the co-occurrence relationship between the candidate video tags output by two different video classification models;

[0016] The display module is used to display the target video and the error-corrected alternative video tags in the interface.

[0017] An embodiment of the present application further discloses an electronic device, including: a processor; and a memory, on which executable code is stored. When the executable code is executed, the processor executes one or more methods described in the embodiments of the present application.

[0018] The embodiments of the present application also disclose one or more machine-readable media on which executable codes are stored. When the executable codes are executed, the processor executes one or more methods described in the embodiments of the present application.

[0019] Compared with the prior art, the embodiments of the present application have the following advantages:

[0020] In an embodiment of the present application, a multi-dimensional video classification model can be used to accurately extract alternative video tags that match short videos in multiple dimensions, and further utilize the co-occurrence relationship between the alternative video tags output by two different video classification models to correct the alternative video tags by analyzing the correlation between the identification tags in different classification dimensions, thereby improving the accuracy of the output alternative video tags. Finally, the short video and the corrected alternative video tags are displayed through the interface, thereby improving the intuitiveness and convenience of the video labeling process. The entire process is less dependent on manual labor, and the identification process of the alternative video tags is based on machine learning and does not rely on manual subjective judgment, thereby improving the accuracy of the labeling process. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is an architecture diagram of a video tag processing method according to an embodiment of the present application;

[0022] Figure 2 This is an interface diagram of a method for processing labels of a sports short video according to an embodiment of the present application;

[0023] Figure 3 This is an interface diagram of a label processing method for identifying a courier package video in an embodiment of the present application;

[0024] Figure 4 is a flowchart of the steps of a video tag processing method according to an embodiment of the present application;

[0025] Figure 5 This is a flowchart of the specific steps of a video tag processing method according to an embodiment of the present application;

[0026] Figure 6 is a bipartite graph in an embodiment of the present application;

[0027] Figure 7 This is an interface diagram of another method for processing labels of sports short videos according to an embodiment of the present application;

[0028] Figure 8 This is an interface diagram of another method for processing labels of sports short videos according to an embodiment of the present application;

[0029] Fig. 9 is a structural block diagram of a video tag processing device according to an embodiment of the present application;

[0030] Fig.10 It is a schematic diagram of the structure of a device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0032] In order to enable those skilled in the art to better understand the present application, the concepts involved in the present application are explained below:

[0033] Video classification model: A network model constructed based on deep learning classification technology, which has a high-level artificial neural network structure and can be used to realize the classification of multimodal content such as images, videos, voices, and texts. In the embodiment of the present application, the video classification model can take the video as input and the classification candidate video labels corresponding to the video and the confidence of the candidate video labels as output.

[0034] Video tag: In the embodiment of the present application, it can be a classification tag, specifically a text description, used to describe the characteristics of the content of the video. For example, a video of playing football, its corresponding alternative video tags may include: "football", "sports", etc.

[0035] Confidence: In statistics, the confidence interval of a probability sample is an interval estimate of a population parameter of the sample. The confidence shows the degree to which the true value of the parameter has a certain probability of falling around the measurement result. In the embodiment of the present application, the confidence of the candidate video tag reflects the degree of match between the candidate video tag and the video. The greater the confidence, the more the candidate video tag matches the video.

[0036] Co-occurrence relationship: used to characterize the situation where multiple feature items appear together. In the embodiment of the present application, the feature item can be an alternative video label in text form. The co-occurrence relationship between the alternative video labels can be used to characterize the correlation between the multiple alternative video labels. That is, if there is a co-occurrence relationship between two alternative video labels, it means that there is a correlation between the two alternative video labels; if there is no co-occurrence relationship between the two alternative video labels, it means that there is no correlation between the two alternative video labels.

[0037] Reference Figure 1 , which shows an architecture diagram of a video tag processing method provided in an embodiment of the present application, including: an alternative video tag generation module, an alternative video tag post-processing module, and a display module.

[0038] The architecture of the video label processing method of the embodiment of the present application can be applied to the field of labeling short videos. Specifically, the short video can be input into the video classification models corresponding to the multiple classification dimensions in the alternative video label generation module, and the multi-dimensional video classification model can be used to output the alternative video labels under each classification dimension. After that, the alternative video label post-processing module can use the co-occurrence relationship between the alternative video labels output by two different video classification models to correct the alternative video labels, and use the display module to display the short video and the corrected alternative video labels in the interface. For the video author, he can conveniently preview the short video in the interface and The selected alternative video tags are selected and bound to the short video, so that the embodiment of the present application first uses a multi-dimensional video classification model to accurately extract the alternative video tags that match the short video in multiple dimensions, and further uses the co-occurrence relationship between the alternative video tags output by two different video classification models to improve the confidence of the alternative video tags with higher mutual correlation in different dimensions, and at the same time reduce the confidence of the alternative video tags with lower mutual correlation in different dimensions, so as to achieve the purpose of error correction of the alternative video tags. Finally, the short video and the corrected alternative video tags are displayed through the interface, and combined with the author's operation, the intuitiveness and convenience of the tag binding process are improved. In addition, the selected alternative video tags can also be displayed during the playback of the short video, and the user can also use the video tag as a search term to quickly search for short videos related to the search term.

[0039] The architecture of the video tag processing method of the embodiment of the present application can also be applied to the field of live broadcasting. Specifically, during the live broadcast of the anchor, a target video can be captured at preset time intervals, and the target video can be input into the video classification models corresponding to the multiple classification dimensions in the alternative video tag generation module, and the alternative video tags under each classification dimension can be output. After that, the alternative video tags can be corrected by the alternative video tag post-processing module, and the short video and the corrected alternative video tags can be displayed in the interface using the display module. For the live broadcaster, he can conveniently preview the live video clips in the live broadcast management interface, and by selecting the alternative video tags shown in the interface, the selected alternative video tags can be bound to the live video clips, so that the selected alternative video tags can be displayed in the live broadcast screen in real time. In addition, the selected alternative video tags can also be displayed in the anchor's profile, thereby improving the intuitiveness of the display of key information during the live broadcast.

[0040] Specifically, the alternative video label generation module can first determine the various classification dimensions and specific alternative video label sets covered by the alternative video label system according to the actual needs in the video production application scenario. For example, according to the actual needs, the classification dimensions such as action, commodity, content... can be divided. The alternative video label set of the action dimension may include: football, basketball, running, gymnastics... and other alternative video labels; the alternative video label set of the commodity dimension may include: football, basketball, breakfast machine, TV... and other alternative video labels; the alternative video label set of the content dimension may include: live streaming, singing, dancing, explaining, online classes... and other alternative video labels.

[0041] Furthermore, after dividing each classification dimension, the embodiment of the present application can design a video classification model corresponding to each classification dimension according to the characteristics of each classification dimension, so that the video classification model can process the characteristics of the corresponding classification dimension, thereby obtaining a higher-precision candidate video label output, such as the content dimension has a corresponding content-class label video classification model; the commodity dimension has a corresponding commodity-class label video classification model; the action dimension has a corresponding action-class label video classification model. The video classification model can convert the picture frames of the input video content into recognizable feature vectors. Based on the distance measurement method between feature vectors, the search technology can be used to find the candidate video labels that are similar to the feature vectors from the candidate video label set, and the confidence of the candidate video labels can be determined according to the similarity. The video classification model under each dimension has made corresponding improvements based on the data characteristics of the dimension, actual scene requirements and other factors.

[0042] In a scenario of labeling alternative videos for sports videos, since action videos often use multiple consecutive frames to show the actions of characters, the action-labeled video classification model designed under the action dimension will encode and combine the features based on the timing of the feature-corresponding frames when extracting the features of the action video frames (such as the features of multiple frames that make up a jumping action can be combined into one according to the timing coding). Then, when matching the alternative video labels based on the timing features based on the timing coding combination, more accurate action label matching results can be obtained. In addition, for the commodity-labeled video classification model, the product names of the sports equipment covered by the sports video can be identified, and for the content-labeled video classification model, the sports video can be identified as a sports video, travel photography, action movie, and other content forms.

[0043] As reference Figure 2 , Figure 2 An interface diagram of a method for processing labels for sports short videos provided in an embodiment of the present application is shown. For a short video of playing football, the football and jersey labels of the product dimension are identified; the football and running labels of the action dimension; the sports video and scenery labels of the content dimension are identified; in addition to displaying the short video and the video name in the interface, alternative video labels of each dimension can also be displayed. The user can drag the selected alternative video label into area 10 to bind it to the short video. The user can also delete, modify, and other operations on inappropriate alternative video labels. In addition, the user can also add alternative video labels in the corresponding field.

[0044] In another scenario of labeling alternative videos for identifying express packages, which is similar to the live broadcast scenario, a target video can be captured at preset time intervals, and based on the video label processing method of the embodiment of the present application, the product name, delivery address, shipping address and other information corresponding to the express barcode of the express package appearing in each target video can be identified. Since one of the core requirements of this scenario is to be able to identify the product name of the express package, and the express barcode including the product name of the express package is attached to the surface of the express package, but occupies a small area, if the feature recognition of the product name is directly performed on the video frame, the recognition accuracy will be low due to the interference of the area outside the express barcode in the picture frame. Therefore, the embodiment of the present application can design a product category label video classification model under the product dimension, which can first determine the express barcode area in the video frame through feature recognition, and then extract the image features of the express barcode area, and then perform alternative video label matching based on the image features of the express barcode area. Since external interference is reduced, a more accurate matching result can be obtained. In addition, for the action-labeled video classification model, the moving trajectory label of the express package can be identified, and for the content-labeled video classification model, the video can be identified as the content form of the express package.

[0045] As reference Figure 3 , Figure 3 An interface diagram of a label processing method for identifying an express package video provided by an embodiment of the present application is shown. For a live broadcast of an express package video, a video clip is captured, which includes the sorting and moving process of package 1 and package 2, and the labels of the product dimension are identified: package 1: TV, package 2: mobile phone; the label of the action dimension: package 1: forward, package 2: left; the package identification label of the content dimension; in addition to displaying the video clip in the interface, alternative video labels under various dimensions can also be displayed. The user can drag the selected alternative video label into the area 21 corresponding to package 1 or the area 22 corresponding to package 2 to bind it to the video clip, and the bound alternative video label is displayed in the live video. The user can also delete, modify, and other operations on inappropriate alternative video labels. In addition, the user can also add alternative video labels in the corresponding field.

[0046] In addition, for the video classification model of the content classification dimension, it can identify labels based on multimodal fusion recognition technology, such as extracting text, sound, image and other features in the target video and fusing them into multimodal features. When performing label matching based on multimodal features, it can utilize richer and more modal information, thereby improving the accuracy of label matching.

[0047] Furthermore, after constructing the system of the multi-dimensional video classification model, the alternative video label generation module can use the target video as input and the label set output by each video classification model as the module output. Each label set can be further input into the label post-processing module for error correction. In the video labeling scenario of the embodiment of the present application, although the label sets corresponding to the multiple classification dimensions are divided, there is still a certain semantic correlation between the labels of different label sets. For example, if there is a label "breakfast machine" in the label set of the commodity dimension, then the probability of the label "nutritious breakfast" in the label set of the content dimension will be greater, while the probability of the label in the sports label set will be smaller.

[0048] Assume that among all the label sets output by the device selection video label generation module, there is a label in the product dimension label set: breakfast machine (confidence 80); there is a label in the content dimension label set: nutritious breakfast (confidence 70); there is a label in the sports label set: running (confidence 50); then according to the above definition, it can be known that there is a co-occurrence relationship between label: breakfast machine and label: nutritious breakfast, while there is no co-occurrence relationship between label: running and the other two labels. Therefore, according to the co-occurrence relationship between the alternative video labels output by two different video classification models, the confidence of label: breakfast machine and label: nutritious breakfast can be increased, and the confidence of label: running can be reduced, so as to correct the errors of the alternative video labels.

[0049] In addition to providing an interface for displaying the target video, the display module can further display the corrected alternative video tags, and provide corresponding interactive methods, allowing users to select alternative video tags to bind the target video, and add, delete, modify, etc. the alternative video tags.

[0050] In an embodiment of the present application, a multi-dimensional video classification model can be used to accurately extract alternative video tags that match short videos in multiple dimensions, and further utilize the co-occurrence relationship between the alternative video tags output by two different video classification models to correct the alternative video tags by analyzing the correlation between the identification tags in different classification dimensions, thereby improving the accuracy of the output alternative video tags. Finally, the short video and the corrected alternative video tags are displayed through the interface, thereby improving the intuitiveness and convenience of the video labeling process. The entire process is less dependent on manual labor, and the identification process of the alternative video tags is based on machine learning and does not rely on manual subjective judgment, thereby improving the accuracy of the labeling process.

[0051] Reference Figure 4 , which shows a flowchart of the steps of a video tag processing method provided by an embodiment of the present application, including:

[0052] Step 101, obtaining a target video.

[0053] In the embodiment of the present application, the target video may be a short video, a long video, a video clip captured from a continuous video stream, etc., and the embodiment of the present application is not limited to this.

[0054] Step 102: Input the target video into a plurality of different video classification models respectively, and obtain candidate video labels output by the video classification models.

[0055] In an embodiment of the present application, the various classification dimensions and the sets of specific alternative video tags covered by the alternative video tag system can be first determined according to the actual needs in the video production application scenario. For example, the classification dimensions such as action, commodity, content... can be divided according to actual needs. The set of alternative video tags in the action dimension may include: football, basketball, running, gymnastics... and other alternative video tags; the set of alternative video tags in the commodity dimension may include: football, basketball, breakfast machine, TV... and other alternative video tags; the set of alternative video tags in the content dimension may include: live streaming, singing, dancing, explaining, online classes... and other alternative video tags.

[0056] After dividing each classification dimension, the embodiment of the present application can design a video classification model corresponding to each classification dimension according to the characteristics of each classification dimension, so that the video classification model can process the characteristics of the corresponding classification dimension, thereby obtaining a higher-precision candidate video label output. The video classification model under each dimension has made corresponding improvements based on factors such as the data characteristics of the dimension and actual scene requirements. Furthermore, after constructing a system of multi-dimensional video classification models, the target video can be used as the input of each video classification model, and the set of candidate video labels output by each video classification model can be used as the output.

[0057] Step 103: Correct the candidate video tags according to the co-occurrence relationship between the candidate video tags output by two different video classification models.

[0058] In the video tagging scenario of the embodiment of the present application, although the tag sets corresponding to the multiple classification dimensions are divided, there is still a certain semantic correlation between the tags of different tag sets, and this semantic correlation can be expressed by the co-occurrence relationship between the alternative video tags.

[0059] Assume that among all the tag sets output by the device-selected video tag generation module, there is a tag in the tag set of the product dimension: breakfast machine; there is a tag in the tag set of the content dimension: nutritious breakfast; there is a tag in the sports tag set: running; then according to the above definition, it can be known that there is a co-occurrence relationship between the tag: breakfast machine and the tag: nutritious breakfast, while there is no co-occurrence relationship between the tag: running and the other two tags. Therefore, the error correction operation may include: One implementation method: for multiple candidate video tags with a co-occurrence relationship, increase their respective corresponding confidences, and for multiple candidate video tags without a co-occurrence relationship, reduce their respective corresponding confidences. Another implementation method: retain multiple candidate video tags with a co-occurrence relationship, and delete multiple candidate video tags without a co-occurrence relationship. Another implementation method: give priority to displaying multiple candidate video tags with a co-occurrence relationship.

[0060] Step 104: display the target video and the error-corrected alternative video labels in the interface.

[0061] In this step, in addition to displaying the target video by providing an interface, you can further display the corrected alternative video tags in the interface, and provide corresponding interactive methods, so that users can select alternative video tags to bind the target video, and add, delete, modify, etc. the alternative video tags.

[0062] After binding the target video with a video tag, the bound video tag can be displayed during the subsequent playback of the target video. Users can also use the video tag as a search term to quickly search for videos related to the search term. Of course, the video platform can also classify and manage videos according to the video tags bound to the videos, and can also implement classified recommendations for videos according to the bound video tags in the video recommendation process.

[0063] To summarize, in the embodiments of the present application, a multi-dimensional video classification model can be used to accurately extract alternative video tags that match short videos in multiple dimensions, and further utilize the co-occurrence relationship between the alternative video tags output by two different video classification models to correct the alternative video tags by analyzing the correlation between the identification tags in different classification dimensions, thereby improving the accuracy of the output alternative video tags. Finally, the short video and the corrected alternative video tags are displayed through the interface, thereby improving the intuitiveness and convenience of the video labeling process. The entire process is less dependent on manual labor, and the identification process of the alternative video tags is based on machine learning and does not rely on manual subjective judgment, thereby improving the accuracy of the labeling process.

[0064] Reference Figure 5 , shows a flowchart of another video tag processing method embodiment of the present application. It includes:

[0065] Step 201, obtaining a target video.

[0066] This step may specifically refer to the above step 101, which will not be described in detail here.

[0067] Step 202: Input the target video into a plurality of different video classification models respectively, and obtain candidate video labels output by the video classification models.

[0068] This step may specifically refer to the above step 102, which will not be described in detail here.

[0069] Optionally, when the video classification model is a video classification model under the action dimension, the video classification model includes a time series model; the step 202 includes:

[0070] Sub-step 2021: input the target video into the time series model to extract image features of the video frames of the target video.

[0071] Sub-step 2022: Encode the image features of the video frame according to the corresponding moment of the video frame on the playback time axis of the target video to obtain the encoded features.

[0072] Sub-step 2023: Obtain, based on the encoding features, candidate video tags for characterizing the category of the action in the target video.

[0073] In an embodiment of the present application, since action videos often use multiple consecutive frames to show the actions of characters, the video classification model under the action dimension is designed to be a time series model. The time series model can learn the changes between frames, so that the prediction results are more stable between adjacent frames. When extracting the features of the picture frames of the action video, the time series model will encode and combine the various features based on the timing of the feature corresponding to the picture frames (such as the features of multiple frames that make up a jumping action can be combined into one according to the timing coding) to obtain the coding features. Subsequently, when matching the alternative video labels based on this coding feature based on the timing coding combination, more accurate action label matching results can be obtained.

[0074] It should be noted that in the action dimension, when performing feature encoding, a 3D convolutional neural network (3D CNN) can be specifically selected for feature encoding. Through convolution operations and multi-layer neural networks, the accuracy and efficiency of encoding can be improved. The construction of the time series model includes: 1. Building a model based on a 3D CNN network, and expressing it by a set of learnable parameters with a specific structure. 2. Constructing a training set corresponding to the model, specifically including a label set, and a number of sample videos corresponding to each label. 3. Training the model, based on the training set and classification target, iteratively update the parameters of the model through back propagation and gradient descent method, so that it can correctly classify sample videos on the training set. Stop after a certain number of iterations to obtain the parameters of the final model.

[0075] In one implementation, the time series model can be specifically a time series model based on optical flow analysis. The optical flow method uses the changes in pixels in an image sequence in the time domain and the correlation between adjacent frames to find the correspondence between the previous frame and the current frame, thereby calculating the motion information of objects between adjacent frames. The time series model based on optical flow analysis can process dynamic action videos well to accurately extract features related to the action and identify the category of action features.

[0076] Optionally, when the video classification model is a video classification model under the commodity information dimension, the video classification model includes a commodity information detection model; the commodity information detection model includes a detector and a classifier; step 202 includes:

[0077] Sub-step 2024: input the target video into the product information detection model respectively, and extract the product information area in the video frame of the target video through the detector.

[0078] Sub-step 2025: extracting the regional features of the product information area through the classifier, and obtaining the candidate video tags for characterizing the product information in the target video based on the regional features.

[0079] In an embodiment of the present application, based on a scenario where product information needs to be identified, since the product information occupies a relatively small area in the overall picture, if the feature recognition of the product name is directly performed on the video picture frame, the recognition accuracy will be low due to the interference of areas outside the product information in the picture frame. Therefore, an embodiment of the present application can design a video classification model under the product dimension including a product information detection model; the product information detection model includes a detector and a classifier. Feature recognition can be first performed through the detector to determine the region of interest (ROI) where the product information in the video picture frame is located, and then image features of the region of interest are extracted, and alternative video label matching is performed based on the image features of the region of interest through the classifier. Since external interference is reduced, a more accurate matching result can be obtained, that is, accurate video features for characterizing product information are identified.

[0080] For example, in the scenario of express package identification, although the express barcode of the product name of the express package is attached to the surface of the express package, it occupies a small area. If the feature recognition of the product name is directly performed on the video frame, the recognition accuracy will be low due to the interference of the area outside the express barcode in the picture frame. Therefore, the embodiment of the present application can design a product category label video classification model under the product dimension. It can first determine the express barcode area in the video frame through feature recognition, and then extract the image features of the express barcode area, and then match the alternative video labels based on the image features of the express barcode area.

[0081] Step 203: updating the confidence of the candidate video tags according to the co-occurrence relationship between the candidate video tags output by two different video classification models.

[0082] The video classification model also outputs a confidence level corresponding to the candidate video label.

[0083] In the embodiment of the present application, the confidence of the candidate video tag reflects the degree of matching between the candidate video tag and the video. The greater the confidence, the more the candidate video tag matches the video. Since the co-occurrence relationship between candidate video tags of different dimensions has a direct relationship with the confidence of the candidate video tag, in one implementation, for multiple candidate video tags with a co-occurrence relationship, their respective corresponding confidences are improved, and for multiple candidate video tags without a co-occurrence relationship, the purpose of updating the confidence of the candidate video tag by using the co-occurrence relationship is achieved.

[0084] Optionally, step 203 may specifically include:

[0085] Sub-step 2031: obtaining the joint probability between the first candidate video label and all the second candidate video labels according to the confidence of the first candidate video label, the confidence of the second candidate video label, and the co-occurrence relationship value annotated for the corresponding relationship between the first candidate video label and the second candidate video label; the first candidate video label and the second candidate video label are candidate video labels output by two different video classification models.

[0086] In the embodiment of the present application, the confidence of the first candidate video label output by a video classification model can be expressed as {f j}, the confidence of the second candidate video label output by another video classification model can be represented by {g i}; and whether there is a co-occurrence relationship between the first candidate video tag and the second candidate video tag can be determined by pre-marking. For example, if there is a semantic association between the tag: nutritious breakfast and the tag: breakfast machine, the corresponding relationship between the two tags can be marked as having a co-occurrence relationship; if there is no semantic association between the tag: nutritious breakfast and the tag: running, the corresponding relationship between the two tags can be marked as not having a co-occurrence relationship. In addition, the co-occurrence relationship can be specifically quantified as a co-occurrence relationship value. If there is a co-occurrence relationship between the two tags, the co-occurrence relationship value of the corresponding relationship annotation between the two is 1; if there is no co-occurrence relationship between the two tags, the co-occurrence relationship value of the corresponding relationship annotation between the two is 0.

[0087] Through the above definition, we can see that there is a constraint of co-occurrence relationship between labels of different dimensions. This constraint can be specifically reflected in a bipartite graph. Figure 6 , which shows that the embodiment of the present application provides a bipartite graph, including: a confidence score of a first candidate video label output by a video classification model {f j}, the confidence of the second candidate video label output by another video classification model {g i}, in the existing data, there is a co-occurrence relationship between the two dimension labels, such as a data with both f j The corresponding first candidate video tag and g i The corresponding second candidate video label, then use the corresponding edge w {i,j} = 1 (i.e., the dotted line in the figure, which is also equivalent to the co-occurrence relationship value) to indicate that the two have a co-occurrence relationship, w {i,j} =0 means there is no co-occurrence relationship between the two.

[0088] Furthermore, the confidence of the first candidate video tag {f j}, the confidence of the second candidate video label {g i}, and the co-occurrence relationship value w for the corresponding relationship between the first candidate video tag and the second candidate video tag {i,j} , obtain the joint probability between the first candidate video label and the second candidate video label, the joint probability is used to characterize the probability of the two appearing at the same time, where the joint probability between a first candidate video label and a second candidate video label

[0089] Where Z is the normalization coefficient,

[0090] Sub-step 2032: taking the average value of the joint probabilities between the first candidate video label and all the second candidate video labels as the updated confidence of the first candidate video label.

[0091] In this step, the confidence of the first candidate video label after update is:

[0092]

[0093] It is expressed as taking the average value of the joint probability between the first candidate video label and all the second candidate video labels as the updated confidence of the first candidate video label.

[0094] Optionally, the method may further include:

[0095] Step A1: Obtain a preset bipartite graph for the first candidate video label and the second candidate video label.

[0096] In this step, in the embodiment of the present application, there is a constraint on the co-occurrence relationship between tags of different dimensions, and this constraint can be specifically embodied as follows: Figure 6 The bipartite graph shown in the figure includes: the confidence of the first candidate video label output by a video classification model {f j}, the confidence of the second candidate video label output by another video classification model {g i}, and the connecting line indicates that there is a co-occurrence relationship between two candidate video tags.

[0097] Step A2: when a co-occurrence relationship line is marked between the first candidate video tag and the second candidate video tag in the bipartite graph, the co-occurrence relationship value corresponding to the first candidate video tag and the second candidate video tag is determined as a first value.

[0098] Step A3: when no co-occurrence relationship line is marked between the first candidate video tag and the second candidate video tag in the bipartite graph, the co-occurrence relationship value corresponding to the first candidate video tag and the second candidate video tag is determined as a second value, and the first value and the second value are different values.

[0099] In the embodiments of this application, refer to Figure 6 , using the preset bipartite graph, since a data also has f j The corresponding first candidate video tag and g i The corresponding second candidate video label can be used with the corresponding edge w {i,j} = 1 (i.e., the dotted line in the figure, which is also equivalent to the co-occurrence relationship value) to indicate that the two have a co-occurrence relationship, w {i,j} =0 means there is no co-occurrence relationship between the two.

[0100] Step 204: Correct the candidate video tags according to the confidence level.

[0101] Optionally, in one implementation of the embodiment of the present application, error correction of alternative video tags using confidence can be achieved in the following manner: remove alternative video tags with confidence less than a confidence threshold. For example, in the alternative video tags after the confidence is updated, alternative video tags with confidence less than 50 can be removed to achieve the purpose of removing alternative video tags with poor semantic relevance to the video content. Alternatively, mark information is provided for alternative video tags with confidence less than the confidence threshold; the mark information is used to mark the alternative tag when the alternative video tag with confidence less than the confidence threshold is displayed in the interface. Marking information can also be provided for alternative video tags with confidence less than the confidence threshold, such as bold mark, highlight mark, warning mark, etc., to highlight the alternative tag while displaying the alternative video tag, reminding the user that the confidence of the alternative tag is very low. The user can then modify these alternative video tags or confirm deletion or confirmation to retain them.

[0102] Alternatively, alternative video tags with confidence less than the confidence threshold are displayed in a preset area of ​​the interface. In practical applications, an additional area can be set in addition to the area for displaying alternative video tags with confidence greater than or equal to the confidence threshold, in which alternative video tags with confidence less than the confidence threshold are displayed. The user can then modify these alternative video tags or confirm to delete or confirm to keep them.

[0103] In addition, refer to Figure 2 , assuming that after error correction, the confidence levels of the alternative video tags: character and the alternative video tags: story are both less than 50, then the embodiment of the present application may also display the alternative video tags whose confidence levels are less than the confidence threshold in the preset area 30 of the interface, and prompt that these video tags are low-confidence tags, so that the user can modify and edit these tags according to the prompts, such as deleting these tags or modifying the contents of these tags.

[0104] Step 205: display the target video and the error-corrected alternative video labels in the interface.

[0105] This step may specifically refer to the above step 101, which will not be described in detail here.

[0106] Optionally, step 205 may specifically include:

[0107] Sub-step 2051: display the candidate video tags under the classification dimension in the input box corresponding to the classification dimension in the interface.

[0108] Reference Figure 2 It provides a scenario for processing sports short video tags. Assume that for a short video of playing football, the football and jersey tags in the product dimension are identified; the football and running tags in the action dimension are identified; the sports video and scenery tags in the content dimension are identified; in addition to displaying the short video and the video name in the interface, the alternative video tags under each classification dimension can also be displayed, that is, by displaying the alternative video tags in the interface according to the input boxes corresponding to the classification dimensions, the alternative video tags in the interface are distinguished based on different dimensions, which improves the intuitiveness of the classification of the alternative video tags in the interface.

[0109] In addition, the interface may also sort the candidate video tags in descending order of confidence, so as to give priority to displaying the candidate video tags with high confidence.

[0110] Optionally, after step 205, the following steps may also be included:

[0111] Step 206: In response to the selection operation of the candidate video tag, at least one selected candidate video tag is bound to the target video.

[0112] In this step, refer to Figure 2 After the alternative video tags are displayed in the interface, the alternative video tags in the interface can be selected. The selection operation can be dragging the draggable alternative video tags into area 10 to bind with the short video, or directly clicking the alternative video tags to complete the selection operation. After the alternative video tags are bound to the target video, the content of the alternative video tags can be used as a description attribute of the target video for subsequent users who watch the target video to see.

[0113] Step 207: in response to the triggering operation on the candidate video tag, one or more operations of deleting, adding, and editing the candidate video tag are performed.

[0114] In the embodiment of the present application, the user can also perform related trigger operations on the candidate video tags in the interface according to actual needs, and perform one or more of the operations of deleting, adding, and editing the candidate video tags. Figure 2You can delete the corresponding alternative video tag by clicking the Delete control in the alternative video tag, or you can right-click the alternative video tag to display the Edit menu and select the Edit item in the Edit menu to edit the content in the alternative video tag, or you can select the Add item in the Edit menu to add a new alternative video tag.

[0115] Optionally, the video classification model further outputs the playback progress corresponding to the candidate video tag in the target video; step 205 may further specifically include:

[0116] Sub-step 2052: based on the correspondence between the candidate video tags and the playback progress, the target video is played on the interface and the candidate video tags corresponding to the playback progress are displayed.

[0117] In an embodiment of the present application, the video classification model outputs the alternative video label and also outputs the corresponding playback progress of the alternative video label in the target video. For example, when the video classification model detects that the alternative video label: car exists in the current picture when the target video is played to the time 00:20, the video classification model will output the alternative video label: car and the time 00:20 corresponding to the alternative video label: car.

[0118] Specifically, in the interface, when the target video is played, the alternative video tags corresponding to the playback progress can be displayed, for example, when the target video is played to the target playback progress, the alternative video tags corresponding to the target playback progress are highlighted in the tag area, or when the target video is played to the target playback progress, the alternative video tags corresponding to the target playback progress are displayed in the target video. The close correlation between the video tags and the target video is displayed in an intuitive way.

[0119] Furthermore, the playback progress corresponding to the alternative video tag can be a moment, such as the moment corresponding to the xth frame of the target video identifying the alternative video tag; the playback progress corresponding to the alternative video tag can be a time period, such as the time period corresponding to the video segment consisting of the xth frame to the yth frame of the target video identifying the alternative video tag. In this way, when the user drags the play bar of the target video, the displayed alternative video tags will also change as the user drags. For example, when dragging to segment a, only the alternative video tags corresponding to segment a are displayed, and when dragging to another segment b, only the alternative video tags corresponding to segment b are displayed, so that the user can determine whether the tag binding is appropriate in an intuitive way.

[0120] Optionally, sub-step 2052 may further include:

[0121] Sub-step B1: when the target video in the interface is played to the target playback progress, display the alternative video tags corresponding to the target playback progress according to the corresponding relationship.

[0122] In one implementation of the embodiment of the present application, when the target video in the interface is played to the target playback progress, the alternative video tags corresponding to the target playback progress can be displayed according to the corresponding relationship, so that the user can accurately locate the playback progress corresponding to each alternative video tag during the playback of the target video, thereby conveniently analyzing whether the alternative video tags are accurate under the playback progress, thereby improving the efficiency of tagging. In this way, the alternative video tags can be displayed according to the playback progress, and the alternative video tags that do not correspond to the playback progress can be not displayed at the playback progress moment.

[0123] Reference Figure 7 , which shows an interface diagram of another label processing method for sports short videos provided by an embodiment of the present application. For a short video of playing football, at 00:20, the football and jersey labels of the product dimension and the football playing label of the action dimension are identified; at 02:20, the running label of the action dimension is identified; at 01:40, the sports video and scenery labels of the content dimension are identified; when the video classification model outputs these alternative video labels, it also outputs the playback progress corresponding to the alternative video labels, and displays the alternative video labels with the corresponding playback progress on the playback progress bar 40.

[0124] Furthermore, when the short video in the interface is currently playing to 00:20, the short video can display the football and jersey labels of the product dimension and the football kicking label of the action dimension corresponding to the playback progress 00:20 according to the corresponding relationship between the alternative video labels and the playback progress. In addition, in the classification display area of ​​the labels, the label corresponding to the playback progress 00:20 can also be highlighted and displayed, so as to facilitate the user to analyze and correct the labels. Assuming that the user subsequently drags the progress bar to 02:20, the sports video and scenery labels of the content dimension corresponding to the playback progress 02:20 can be displayed in the short video, so that the displayed labels change as the progress bar is dragged.

[0125] Optionally, sub-step 2052 may further include:

[0126] Sub-step B2: playing the target video and the alternative video tags in the interface, and when the target video is played to the target playback progress, highlighting the alternative video tags corresponding to the target playback progress according to the corresponding relationship.

[0127] In another implementation method of the embodiment of the present application, while the target video is being played in the interface, all alternative video tags can also be displayed, and when the target video is played to the target playback progress, the alternative video tags corresponding to the target playback progress are highlighted according to the corresponding relationship, thereby facilitating analysis of whether the alternative video tags are accurate under the playback progress, thereby improving the efficiency of labeling.

[0128] Reference Figure 8 , which shows an interface diagram of another label processing method for sports short videos provided by an embodiment of the present application. For a short video of playing football, at 00:20, the football and jersey labels of the product dimension and the football playing label of the action dimension are identified; at 02:20, the running label of the action dimension is identified; at 01:40, the sports video and scenery labels of the content dimension are identified; when the video classification model outputs these alternative video labels, it also outputs the playback progress corresponding to the alternative video labels, and displays the alternative video labels with the corresponding playback progress on the playback progress bar 40.

[0129] Furthermore, during the video playback in the interface, all the alternative video tags can be displayed. When the video is played to 00:20, the short video can highlight and display the football and jersey tags of the product dimension and the football kicking tag of the action dimension corresponding to the playback progress of 00:20 in the short video and tag classification display area according to the corresponding relationship between the alternative video tags and the playback progress, so as to facilitate the user to analyze and correct the tags. Assuming that the user subsequently drags the progress bar to 02:20, the sports video and scenery tags of the content dimension corresponding to the playback progress of 02:20 can be highlighted and displayed in the short video and tag classification display area, so that the displayed tags change as the progress bar is dragged.

[0130] Among them, the highlighting and lighting display methods include but are not limited to bolding the label frame, highlighting the label, and adding a background display light to the label.

[0131] To summarize, in the embodiments of the present application, a multi-dimensional video classification model can be used to accurately extract alternative video tags that match short videos in multiple dimensions, and further utilize the co-occurrence relationship between the alternative video tags output by two different video classification models to correct the alternative video tags by analyzing the correlation between the identification tags in different classification dimensions, thereby improving the accuracy of the output alternative video tags. Finally, the short video and the corrected alternative video tags are displayed through the interface, thereby improving the intuitiveness and convenience of the video labeling process. The entire process is less dependent on manual labor, and the identification process of the alternative video tags is based on machine learning and does not rely on manual subjective judgment, thereby improving the accuracy of the labeling process.

[0132] Reference Fig. 9 , which shows a block diagram of a video tag processing device provided by an embodiment of the present application, including:

[0133] An acquisition module 301 is used to acquire a target video;

[0134] A generation module 302 is used to input the target video into a plurality of different video classification models respectively, and obtain candidate video labels output by the video classification models;

[0135] An error correction module 303, configured to correct errors of the candidate video tags according to the co-occurrence relationship between the candidate video tags output by two different video classification models;

[0136] The display module 304 is used to display the target video and the error-corrected candidate video tags in the interface.

[0137] Optionally, the video classification model further outputs a confidence level corresponding to the candidate video label; the error correction module 303 includes:

[0138] An updating submodule, used for updating the confidence of the candidate video tags according to the co-occurrence relationship between the candidate video tags output by two different video classification models;

[0139] The error correction submodule is used to correct the error of the candidate video label according to the confidence level.

[0140] Optionally, the updating submodule includes:

[0141] A first calculation unit is used to obtain a joint probability between the first candidate video label and all the second candidate video labels according to the confidence of the first candidate video label, the confidence of the second candidate video label, and the co-occurrence relationship value annotated for the corresponding relationship between the first candidate video label and the second candidate video label; the first candidate video label and the second candidate video label are candidate video labels output by two different video classification models;

[0142] The second calculation unit is configured to use an average value of the joint probabilities between the first candidate video label and all the second candidate video labels as the updated confidence of the first candidate video label.

[0143] Optionally, the device further comprises:

[0144] A bipartite graph module, used to obtain a preset bipartite graph for the first candidate video label and the second candidate video label;

[0145] A first calculation module is used to determine the co-occurrence relationship value corresponding to the first candidate video tag and the second candidate video tag as a first value when a co-occurrence relationship line is marked between the first candidate video tag and the second candidate video tag in the bipartite graph;

[0146] The second calculation module is used to determine the co-occurrence relationship value corresponding to the first candidate video tag and the second candidate video tag as a second value when no co-occurrence relationship line is marked between the first candidate video tag and the second candidate video tag in the bipartite graph, and the first value and the second value are different values.

[0147] Optionally, the error correction submodule includes:

[0148] The screening unit is used to remove the candidate video tags whose confidence is less than the confidence threshold, or to display the candidate video tags whose confidence is less than the confidence threshold in a preset area of ​​the interface.

[0149] Optionally, different video classification models correspond to different classification dimensions, and the display module 404 includes:

[0150] The display submodule is used to display the candidate video tags under the classification dimension in the input box corresponding to the classification dimension in the interface.

[0151] Optionally, the device further comprises:

[0152] The first response module is used to bind at least one selected candidate video tag to the target video in response to the selection operation of the candidate video tag.

[0153] Optionally, the device further comprises:

[0154] The second response module is used to execute one or more operations of deleting, adding, and editing the candidate video tag in response to a triggering operation on the candidate video tag.

[0155] Optionally, when the video classification model is a video classification model under the action dimension, the video classification model includes a time series model;

[0156] The generating module 302 includes:

[0157] A first input submodule, used to input the target video into the time series model to extract image features of video frames of the target video;

[0158] An encoding submodule, used for encoding the image features of the video frame according to the corresponding moment of the video frame on the playback time axis of the target video to obtain encoding features;

[0159] The first matching submodule is used to obtain a candidate video tag for characterizing a category of an action in the target video according to the encoding feature.

[0160] Optionally, when the video classification model is a video classification model under the commodity information dimension, the video classification model includes a commodity information detection model; the commodity information detection model includes a detector and a classifier;

[0161] The generating module 302 includes:

[0162] A second input submodule, used to input the target video into the commodity information detection model respectively, and extract the commodity information area in the video frame of the target video through the detector;

[0163] The second matching submodule is used to extract the regional features of the commodity information area through the classifier, and obtain the candidate video tags for representing the commodity information in the target video according to the regional features.

[0164] Optionally, the video classification model further outputs the playback progress corresponding to the candidate video tag in the target video;

[0165] The display module 304 includes:

[0166] The display submodule is used to play the target video and display the alternative video tags corresponding to the playback progress on the interface according to the corresponding relationship between the alternative video tags and the playback progress.

[0167] Optionally, the display submodule includes:

[0168] The first display unit is used to display the candidate video tags corresponding to the target playback progress according to the corresponding relationship when the target video in the interface is played to the target playback progress.

[0169] Optionally, the display submodule includes:

[0170] The second display unit plays the target video and the alternative video tag in the interface, and when the target video is played to the target playback progress, the alternative video tag corresponding to the target playback progress is highlighted according to the corresponding relationship.

[0171] In summary, in the embodiments of the present application, a multi-dimensional video classification model can be used to accurately extract alternative video tags that match short videos in multiple dimensions, and further utilize the co-occurrence relationship between the alternative video tags output by two different video classification models. By analyzing the correlation between the identification tags in different classification dimensions, the alternative video tags can be corrected to improve the accuracy of the output alternative video tags. Finally, the short video and the corrected alternative video tags are displayed through the interface, which improves the intuitiveness and convenience of the video labeling process. The entire process is less dependent on manual labor, and the identification process of the alternative video tags is based on machine learning, and does not rely on manual subjective judgment, thereby improving the accuracy of the labeling process.

[0172] The embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0173] The present application embodiment provides one or more machine-readable media on which instructions are stored, and when executed by one or more processors, an electronic device executes one or more of the methods described in the above embodiments. In the present application embodiment, the electronic device includes various types of devices such as terminal devices and server terminals (clusters).

[0174] The embodiments of the present disclosure may be implemented as a device that uses any appropriate hardware, firmware, software, or any combination thereof to perform desired configurations, and the device may include electronic devices such as terminal devices and server (cluster). Fig.10 An exemplary device 1000 that can be used to implement various embodiments described in the embodiments of the present application is schematically shown.

[0175] For one embodiment, Fig.10 An exemplary apparatus 1000 is shown having one or more processors 1002, a control module (chip set) 1004 coupled to at least one of the (one or more) processors 1002, a memory 1006 coupled to the control module 1004, a non-volatile memory (NVM) / storage device 1008 coupled to the control module 1004, one or more input / output devices 1010 coupled to the control module 1004, and a network interface 1012 coupled to the control module 1004.

[0176] The processor 1002 may include one or more single-core or multi-core processors, and the processor 1002 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.). In some embodiments, the device 1000 can be used as a terminal device, a server (cluster), and other devices described in the embodiments of the present application.

[0177] In some embodiments, the apparatus 1000 may include one or more computer-readable media (e.g., memory 1006 or NVM / storage device 1008) having instructions 1014 and one or more processors 1002 combined with the one or more computer-readable media and configured to execute the instructions 1014 to implement a module to perform the actions described in the present disclosure.

[0178] For one embodiment, the control module 1004 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 1002 and / or any suitable device or component in communication with the control module 1004 .

[0179] The control module 1004 may include a memory controller module to provide an interface to the memory 1006. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0180] The memory 1006 may be used, for example, to load and store data and / or instructions 1014 for the device 1000. For one embodiment, the memory 1006 may include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the memory 1006 may include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

[0181] For one embodiment, control module 1004 may include one or more input / output controllers to provide an interface to NVM / storage device 1008 and input / output device(s) 1010 .

[0182] For example, NVM / storage 1008 may be used to store data and / or instructions 1014. NVM / storage 1008 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).

[0183] NVM / storage device 1008 may include storage resources that are physically part of the device on which apparatus 1000 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 1008 may be accessed via input / output device(s) 1010 over a network.

[0184] (One or more) input / output devices 1010 may provide an interface for the apparatus 1000 to communicate with any other appropriate device, and the input / output device 1010 may include a communication component, an audio component, a sensor component, etc. The network interface 1012 may provide an interface for the apparatus 1000 to communicate through one or more networks, and the apparatus 1000 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof for wireless communication.

[0185] For one embodiment, at least one of the processor(s) 1002 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 1004. For one embodiment, at least one of the processor(s) 1002 may be packaged together with the logic of one or more controllers of the control module 1004 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 1002 may be integrated on the same die with the logic of one or more controllers of the control module 1004. For one embodiment, at least one of the processor(s) 1002 may be integrated on the same die with the logic of one or more controllers of the control module 1004 to form a system-on-chip (SoC).

[0186] In various embodiments, the device 1000 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, the device 1000 may have more or fewer components and / or different architectures. For example, in some embodiments, the device 1000 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0187] Among them, the main control chip can be used as a processor or control module in the detection device, sensor data, location information, etc. are stored in a memory or NVM / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.

[0188] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0189] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0190] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0191] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0192] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0193] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0194] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0195] The above is a detailed introduction to a video tag processing method, device, electronic device and storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A video tag processing method, characterized in that: The method comprises: Get the target video; Inputting the target video into a plurality of different video classification models respectively, and obtaining candidate video labels output by the video classification models; Correcting the candidate video tags according to the co-occurrence relationship between the candidate video tags output by two different video classification models; Displaying the target video and the labels of the alternative videos after error correction in the interface; The video classification model also outputs a confidence level corresponding to the candidate video label; The step of correcting the candidate video tags according to the co-occurrence relationship between the candidate video tags output by two different video classification models includes: Improve the confidence of multiple candidate video labels that have a co-occurrence relationship, and reduce the confidence of multiple candidate video labels that do not have a co-occurrence relationship; According to the updated confidence level, error correction is performed on the candidate video label.

2. The method according to claim 1, characterized in that The updating of the confidence of the candidate video tags according to the co-occurrence relationship between the candidate video tags output by two different video classification models includes: According to the confidence of the first candidate video label, the confidence of the second candidate video label, and the co-occurrence relationship value annotated for the corresponding relationship between the first candidate video label and the second candidate video label, the joint probability between the first candidate video label and all the second candidate video labels is obtained; the first candidate video label and the second candidate video label are candidate video labels output by two different video classification models; An average value of the joint probabilities between the first candidate video label and all the second candidate video labels is used as the updated confidence of the first candidate video label.

3. The method according to claim 1, characterized in that The step of correcting the candidate video tag according to the confidence level includes: Remove candidate video labels whose confidence is less than the confidence threshold; Or provide marking information for candidate video tags whose confidence is less than the confidence threshold; the marking information is used to mark the candidate video tags when displaying the candidate video tags whose confidence is less than the confidence threshold in the interface.

4. The method according to claim 1, characterized in that: Different video classification models correspond to different classification dimensions. The candidate video labels after displaying error correction include: The alternative video tags under the classification dimension are displayed in the input box corresponding to the classification dimension in the interface.

5. The method according to claim 1, characterized in that: The video classification model also outputs the playback progress corresponding to the candidate video tag in the target video; The displaying of the target video and the corrected alternative video labels in the interface includes: According to the correspondence between the candidate video tags and the playback progress, the target video is played on the interface and the candidate video tags corresponding to the playback progress are displayed.

6. The method according to claim 5, characterized in that The step of playing the target video and displaying the alternative video tags corresponding to the playback progress on the interface according to the corresponding relationship between the alternative video tags and the playback progress includes: When the target video in the interface is played to the target playback progress, the alternative video tags corresponding to the target playback progress are displayed according to the corresponding relationship.

7. The method according to claim 5, characterized in that The step of playing the target video and displaying the alternative video tags corresponding to the playback progress on the interface according to the corresponding relationship between the alternative video tags and the playback progress includes: The target video and the alternative video tags are played in the interface, and when the target video is played to the target playback progress, the alternative video tags corresponding to the target playback progress are highlighted according to the corresponding relationship.

8. The method according to claim 1, characterized in that After displaying the error-corrected candidate video tags, the method further includes: In response to a selection operation on the candidate video tag, at least one selected candidate video tag is bound to the target video.

9. The method according to claim 1, characterized in that: In the case where the video classification model is a video classification model under the action dimension, the video classification model includes a time series model; The step of inputting the target video into a plurality of different video classification models respectively and obtaining candidate video labels output by the video classification models comprises: Inputting the target video into the time series model to extract image features of video frames of the target video; Encoding the image features of the video frame according to the corresponding moment of the video frame on the playback time axis of the target video to obtain the encoded features; According to the encoding feature, a candidate video tag for characterizing a category of an action in the target video is obtained.

10. The method according to claim 1, characterized in that In the case where the video classification model is a video classification model under the commodity information dimension, the video classification model includes a commodity information detection model; the commodity information detection model includes a detector and a classifier; The step of inputting the target video into a plurality of different video classification models respectively and obtaining candidate video labels output by the video classification models comprises: Inputting the target videos into the commodity information detection model respectively, and extracting the commodity information area in the video frame of the target video through the detector; The classifier is used to extract the regional features of the commodity information region, and based on the regional features, a candidate video tag for characterizing the commodity information in the target video is obtained.

11. A video tag processing device, characterized in that: The device comprises: An acquisition module, used to acquire a target video; A generation module, used to input the target video into a plurality of different video classification models respectively, and obtain candidate video labels output by the video classification models; An error correction module, configured to correct errors of the candidate video tags according to the co-occurrence relationship between the candidate video tags output by two different video classification models; A display module, used to display the target video and the error-corrected alternative video labels in the interface; The error correction module is specifically used to improve the confidence of multiple candidate video tags with co-occurrence relationship and reduce the confidence of multiple candidate video tags without co-occurrence relationship; and correct the candidate video tags according to the updated confidence.

12. An electronic device, characterized in that: include: processor; and A memory having executable codes stored thereon, which, when executed, causes the processor to execute the method according to any one of claims 1 to 10.

13. One or more machine-readable media having executable codes stored thereon, which, when executed, cause a processor to perform the method according to any one of claims 1 to 10.

Images