Video labeling method and device, equipment, medium and product

Abstract

This application relates to a video annotation method, apparatus, device, medium, and product. The method includes: acquiring media information of a video to be annotated, the media information including image data and descriptive text of the video; using a video classification model to determine the confidence level of each category label mapped from the image data to a category label set, and identifying category labels with confidence levels exceeding a preset confidence threshold as target category labels; using category labels with confidence levels below the confidence threshold as pending category labels, and verifying whether the pending category labels are target category labels based on the descriptive text; and annotating the video to be annotated using the target category labels. By using the descriptive text of the video to be annotated to verify and confirm the low-confidence category labels identified in the video classification model, the accuracy of video annotation can be improved, overcoming the long-tail effect caused by insufficient training samples for category labels, and also improving the utilization efficiency of video information resources.

Description

Technical Field

[0001] This application relates to video information processing technology, and more particularly to a video annotation method and its apparatus, equipment, medium, and product. Background Technology

[0002] Users and user-uploaded videos are the two fundamental elements of short videos. Users engage in behaviors such as watching, commenting, liking, and sharing videos, and they also follow and send private messages to other users. The goals of short video platforms are: 1. To create closer connections between users, increasing their sense of participation; 2. To allow users to access video content that interests them, achieving emotional satisfaction, and further fostering user-to-user following and engagement.

[0003] The type of videos delivered to users is crucial for user retention, thus requiring the identification of user and video interest categories. However, user interest categories are primarily derived from the tags of videos that users have uploaded, watched, commented on, liked, and shared; therefore, tagging videos based on their interest categories is fundamental.

[0004] Due to the need for refined operations, video tagging typically employs multi-level labeling, with hundreds of labels. In practice, tagging videos presents numerous challenges, such as:

[0005] On the one hand, it is extremely difficult to label videos using a video classification model alone. Moreover, the amount of training sample data required to train a video classification model often exceeds millions, but it can only obtain classification capabilities for hundreds of categories. If the labels need to be expanded, the training samples need to be re-labeled, which is extremely costly. Furthermore, the accuracy of the video classification model itself is not high, and infinitely increasing the number of categories will lead to a significant decrease in its classification accuracy.

[0006] On the other hand, video classification has a long-tail effect, with a small number of training samples for certain categories. During the training of the video classification model, the lack of sufficient training samples will cause the video classification model to fail to accurately identify the corresponding category of video, and thus fail to achieve accurate classification of such videos.

[0007] On the other hand, user-uploaded videos contain a large amount of relevant information, but the quality of this information varies greatly. As a result, traditional tagging methods often overlook this information and fail to effectively mine this relevant information, leading to a waste of useful information resources. Summary of the Invention

[0008] The purpose of this application is to solve the above-mentioned problems by providing a video annotation method and corresponding apparatus, devices, non-volatile readable storage media, and computer program products.

[0009] According to one aspect of this application, a video annotation method is provided, comprising the following steps:

[0010] Obtain the media information of the video to be labeled, including the image data and descriptive text of the video to be labeled;

[0011] A video classification model is used to determine the confidence level of each category label in the image data mapping to the category label set, and the category labels with confidence levels exceeding a preset confidence threshold are determined as target category labels;

[0012] Category labels whose confidence level does not reach the confidence threshold are designated as pending category labels, and the description text is used to verify whether the pending category labels are target category labels.

[0013] The target category label is used to label the video to be labeled.

[0014] According to another aspect of this application, a video annotation apparatus is provided, comprising:

[0015] The information acquisition module is configured to acquire media information of the video to be labeled, the media information including image data and descriptive text of the video to be labeled;

[0016] The default classification module is set to use a video classification model to determine the confidence level of each category label in the category label set that the image data is mapped to, and to determine the category label that has a confidence level exceeding a preset confidence level threshold as the target category label;

[0017] The auxiliary verification module is configured to use category labels whose confidence level does not reach the confidence threshold as pending category labels, and verify whether the pending category labels are target category labels based on the description text;

[0018] The video annotation module is configured to annotate the video to be annotated using the target category label.

[0019] According to another aspect of this application, a video annotation device is provided, including a central processing unit and a memory, wherein the central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the video annotation method described in this application.

[0020] According to another aspect of this application, a non-volatile readable storage medium is provided, which stores a computer program implemented according to the video annotation method in the form of computer-readable instructions, wherein the computer program, when invoked by a computer, performs the steps included in the method.

[0021] According to another aspect of this application, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the method described in any embodiment of this application.

[0022] Compared with existing technologies, this application has several technological advantages, including but not limited to:

[0023] First, after the video classification model predicts the corresponding category label for the video to be labeled, the category label with higher confidence is directly determined as the target category label for the video to be labeled. For the category label with lower confidence, the descriptive text in the media information of the video to be labeled is used as reference information to verify these category labels to confirm whether they belong to the target category label. This improves the accuracy of video labeling based on the video classification model without increasing the high training cost.

[0024] Secondly, since descriptive text in media information is usually used to introduce the content of the video, using descriptive text to verify the low-confidence category labels predicted by the video classification model can effectively overcome the long-tail effect. For some category labels that the video classification model cannot effectively distinguish due to a lack of sufficient training samples, the ability to recognize these category labels can be improved with the help of descriptive text.

[0025] Finally, the descriptive text of a video often contains information about its content, value, and characteristics. In the process of tagging videos, making good use of its descriptive text for data mining can enhance the utilization value of information resources and avoid wasting information resources. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 A schematic diagram of the network architecture corresponding to the video service used in this application;

[0028] Figure 2 This is a flowchart illustrating one embodiment of the video annotation method of this application;

[0029] Figure 3 This is a schematic diagram of an exemplary network architecture for the video classification model used in this application;

[0030] Figure 4This is another exemplary network architecture diagram of the video classification model used in this application;

[0031] Figure 5 This is another exemplary network architecture diagram of the video classification model used in this application;

[0032] Figure 6 This is a schematic diagram illustrating the process by which a video classification model determines the target category label based on image data in an embodiment of this application.

[0033] Figure 7 This is a flowchart illustrating the process of verifying undetermined category labels based on word frequency statistics in an embodiment of this application.

[0034] Figure 8 This is a flowchart illustrating the process of constructing word frequency statistics tables corresponding to each category label in the embodiments of this application;

[0035] Figure 9 This is a schematic diagram illustrating the process of using a text classification model to verify undetermined category labels in an embodiment of this application;

[0036] Figure 10 This is a flowchart illustrating the further application of target category tags to the videos to be labeled in this embodiment of the application;

[0037] Figure 11 This is a schematic block diagram of the video annotation device of this application;

[0038] Figure 12 This is a schematic diagram of the structure of a video annotation device used in this application. Detailed Implementation

[0039] The models cited or potentially cited in this application, including traditional machine learning models or deep learning models, can be deployed on a remote server and invoked remotely on the client, or deployed on a client with the capability to invoke directly, unless explicitly specified in the text. In some embodiments, when running on the client, the corresponding intelligence can be obtained through transfer learning in order to reduce the requirements on the client's hardware operating resources and avoid excessive occupation of the client's hardware operating resources.

[0040] Please see Figure 1 The network architecture used in the exemplary application scenario of this application can be used to deploy video services, allowing users to upload various videos and push videos with corresponding category tags to users based on category tags representing user interests. The videos can be short videos, or mini-videos, or longer videos such as movies or TV series, or live videos of varying lengths generated by user live streaming. Figure 1The application server 81 shown can be used to support the operation of the video service, while the media server 82 can be used to store or forward user videos. Terminal devices such as computers 83 and mobile phones 84, acting as clients, are generally provided to end users and can be used to upload or download videos. The method or apparatus of this application can run on the application server 81, media server 82, or other computer devices that can access the video, in order to implement the retrieval and annotation of the video.

[0041] Please see Figure 2 According to a video annotation method provided in this application, in one embodiment, the method includes the following steps:

[0042] Step S1100: Obtain the media information of the video to be labeled, wherein the media information includes the image data and descriptive text of the video to be labeled;

[0043] Videos in video services are typically stored on servers as video files. Each video file is associated with a descriptive text file, which describes its content, value, and characteristics. This descriptive text is usually set by the user. Each video is typically assigned a unique identifier, allowing the retrieval of the corresponding video, including its video file and descriptive text. For user-uploaded videos, they are usually also associated with a corresponding user identifier to identify the user of each video.

[0044] To meet the requirements of this application, a set of category labels can be set up according to certain classification standards. The videos of this application are then labeled using the category labels in the set, a process commonly known as tagging. When tagging videos, a video is identified as the video to be tagged. Based on the media information of the video to be tagged, its corresponding target category label in the set of category labels is determined. These target category labels are then associated with the video to be tagged, thus realizing the tagging operation of the video to be tagged.

[0045] In one embodiment, the media information of the video to be labeled includes at least image data and descriptive text of the video. The image data of the video can be extracted from the video file of the video to be labeled, such as multiple image frames. The descriptive text of the video to be labeled is associated with the corresponding descriptive text of the corresponding video file. In other embodiments not detailed, adapting to different labeling principles, the media information may further include other related information of the video to be labeled, such as the feature identifier of the user who uploaded the video to be labeled, the playback duration of the video to be labeled, or any other related information that can provide reference value for the reasoning process of category labeling.

[0046] In one embodiment, the timing for obtaining media information of the video to be labeled can be automatically triggered after the user completes the upload of the video to be labeled. In another embodiment, it can be triggered by a pre-set timed task of the video service, or it can be triggered by the active request of the user who uploaded the video to be labeled.

[0047] Step S1200: Use a video classification model to determine the confidence level of each category label in the category label set that the image data is mapped to, and determine the category label with a confidence level exceeding a preset confidence level threshold as the target category label;

[0048] After obtaining the media information of the video to be labeled, a video classification model can be used. According to the input parameter requirements of the video classification model, some or all of the information in the media information can be selected to construct the input of the video classification model. The video classification model can then infer based on the input information and predict the confidence level of each category label in the preset category label set. This allows the category label that the video to be labeled should be identified based on the confidence level of each category label.

[0049] The video classification model described typically employs a convolutional neural network model. It extracts deep semantic information from the input information and then maps this deep semantic information through a fully connected layer to a classification space corresponding to the category label set. The categories in the classification space correspond one-to-one with the category labels in the category label set, thereby obtaining the confidence level corresponding to each category label mapped from the deep semantic information. It is easy to understand that the video classification model is pre-trained to convergence using sufficient training samples, enabling it to learn the ability to infer and determine the confidence level of each category label based on the corresponding input, before being used in the technical solution of this application.

[0050] In one embodiment, the video classification model is constructed as a unimodal video classification model, which relies solely on image data from the media information of the video to be labeled for category label inference. Therefore, it can be as follows: Figure 3 As shown, a convolutional neural network is used to extract deep semantic information from image data, specifically from several image frames. A classifier is then used to perform classification mapping, determining the confidence level of each category label as the classification result. This video classification model, based on a single modality, has relatively low computational cost, controllable training cost, and high training efficiency.

[0051] In another embodiment, such as Figure 4As shown, the video classification model is constructed as a multimodal video classification model. It receives image data from the media information as one input and uses a convolutional neural network to extract image feature information. It also receives descriptive text from the media information as another input and uses a text feature extractor to extract text feature information. Then, a concatenation layer combines the image and text feature information to form deep semantic information. A subsequent classifier performs classification mapping to obtain the classification result and determines the confidence level of each category label. Because the video classification model is based on multiple sources of information that are cross-referenced during inference, it is expected to achieve higher prediction accuracy.

[0052] In another embodiment, when the media information includes other related information besides the descriptive text, this related information can also be used as supplementary information to the descriptive text and merged into the descriptive text for unified encoding to construct the input of a multimodal video classification model, and then processed accordingly through the multimodal video classification model.

[0053] In another embodiment, considering that the image data of the video to be labeled includes multiple image frames, and that there is an order relationship between these image frames, implying contextual association, in order to utilize the contextual information, a recurrent neural network can be connected after the convolutional neural network used in the video classification models of the above embodiments. This recurrent neural network is used to perform contextual analysis on the comprehensive feature information obtained by concatenating the image feature information output by the convolutional neural network and the descriptive text output by the text feature extractor, thereby obtaining the corresponding deep semantic information, such as... Figure 5 As shown, the data is then input into a classifier for classification mapping, in order to further reference the context in the image data and descriptive text to improve the prediction accuracy of the video classification model.

[0054] In the above embodiments, the convolutional neural network (CNN) used to extract features from the image data can be any neural network model suitable for processing image features, developed based on the convolutional neural network, such as the ResNet (Residual Network) residual network model; and the text feature extractor used to extract features from descriptive text can be implemented based on a recurrent neural network (RNN). The recurrent neural network described in this application can also be any neural network suitable for extracting text features, developed based on the recurrent neural network, such as LSTM (Long Short-Term Memory) network, Transformer series encoders, etc.

[0055] As can be understood from the above description of different embodiments, the video classification models implemented in different embodiments can all serve the implementation of this application, and are used to determine the confidence level of the image data of the video to be labeled to each category label in the category label set.

[0056] When using any of the video classification models disclosed in any embodiment, and performing classification mapping based on input information including the video image data to be labeled, and determining the confidence level of each category label mapped to the category label set, it is easy to understand that category labels with higher confidence levels are more reliable in terms of accuracy. Therefore, a confidence threshold can be preset, and the confidence level of each category label can be compared with this confidence threshold. Category labels with confidence levels exceeding the confidence threshold are determined as suitable target category labels for labeling the video to be labeled. Category labels with confidence levels not exceeding the confidence threshold can be left unprocessed in this step. The confidence threshold is an empirical or experimental threshold and can be flexibly set as needed.

[0057] It should be noted that, in one embodiment, the category tags in the category tag set of this application can be single-level category tags; in another embodiment, they can be multi-level category tags, that is, each level includes multiple category tags, and each superior category tag can include multiple subordinate category tags. Category tags at each level can be divided according to different criteria such as interest, function, or domain. A vertical category tag structure is formed by tracing back along a category tag at its lowest level towards its superior level. Each vertical category tag structure can be represented by the feature identifier corresponding to its lowest-level category tag or by a vector composed of the feature identifiers of each level of category tags along that path.

[0058] Step S1300: Using category labels whose confidence level does not reach the confidence threshold as pending category labels, verify whether the pending category labels are target category labels based on the description text;

[0059] For category labels whose confidence level determined by the video classification model does not reach the aforementioned confidence threshold, these category labels can be designated as pending category labels. This indicates that, given their confidence level is below the threshold, it cannot be confirmed whether they can be included in the target category label category, and therefore further verification is required. The reason for further verification of pending category labels is that the video classification model may have insufficient training or inference ability, or the existence of long-tail effects, leading to inaccurate judgments of these low-confidence category labels. Therefore, these pending category labels serve as a fallback to help the video classification model improve the accuracy of labeling the videos to be labeled.

[0060] As mentioned earlier, the media information prepared for labeling also includes descriptive text corresponding to the video to be labeled. This descriptive text is typically a brief summary of the content, characteristics, and value of the video, thus providing valuable information for determining the category label for the video. Therefore, the descriptive text can be used to assist in verifying whether the undetermined category label predicted by the video classification model is suitable as the target category label.

[0061] In one embodiment, a rule-based matching method can be used to query whether the text corresponding to the pending category tag exists in the description text. If it exists, the pending category tag can be directly confirmed as the target category tag; otherwise, it is not the target category tag. This method of verifying pending category tags is efficient and fast.

[0062] In another embodiment, the data distance between the text vector of the undetermined category label and the text vector of the descriptive text can be calculated based on semantic matching. If the data distance exceeds a preset threshold, the undetermined category label is confirmed as the target category label; otherwise, it is not. This method of verifying the undetermined category label strikes a balance between efficiency and accuracy, making it economical and practical.

[0063] In another embodiment, the following steps can be set: when the confidence level of the undetermined category label reaches the confidence level threshold, the hit category label of the video to be labeled in the category label set is determined according to the word frequency statistical features of the descriptive text; when the hit category label includes the undetermined category label, the undetermined category label is determined as the target category label, and the confidence level threshold is lower than the confidence level threshold.

[0064] Specifically, based on word frequency statistics, the word frequencies of each word in the segmentation set used by each category label in the category label set are first determined. Then, the description text of the video to be labeled is segmented to determine which category labels each word in the description text has a high word frequency in. Category labels with high word frequencies can be used as hit category labels. Then, it is determined whether the hit category labels include one or more of the undetermined category labels. When an undetermined category label is included, it can be confirmed as the target category label. Otherwise, if an undetermined category label is not identified as any of the hit category labels, it is not the target category label. The verification of undetermined category labels based on word frequency statistics can take advantage of the relatively low computational cost of word segmentation-level operations and can overcome the influence of some irrelevant words. After obtaining the word frequency statistics table once, it can be queried and reused infinitely, which is convenient, fast and efficient for identifying undetermined category labels.

[0065] In this embodiment, it should be noted that if the confidence level of the undetermined category label is too low, it is not trustworthy. Therefore, only the confidence level identified by the video classification model that is lower than the confidence level threshold but higher than the confidence level threshold preset in this embodiment can be processed. The confidence level threshold is lower than the confidence level threshold.

[0066] In another embodiment, the following steps can be set: when the confidence level of the undetermined category label reaches the median confidence level, a text classification model is used to determine the predicted category label that the descriptive text is mapped to in the category label set; when the predicted category label includes the undetermined category label, the undetermined category label is determined as the target category label, and the median confidence level is lower than the confidence threshold.

[0067] Specifically, a pre-trained text classification model that has reached convergence can be used. Based on the deep semantic information of the encoding vector of the descriptive text, a classification mapping is performed, mapping it to various category labels in the category label set to determine the corresponding classification probability of each category label. Then, category labels whose classification probabilities reach a preset probability threshold are selected as predicted category labels. If these predicted category labels include a certain undetermined category label, then the undetermined category label can be used as the target category label; otherwise, if an undetermined category label is not identified as any of the predicted category labels, then the undetermined category label is not the target category label. Extracting deep semantic information from the descriptive text for classification mapping separately allows for the utilization of the deep semantics of the descriptive text and can accommodate occasional irrelevant or nonsensical situations in the descriptive text. This is expected to determine predicted category labels with higher reliability, thus providing a more credible decision-making reference for the identification of undetermined category labels predicted by the video classification model.

[0068] In this embodiment, it should be noted that if the confidence level obtained by the undetermined category label in the video classification model is too low, it is not trustworthy. Therefore, it is possible to only process the video classification model that identifies a confidence level lower than the confidence level threshold but higher than the median confidence level preset in this embodiment, where the median confidence level is lower than the confidence level threshold.

[0069] In a further embodiment, the various specific implementation methods for verifying pending category labels given above can be flexibly combined. Any two or more specific implementation methods can be progressively combined. When one method identifies a pending category label as a target category label through verification, there is no need to use other methods to verify the pending category label further. If one method fails to determine that a pending category label is a target category label, the next method is used for auxiliary verification, and so on. Following this approach allows for more comprehensive fallback verification of pending category labels, reducing the omission rate of video annotation and improving the accuracy of video annotation.

[0070] When combining the verification methods based on word frequency statistics and text classification models, the characteristics of each method can be used to process undetermined category labels falling into different confidence intervals, allowing for flexible application. In one improved embodiment, the word frequency statistics method can further provide manual verification after word frequency statistics, making manual identification more accurate and reliable. Therefore, it can process only undetermined category labels whose confidence falls within the first confidence interval defined by the confidence threshold and the confidence limit. Since the text classification model relies on the reasoning ability of a neural network model, its understanding ability is limited. Therefore, the text classification model only processes undetermined category labels whose confidence falls within the second confidence interval defined by the confidence threshold and the median confidence. The median confidence is higher than the confidence threshold, meaning the second confidence interval is a subdivision of the first confidence interval.

[0071] When dealing with situations where there are multiple pending category labels, in applying the above-mentioned specific implementation methods for verifying pending category labels, each pending category label can be identified one by one in order to determine whether each pending category label is the target category label.

[0072] Step S1400: Label the video to be labeled using the target category label.

[0073] After the above process, it is easy to understand that the video to be labeled may obtain one or more target category labels. By using these target category labels as category labels for labeling the video to be labeled and establishing a mapping relationship with the video to be labeled, the labeling of the video to be labeled can be achieved.

[0074] After the videos to be annotated are completed, they can be retrieved according to the target category tags to push the corresponding videos to the relevant users. Alternatively, the target category tags can be considered as category tags of interest to the user who uploaded the video to be annotated, and videos carrying those target category tags can be retrieved and pushed to that user.

[0075] As can be seen from the above embodiments, this application has multiple technical advantages, including but not limited to:

[0076] First, after the video classification model predicts the corresponding category label for the video to be labeled, the category label with higher confidence is directly determined as the target category label for the video to be labeled. For the category label with lower confidence, the descriptive text in the media information of the video to be labeled is used as reference information to verify these category labels to confirm whether they belong to the target category label. This improves the accuracy of video labeling based on the video classification model without increasing the high training cost.

[0077] Secondly, since descriptive text in media information is usually used to introduce the content of the video, using descriptive text to verify the low-confidence category labels predicted by the video classification model can effectively overcome the long-tail effect. For some category labels that the video classification model cannot effectively distinguish due to a lack of sufficient training samples, the ability to recognize these category labels can be improved with the help of descriptive text.

[0078] Finally, the descriptive text of a video often contains information about its content, value, and characteristics. In the process of tagging videos, making good use of its descriptive text for data mining can enhance the utilization value of information resources and avoid wasting information resources.

[0079] Based on any embodiment of this application, please refer to Figure 6 The video classification model is used to determine the confidence level of each category label in the image data mapping to the category label set, and the category labels with confidence levels exceeding a preset confidence threshold are determined as target category labels, including:

[0080] Step S1210: Extract several image frames from the image data of the video to be labeled;

[0081] The image data of the video to be labeled is encapsulated in the video file of the video to be labeled. Therefore, the video file can be decoded to read the corresponding image data. In one embodiment, after the video file is decoded into the image space, several image frames are read from the image space at equal intervals to construct the input information corresponding to the image data required for the video classification model.

[0082] The interval between each image frame can be determined based on the number of frames or time, using a fixed value or a dynamic method. For example, the interval can be determined according to the following formula:

[0083] Span = Length / (N-1)

[0084] Where Span represents the time-based interval, Length represents the playback duration of the video to be labeled, and N represents the total number of image frames to be acquired.

[0085] Step S1220: Extract image feature information of the several image frames using the video classification model, classify and map the image feature information to the classification space corresponding to the category label set, and obtain the confidence level corresponding to each category label;

[0086] The video classification model used in this application has been pre-trained to a convergent state; please refer to [link / reference]. Figures 3 to 5 In any example, there is an input path that depends on image data. For several image frames acquired corresponding to the image data, image formatting preprocessing can be performed according to the input constraints of the video classification model, such as adjusting their size. Then, the images are input into the video classification model, where the image feature information is extracted through the convolutional neural network part. Then, the image feature information or the deep semantic information obtained based on the image feature information is classified and mapped to the classification space corresponding to the category label set through the classifier part, thereby obtaining the confidence of each category label in the category label set.

[0087] Step S1230: Determine the category labels in the classification space whose confidence exceeds the confidence threshold, and use them as the target category labels for the video to be labeled.

[0088] The confidence scores corresponding to each category label obtained in the classification space are used to determine which category labels have higher confidence scores, such as exceeding a preset confidence threshold of 0.95. These category labels are considered trustworthy, and therefore, they can be directly used as the target category labels for the video to be labeled. It is easy to understand that the confidence threshold can be an empirical or experimental threshold, and can be flexibly predetermined.

[0089] According to the above embodiments, the main information relied upon by the video classification model to label the video to be labeled is the multiple image frames of the video to be labeled. The image frames effectively represent the content of the video to be labeled. Based on this, the high-confidence category label obtained by making predictions on the video to be labeled using the semantic understanding ability of the video classification model is a more reliable target category label.

[0090] Based on any embodiment of this application, please refer to Figure 7 Based on the word frequency statistical features of the descriptive text, the hit category label of the video to be labeled in the category label set is determined. When the hit category label includes the undetermined category label, the undetermined category label is determined as the target category label, including:

[0091] Step S2100: Perform word segmentation on the description text to obtain a word segmentation set, wherein the word segmentation set includes multiple words;

[0092] For the described text, it can be segmented in any way to obtain its corresponding word set, which includes multiple words. The specific word segmentation method can be a statistical method such as the N-Gram multi-word segmentation algorithm, or a method based on the bag-of-words (BOW) model, or any other known method.

[0093] In one embodiment, all the word segments obtained from each description text can be constructed as a word segment set in a set manner. That is, the same word segment that appears multiple times in the description text is represented as a single element in the word segment set, so as to avoid repeated subsequent processing for the same word segment that appears multiple times and save computation.

[0094] Step S2200: Query the word frequency of each word segment from the word frequency statistics table corresponding to each category label in the category label set, and construct the word frequency vector corresponding to each category label of each word segment.

[0095] Each category tag in the category tag set described in this application is pre-equipped with a word frequency statistics table to store the word frequencies of various word segments obtained from the descriptive text of the video carrying the corresponding category tag. When it is necessary to determine the word frequency of a certain word corresponding to a certain category tag, it can be determined by querying the word frequency statistics table corresponding to that category tag. Each word frequency statistics table can be represented as a vector, and the word frequency statistics tables corresponding to all category tags can also be combined to represent a two-dimensional table, which can be flexibly implemented.

[0096] For each word in the segmentation set of the descriptive text, the word frequency x corresponding to each category label i can be determined by querying the word frequency statistics table corresponding to each category label in the category label set. i Therefore, for each word in the word segmentation set describing the text, we can obtain its frequency corresponding to all category labels. By summing all the frequency of each word, we can also obtain the total frequency corresponding to each word n. Where total represents the total number of category labels in the category label set, from which the word frequency vector corresponding to each word segment can be further constructed.

[0097] In one embodiment, before constructing the word frequency vector for each word segment, considering that the information reference value of words with low overall word frequency is limited, each word segment is first filtered out, removing those with an overall low word frequency. For example, a word n ​​in the word segmentation set describing the text can be filtered out if it meets any one of the following exemplary conditions:

[0098] sum n <Threshold sum

[0099]

[0100] Among them, Threshold sum The preset minimum threshold for word frequency can be set as needed. When the total word frequency of a word segment is lower than the threshold, the word segment is a low-frequency invalid word and can be filtered out from the word segment set. There is no need to construct a word frequency vector for it in the future.

[0101] The threshold 't' is a preset threshold used to verify the normalization result after normalizing the word frequency distribution of each word in the entire category tag set. It can be set as needed. When the maximum value in the normalization result of the word frequency distribution of each word fails to reach the threshold, the word is a low-frequency invalid word, and the corresponding word can be filtered out from the word set, so there is no need to construct a word frequency vector for it in the future.

[0102] After filtering, the total number of words in the word segmentation set can be greatly reduced. Then, word frequency vectors can be constructed for the few effective words after simplification. The word frequency vectors can be represented by the word frequency of each word n, for example:

[0103] X n =(x1,x2,…,x total ,sum n )

[0104] Of course, in another embodiment, the frequency of each word in each word frequency vector can be further normalized by dividing the frequency of each category label by the total frequency to convert it into a normalized value.

[0105] Step S2300: Determine the category label corresponding to the highest word frequency in the word segmentation set as the hit category label based on the word frequency vector of each word segmentation set;

[0106] Based on the word frequency vectors of effective word segments in the descriptive text, it is possible to further investigate whether they are sufficient to verify the target category label.

[0107] Taking a word frequency vector constructed using numerical values ​​corresponding to word frequencies as an example, it can be determined whether the largest word frequency is greater than a preset value. If it is greater than the preset value, the category label corresponding to the largest word frequency can be confirmed as the category label inferred from the word frequency statistical features of the descriptive text. This category label is then used as the hit category label, and the word segment corresponding to the word frequency vector is regarded as the lower-level category label of the hit category label. This can be used as the lower-level category label for labeling the video to be labeled after subsequent manual confirmation. The same principle applies when the word frequency vector is represented by the normalized numerical values ​​of the word frequencies corresponding to each category label.

[0108] It is easy to understand that based on the word frequency statistical features of the descriptive text, multiple corresponding hit category labels and the corresponding sub-category labels of each hit category label may be determined. The hit category labels are members of the category label set, which are the suspected category labels inferred for the video to be labeled based on the word frequency statistical features corresponding to the word segmentation of the descriptive text. They can be used to verify the undetermined category labels predicted by the video classification model.

[0109] Step S2400: Determine whether the hit category label includes the undetermined category label. If the determination is true, determine the undetermined category label as the target category label.

[0110] The hit category labels determined based on the word frequency statistics of the descriptive text segmentation can be used to verify the undetermined category labels predicted by the video classification model. Specifically, for each undetermined category label, it is determined whether there is a hit category label that matches it. If they match, the undetermined category label is determined to be the target category label. If all hit category labels are different from the undetermined category label, then the undetermined category label is not the target category label. In this way, each undetermined category label can be verified to determine whether it belongs to the target category label.

[0111] According to the above embodiments, the word frequency distribution of each word segment in the descriptive text of the video to be labeled is determined based on the word frequency statistics table of each category label. The category label of the word segment with the higher word frequency is determined as the hit category label based on the word frequency statistics feature. Then, the hit category label is used to verify each undetermined category label one by one. The undetermined category label that passes the verification is actually the hit category label, realizing the utilization of the word frequency statistics feature of the descriptive text. Among them, the word segment corresponding to each hit category label can be used as the lower-level category label of the corresponding hit category label after being accurately confirmed by humans. This realizes the vertical hierarchical subdivision and expansion of the category label set, achieving fine-grained labeling of the video to be labeled.

[0112] Furthermore, inferring the category label of a video to be labeled based on word frequency statistics can effectively filter out invalid words deliberately added by users to attract traffic when writing descriptive text, thus avoiding interference. For video classification models that use the descriptive text to predict the category label of a video to be labeled, although the video classification model may make inaccurate predictions due to invalid words in the descriptive text, further filtering out invalid words can eliminate interference to the prediction results of the video classification model. Data from actual tests in this embodiment shows that, for the unlabeled category labels predicted using this type of video classification model, this embodiment can recall 10% of unlabeled videos with 70% accuracy, achieving significant progress.

[0113] Based on any embodiment of this application, please refer to Figure 8 Before querying the word frequency of each word segment in the word frequency statistics table corresponding to each category label in the category label set and constructing the corresponding word frequency vector, the process includes:

[0114] Step S3100: Based on each category label in the category label set, obtain the description text of the video carrying the category label as a description sample to form a sample set corresponding to the category label;

[0115] In one embodiment, the description text of the tagged video stored in the video service can be used as a description sample to generate a word frequency statistics table for the cost application.

[0116] For each category label in the category label set, all videos carrying that category label can be retrieved as video samples. Then, the descriptive text of all video samples can be obtained as descriptive samples, forming the sample set corresponding to that category label. When determining the corresponding word frequency statistics table for each category label, the corresponding word frequency statistics table can be constructed based on the word frequency statistics of the sample set for that category label.

[0117] In another embodiment, video samples and description samples from other sources but labeled with the same category of tags can also be used to generate the word frequency statistics table.

[0118] Step S3200: Based on the sample set corresponding to each category label, segment each description sample to determine its segmentation sequence;

[0119] To facilitate word frequency statistics, for each description sample in the sample set corresponding to each category label, any feasible word segmentation method can be applied to obtain its corresponding word segmentation sequence. As an optional approach, identical word segments within each segmentation sequence can be retained based on their text position, avoiding merging identical segments into a single word, thus accurately reflecting the frequency of word occurrence. The word segmentation method can employ statistical algorithms such as the N-Gram algorithm, or a bag-of-words model, etc., and can be flexibly selected.

[0120] Step S3300: Based on each word segmentation sequence of the sample set corresponding to each category label, perform word frequency statistics on a word-by-word basis to determine the word frequency corresponding to each word segmentation;

[0121] To facilitate the representation of word frequency statistics, a dictionary can be constructed based on all word segments corresponding to all category labels. This dictionary stores the mapping relationship data from each word segment to the word frequency corresponding to any category label. Accordingly, for the full set of description samples corresponding to a category label, for each word segment, the total number of times it appears in the full word segment sequence corresponding to the current category label is counted as the word frequency. Then, this word frequency is associated with the word segment and stored in the dictionary as the mapping relationship data under the corresponding category label.

[0122] Step S3400: Construct a word frequency statistics table corresponding to each category label by mapping the word segments and their corresponding word frequencies in the sample set corresponding to each category label.

[0123] The above process is performed on each word segment under each category label, resulting in word frequency data for all words under each category label. For each category label, the mapping relationship between its word segments and word frequencies can be considered as a word frequency statistics table corresponding to that category label. This table can be used to determine the word frequency vector of the descriptive text of the video to be labeled, further inferring the matched category label.

[0124] According to the above embodiments, word frequency statistics can be performed on the descriptive text of historically stored video samples in the video service to construct word frequency statistics tables corresponding to each category label in each category label set. These word frequency statistics tables, combined with the descriptive text of the video to be labeled, can be used to infer the hit category label of the video to be labeled, thereby correcting the low-confidence undetermined category label predicted by the video classification model for the video to be labeled. Since the word frequency statistics table is a product of the descriptive samples of historical video samples, its category labels have practical success. Therefore, the word frequency information in the corresponding word frequency statistics table has higher reference value and can characterize the word frequency distribution of each category label. Thus, determining the hit category label of the video to be labeled based on the word segmentation of the descriptive text and then correcting the undetermined category label has high reliability.

[0125] Based on any embodiment of this application, after determining the category label corresponding to the highest word frequency among the word frequency vectors of each word in the word segmentation set as the hit category label, the process includes:

[0126] Step S2500: The word segmentation with the determined hit category label is used as the lower-level candidate label of the determined hit category label and stored as data to be confirmed;

[0127] In the process of determining the hit category label of the video to be labeled by applying word frequency statistics based on the descriptive text of the video to be labeled, as mentioned above, for the word frequency vector of each word segment, the category label corresponding to the maximum word frequency is determined as the hit category label corresponding to that word segment. This hit category label can be used as the superior category label corresponding to that word segment, and the word segment becomes the subordinate category label of this hit category label. Then, the hit category label and its subordinate category label are associated with the feature identifier of the video to be labeled to form the data to be confirmed, which is stored in the database for further manual retrieval and confirmation.

[0128] Step S2600: In response to the user confirmation command, the video to be labeled is labeled with the hit category label and its subordinate candidate labels specified by the command.

[0129] The video service's management or authorized users can retrieve the data to be confirmed from the database, analyze and confirm each piece of data, and once confirmed, trigger a corresponding user confirmation command. In response to this command, the data to be confirmed becomes effective; that is, based on the data to be confirmed, the specified hit category label and corresponding sub-category labels are used to label the specified video to be annotated. It is easy to understand that the hit category label is a member of the category label set. Once manually confirmed and used to annotate the video, it effectively becomes the target category label for manual confirmation. The sub-category label is a corresponding word segment determined from the descriptive text of the video to be annotated based on word frequency statistical features. This word segment semantically subdivides the scope of the hit category label, constituting a sub-vertical classification label of the hit category label.

[0130] Based on the above embodiments, by further utilizing the user confirmation process, the validity of the corresponding hit category tags and sub-category tags can be confirmed manually, which can effectively avoid mislabeling and improve labeling accuracy. Moreover, by using representative and valid word segments as sub-category tags and associating them with their corresponding hit category tags for labeling the videos to be labeled, the labeling work of expanding vertical categories can be saved, and the response speed of the labeling business can be accelerated. In addition, by using word frequency statistics combined with manual review to perform fallback labeling on the videos to be labeled, words that users deliberately fill in that do not match the video content can be filtered out, resulting in strong anti-interference ability.

[0131] Based on any embodiment of this application, please refer to Figure 9 A text classification model is used to determine the predicted category label mapped from the descriptive text to the category label set. When the predicted category label includes the undetermined category label, the undetermined category label is determined as the target category label, including:

[0132] Step S4100: Perform word embedding on the description text to obtain the corresponding encoding vector;

[0133] When it is necessary to apply a text classification model to classify the descriptive text of the video to be labeled, the text classification model should be trained to a convergent state with a sufficient number of corresponding training samples beforehand, so that it can extract deep semantic information from the text and then perform classification mapping.

[0134] To facilitate semantic understanding in the text classification model, word embedding is first performed on the descriptive text of the video to be labeled. Based on the word segmentation of the descriptive text to obtain the corresponding word sequence, the encoding features corresponding to each word are retrieved from the reference vocabulary used by the text classification model. According to the order of each word segmentation, the encoding features of each word segment are constructed into an encoding vector, which is used as the input of the text classification model.

[0135] In one embodiment, based on the encoded vector, positional encodings corresponding to each word can be superimposed according to the position of each word in the word sequence of the descriptive text, so as to enrich the basic semantic information required by the text classification model, making the text classification model easier to train and more accurate in obtaining prediction results.

[0136] Step S4200: Extract text feature information from the encoded vector using the text classification model, classify and map the text feature information to the classification space corresponding to the category label set, and obtain the classification probability corresponding to each category label;

[0137] An exemplary text classification model can be implemented by using a text feature extractor based on a recurrent neural network followed by a classifier, such as RNN, LSTM, BERT (Bidirectional Encoder Representations from Transformer), or a Transformer encoder. Such models are suitable for reasoning by combining contextual information, thereby more accurately understanding semantics and obtaining more accurate prediction results.

[0138] After the encoded vector is input into the text classification model, the text feature extractor extracts deep semantic information from the encoded vector to obtain the corresponding text feature information. Then, the text feature information is fully connected through the subsequent classifier and mapped to the classification space set according to the category label set. The classification probability corresponding to each category label in the classification space is calculated.

[0139] Step S4300: Determine the category labels in the classification space whose classification probability exceeds a preset probability threshold as predicted category labels. When the predicted category labels include the undetermined category labels, determine the undetermined category labels as the target category labels of the video to be labeled.

[0140] The classification probability of each category label in the classification space essentially represents the confidence level of the encoded vector mapped to the corresponding category label. However, the text classification model relies solely on the predictions made by the descriptive text. Considering that the descriptive text may be irrelevant, the prediction results of the text classification model cannot be completely trusted. In this case, it is necessary to filter the classification probabilities of each category label predicted by the text classification model and select the results with higher classification probabilities as the trustworthy results.

[0141] To implement the aforementioned filtering, a probability threshold can be preset. Then, category labels with classification probabilities higher than the probability threshold are selected from the classification space as predicted category labels. These predicted category labels are the category labels that the text classification model predicts the video to be labeled might be mapped to, and can be used to correct the undetermined category labels predicted by the video classification model. The probability threshold can be set as needed.

[0142] Specifically, for each undetermined category label, it is determined whether there is a predicted category label that matches it. If they match, the undetermined category label is determined to be the target category label. If all predicted category labels are different from the undetermined category label, then the undetermined category label is not the target category label. In this way, each undetermined category label can be verified to determine whether it belongs to the target category label.

[0143] In one embodiment, when using a text classification model and a word frequency statistics-based method to jointly verify the undetermined category labels, the undetermined category labels can be verified first using the word frequency statistics-based method. For undetermined category labels that have passed the verification, when applying the text classification model for verification, there is no need to verify them again; only the undetermined category labels that have failed the verification need to be verified accordingly.

[0144] According to the above embodiments, the text classification model has the ability to perform deep semantic understanding of descriptive text and infer the corresponding category labels. The category labels with high confidence are selected by using probability thresholds as the predicted category labels determined by the text classification model. The predicted category labels are used to perform fallback verification on each undetermined category label generated by the video classification model, so as to achieve fallback labeling of the video to be labeled, making the labeling of the video to be labeled more complete and systematic.

[0145] Similarly, in the above embodiments, with the help of the overall semantic targeting of the text classification model, the influence of invalid words that are not consistent with the video content of the video to be labeled, which are intentionally added by the user when filling in the description text, can be weakened, effectively expanding the labeling coverage. In practice, after processing the video to be labeled by the video classification model and the word frequency statistics method respectively, the text classification model can recall 20% of the unlabeled videos with 70% accuracy for effective labeling.

[0146] Based on any embodiment of this application, as an exemplary application, please refer to [link / reference]. Figure 10 After labeling the video to be labeled with the target category tag, the process includes:

[0147] Step S1500: Obtain media information of the target video carrying the target category tag, and construct a video recommendation list based on the media information;

[0148] After the video to be annotated has been annotated using the processes described in the above embodiments, the target category tags annotated on the video to be annotated actually represent, to some extent, the category tags that the user of the video to be annotated is interested in. Therefore, videos can be recommended to the uploading user based on the target category tags annotated on the video to be annotated.

[0149] To recommend videos to the uploading user, the target category tag can be used as a query tag. In the video library corresponding to the video service, the target videos carrying any of the target category tags are queried and recalled. Then, their media information is obtained, including image data and descriptive text in the video file. Then, according to a preset format, such as constructing the cover of the target video from the image data, a video recommendation list is constructed based on the media information of each target video, and each target video is listed in the video recommendation list.

[0150] Step S1600: Push the video recommendation list to the terminal user who uploaded the video to be labeled.

[0151] Once the video recommendation list is determined, it can be pushed to the end user who uploaded the video to be labeled, i.e., the user who uploaded the video. After receiving the video recommendation list, the user parses it and displays each target video in the graphical user interface. The user can click on any target video to access it.

[0152] As can be understood from the above embodiments, after a user uploads a video to be labeled and the corresponding target category tag is determined, the corresponding target category tag can serve as a category tag of interest to the user to recommend videos to the user who uploaded the video to be labeled. This allows the user to expand the boundaries of their interests and obtain recommended videos that are similar to or related to the content of the video to be labeled. In particular, for users who upload a video to be labeled for the first time, the target category video determined first can be used for cold start of recommended videos for that user.

[0153] Please see Figure 11 According to one aspect of this application, a video annotation device includes an information acquisition module 1100, a default classification module 1200, an auxiliary verification module 1300, and a video annotation module 1400. The information acquisition module 1100 is configured to acquire media information of a video to be annotated, the media information including image data and descriptive text of the video to be annotated. The default classification module 1200 is configured to use a video classification model to determine the confidence level of each category label mapped from the image data to a category label set, and to determine category labels with confidence levels exceeding a preset confidence threshold as target category labels. The auxiliary verification module 1300 is configured to use category labels with confidence levels not reaching the confidence threshold as pending category labels, and to verify whether the pending category labels are target category labels based on the descriptive text. The video annotation module 1400 is configured to annotate the video to be annotated using the target category labels.

[0154] Based on any embodiment of this application, the default classification module 1200 includes: an image extraction unit, configured to extract several image frames from the image data of the video to be labeled; a video classification unit, configured to extract image feature information of the several image frames using the video classification model, classify and map the image feature information to the classification space corresponding to the category label set, and obtain the confidence level corresponding to each category label; and a target initial screening unit, configured to determine the category labels in the classification space whose confidence level exceeds the confidence threshold, and use them as the target category labels of the video to be labeled.

[0155] Based on any embodiment of this application, the auxiliary verification module 1300 includes: a word frequency verification module, configured to determine the hit category label of the video to be labeled in the category label set according to the word frequency statistical features of the descriptive text when the confidence of the undetermined category label reaches the confidence threshold, and to determine the undetermined category label as the target category label when the hit category label includes the undetermined category label, wherein the confidence threshold is lower than the confidence threshold.

[0156] Based on any embodiment of this application, the auxiliary verification module 1300 includes: a semantic verification module, configured to determine the predicted category label mapped from the descriptive text to the category label set when the confidence level of the undetermined category label reaches the median confidence level, and to determine the undetermined category label as the target category label when the predicted category label includes the undetermined category label, wherein the median confidence level is lower than the confidence level threshold.

[0157] In the two embodiments disclosed above, the word frequency verification module and the semantic verification module can be flexibly combined and applied. In an improved embodiment, the median confidence level used by the semantic verification module can be greater than the confidence threshold used by the word frequency verification module, but both are lower than the confidence threshold corresponding to the video classification model.

[0158] Based on any embodiment of this application, the word frequency verification module includes: a word segmentation processing unit, configured to perform word segmentation processing on the description text to obtain a word segmentation set, the word segmentation set including multiple words; a word frequency query unit, configured to query the word frequency of each word in the word frequency statistics table corresponding to each category tag in the category tag set, and construct the word frequency vector corresponding to each category tag of each word as the word frequency vector of that word; a superior determination unit, configured to determine the category tag corresponding to the largest word frequency among the word frequency vectors of each word in the word segmentation set as the hit category tag; and a judgment verification unit, configured to judge whether the hit category tag includes the undetermined category tag, and when the judgment is true, determine the undetermined category tag as the target category tag.

[0159] Based on any embodiment of this application, prior to the word frequency query unit in the word frequency verification module, the following is included: a sample acquisition unit, configured to acquire description text of videos carrying the category tag as description samples based on each category tag in the category tag set, forming a sample set corresponding to the category tag; a sample word segmentation unit, configured to segment each description sample in the sample set corresponding to each category tag to determine its word segmentation sequence; a word frequency statistics unit, configured to perform word frequency statistics on each word segmentation sequence in the sample set corresponding to each category tag, determining the word frequency corresponding to each word segmentation; and a statistical table building unit, configured to construct a word frequency statistics table corresponding to the category tag by constructing the mapping relationship between each word segmentation and its corresponding word frequency in the sample set corresponding to each category tag.

[0160] Based on any embodiment of this application, the upper-level determination unit in the word frequency verification module includes: a data storage unit, configured to store the word segmentation with the determined hit category tag as the lower-level candidate tag of the determined hit category tag as data to be confirmed; and a manual annotation unit, configured to respond to the user confirmation command and annotate the video to be annotated with the hit category tag and its lower-level candidate tag specified by the command.

[0161] Based on any embodiment of this application, the semantic verification module includes: a text encoding unit, configured to perform word embedding on the descriptive text to obtain a corresponding encoding vector; a text classification unit, configured to extract text feature information from the encoding vector using the text classification model, classify and map the text feature information to the classification space corresponding to the category label set, and obtain the classification probability corresponding to each category label; and a probability verification unit, configured to determine the category label whose classification probability in the classification space exceeds a preset probability threshold as the predicted category label, and when the predicted category label includes the undetermined category label, determine the undetermined category label as the target category label of the video to be labeled.

[0162] Based on any embodiment of this application, the video annotation module 1400 further includes: a recommendation construction module, configured to obtain media information of target videos carrying the target category tag, and construct a video recommendation list based on the media information; and a recommendation execution module, configured to push the video recommendation list to the terminal user who uploaded the video to be annotated.

[0163] Another embodiment of this application also provides a video annotation device. For example... Figure 12 The diagram shows the internal structure of a video annotation device. This device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The computer-readable, non-volatile storage medium stores an operating system, a database, and computer-readable instructions. The database stores information sequences, and when executed by the processor, these computer-readable instructions enable the processor to implement a video annotation method.

[0164] The processor of the video annotation device provides computing and control capabilities to support the operation of the entire device. The device's memory can store computer-readable instructions, which, when executed by the processor, cause the processor to perform the video annotation method of this application. The network interface of the video annotation device is used for communication with a terminal.

[0165] Those skilled in the art will understand that Figure 12The structure shown is merely a block diagram of a portion of the structure related to the solution of this application and does not constitute a limitation on the video annotation device to which the solution of this application is applied. A specific video annotation device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0166] In this embodiment, the processor is used to execute... Figure 11 The specific functions of each module are described, and the memory stores the program code and various data required to execute the above modules or sub-modules. The network interface is used to realize data transmission between user terminals or servers. The non-volatile readable storage medium in this embodiment stores the program code and data required to execute all modules in the video annotation device of this application, and the server can call the server's program code and data to execute the functions of all modules.

[0167] This application also provides a non-volatile readable storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the video annotation method of any embodiment of this application.

[0168] This application also provides a computer program product, including a computer program / instructions that, when executed by one or more processors, implement the steps of the method described in any embodiment of this application.

[0169] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a computer-readable storage medium such as a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM).

[0170] In summary, this application enables more comprehensive and systematic annotation of videos, avoiding omissions and errors, improving annotation accuracy, and comprehensively enhancing the service quality of video services based on accurate annotation of video category tags.

Claims

1. A video annotation method, characterized in that, include: Obtain the media information of the video to be labeled, including the image data and descriptive text of the video to be labeled; A video classification model is used to determine the confidence level of each category label in the image data mapping to the category label set, and the category labels with confidence levels exceeding a preset confidence threshold are determined as target category labels; When the video classification model is constructed as a multimodal video classification model, the step of using the video classification model to determine the confidence level of each category label mapped from the image data to the category label set includes: inputting the image data and the descriptive text into the video classification model, extracting the corresponding image feature information and text feature information; calling the concatenation layer of the video classification model to concatenate the image feature information and text feature information to obtain deep semantic information; and calling the classifier of the video classification model to classify and map the deep semantic information to determine the confidence level of each category label. The method involves using category labels whose confidence level does not reach the confidence threshold as undetermined category labels, and verifying whether the undetermined category label is a target category label based on the descriptive text. This includes: when the confidence level of the undetermined category label reaches the confidence threshold, determining the hit category label of the video to be labeled in the category label set based on the word frequency statistical features of the descriptive text; when the hit category label includes the undetermined category label, determining the undetermined category label as a target category label, wherein the confidence threshold is lower than the confidence threshold; and / or, when the confidence level of the undetermined category label reaches the median confidence level, using a text classification model to determine the predicted category label mapped from the descriptive text to the category label set; when the predicted category label includes the undetermined category label, determining the undetermined category label as a target category label, wherein the median confidence level is lower than the confidence threshold. The target category label is used to label the video to be labeled.

2. The video annotation method according to claim 1, characterized in that, A video classification model is used to determine the confidence level of each category label in the image data mapping to the category label set. Category labels with confidence levels exceeding a preset confidence threshold are identified as target category labels, including: Extract several image frames from the image data of the video to be labeled; The video classification model is used to extract image feature information from the several image frames, and the image feature information is classified and mapped to the classification space corresponding to the category label set to obtain the confidence level corresponding to each category label. The category labels in the classification space whose confidence exceeds the confidence threshold are determined as the target category labels of the video to be labeled.

3. The video annotation method according to claim 1, characterized in that, Based on the word frequency statistical features of the descriptive text, the hit category label of the video to be labeled in the category label set is determined. When the hit category label includes the undetermined category label, the undetermined category label is determined as the target category label, including: The descriptive text is segmented to obtain a segmentation set, which includes multiple segments; The word frequency of each word segment is retrieved from the word frequency statistics table corresponding to each category label in the category label set, and the word frequency of each word segment corresponding to each category label is used to construct the word frequency vector of that word segment. The category label corresponding to the highest word frequency is determined as the hit category label based on the word frequency vector of each word in the word segmentation set; Determine whether the hit category label includes the undetermined category label. If the determination is true, determine that the undetermined category label is the target category label.

4. The video annotation method according to claim 3, characterized in that, Before querying the word frequency of each word segment in the word frequency statistics table corresponding to each category label in the category label set and constructing the corresponding word frequency vector, the process includes: Based on each category label in the category label set, the descriptive text of the video carrying that category label is obtained as a descriptive sample, thus forming a sample set corresponding to that category label; Based on the sample set corresponding to each category label, each descriptive sample is segmented to determine its segmentation sequence; Based on the word segmentation sequences of the sample set corresponding to each category label, word frequency statistics are performed on a word-by-word basis to determine the word frequency corresponding to each word segment. The mapping relationship between each word segment and its corresponding word frequency in the sample set corresponding to each category label is constructed into a word frequency statistics table corresponding to that category label.

5. The video annotation method according to claim 3, characterized in that, After determining the category label corresponding to the highest word frequency in the word segmentation set as the hit category label based on the word frequency vector of each word, the following steps are included: The word segment that has been identified as a hit category tag is used as the next-level candidate tag of the hit category tag and stored as data to be confirmed. In response to a user confirmation command, the video to be labeled is annotated with the hit category label specified in the command and its subordinate candidate labels.

6. The video annotation method according to claim 1, characterized in that, A text classification model is used to determine the predicted category label mapped from the descriptive text to the category label set. When the predicted category label includes the undetermined category label, the undetermined category label is determined as the target category label, including: Word embedding is performed on the descriptive text to obtain the corresponding encoding vector; The text classification model is used to extract text feature information from the encoded vector, and the text feature information is classified and mapped to the classification space corresponding to the category label set to obtain the classification probability corresponding to each category label; The category labels in the classification space whose classification probability exceeds a preset probability threshold are determined as predicted category labels. When the predicted category labels include the undetermined category labels, the undetermined category labels are determined as the target category labels of the video to be labeled.

7. A video annotation device, characterized in that, include: The information acquisition module is configured to acquire media information of the video to be labeled, the media information including image data and descriptive text of the video to be labeled; The default classification module is set to use a video classification model to determine the confidence level of each category label in the category label set that the image data is mapped to, and to determine the category label that has a confidence level exceeding a preset confidence level threshold as the target category label; When the video classification model is constructed as a multimodal video classification model, the step of using the video classification model to determine the confidence level of each category label mapped from the image data to the category label set includes: inputting the image data and the descriptive text into the video classification model, extracting the corresponding image feature information and text feature information; calling the concatenation layer of the video classification model to concatenate the image feature information and text feature information to obtain deep semantic information; and calling the classifier of the video classification model to classify and map the deep semantic information to determine the confidence level of each category label. The auxiliary verification module is configured to use category labels whose confidence level does not reach the confidence threshold as pending category labels, and to verify whether the pending category labels are target category labels based on the description text. This includes: when the confidence level of the pending category label reaches the confidence threshold, determining the hit category label of the video to be labeled in the category label set based on the word frequency statistical features of the description text; when the hit category label includes the pending category label, determining the pending category label as the target category label, where the confidence threshold is lower than the confidence threshold; and / or, when the confidence level of the pending category label reaches the median confidence level, using a text classification model to determine the predicted category label mapped from the description text to the category label set; when the predicted category label includes the pending category label, determining the pending category label as the target category label, where the median confidence level is lower than the confidence threshold. The video annotation module is configured to annotate the video to be annotated using the target category label.

8. A video annotation device, comprising a central processing unit and a memory, characterized in that, The central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the method as described in any one of claims 1 to 6.

9. A non-volatile readable storage medium, characterized in that, It stores, in the form of computer-readable instructions, a computer program implemented according to any one of claims 1 to 6, which, when invoked by a computer, executes the steps included in the corresponding method.

10. A computer program product, characterized in that, Includes a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 6.

Images