Video labeling method, system, equipment and device

The video annotation method uses a multimodal large language model to automatically generate and match descriptions for efficient and cost-effective video labeling, addressing the inefficiencies of traditional manual annotation methods.

CN119091212APending Publication Date: 2024-12-06BEIJING IQIYI TECH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411192027.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-28
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Traditional video annotation methods are time-consuming and costly due to the need for manual annotation and retraining with changing social labels, leading to inefficiencies.

Method used

A video annotation method utilizing a multimodal large language model to automatically generate natural language descriptions, match them with preset tags and key descriptions, and apply corresponding labels, eliminating the need for manual training and reducing costs.

Benefits of technology

Enables efficient and cost-effective video annotation by leveraging the cognitive abilities of large language models to intelligently label videos, reducing labor costs and improving annotation speed and accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091212A_ABST
    Figure CN119091212A_ABST
Patent Text Reader

Abstract

The invention relates to a video annotation method, system and device and a medium, and the method comprises the steps: inputting a first video into a first model, and obtaining a first description of the first video, the first model comprising a multi-modal large language model; determining a preset label and a key description corresponding to the label, and determining a key description matched with the first description based on a preset matching strategy; and determining an annotation label corresponding to the key description matched with the first description, and annotating the first video through the annotation label. Through the video labeling method provided by the invention, the to-be-labeled video can be accurately and quickly labeled, and manual labeling and corresponding classification model training are not needed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a video annotation method, system, device and medium. Background Art

[0002] With the development of the Internet, video has gradually become a popular form of online activity. Video playback service providers need to classify and manage the videos they serve. The usual method is video annotation, which is to label the actions and expressions of the characters in the video. This is an important part of video understanding. The traditional video annotation method is to train the corresponding video classification model by collecting a large number of label-related samples. The collection process requires a lot of manual annotation, which is time-consuming and has high labor costs. And with the development of social culture, labels will also change. At that time, the corresponding video classification model may need to be retrained. Therefore, the traditional video annotation method has the problem of high cost and low efficiency. Summary of the invention

[0003] The present application provides a video annotation solution to solve the problems of high cost and low efficiency in traditional video annotation implementation methods.

[0004] In a first aspect, the present application provides a video annotation method, comprising:

[0005] Inputting a first video into a first model to obtain a first description of the first video, wherein the first model includes a multimodal large language model;

[0006] Determine a preset tag and a key description corresponding to the tag, and determine a key description matching the first description based on a preset matching strategy;

[0007] Determine a label corresponding to a key description matching the first description, and label the first video using the label.

[0008] In some embodiments of the present application, determining a preset tag and a key description corresponding to the tag, and determining the key description matching the first description based on a preset matching strategy includes:

[0009] Determine the regular expression corresponding to the tag based on the key description of the tag;

[0010] searching the first description for a description statement satisfying the regular expression based on the regular expression, and if the description statement exists, considering that the key description matches the first description;

[0011] If the description statement does not exist, it is considered that the key description does not match the first description.

[0012] In some embodiments of the present application, determining a regular expression corresponding to a tag based on a key description of the tag includes:

[0013] Determining the verbs contained in the key description according to the content of the key description;

[0014] Acquire different expressions of the verb in corresponding tenses based on the verb;

[0015] A regular expression capable of matching different expressions is constructed according to the same parts of the verbs in different expressions.

[0016] In some embodiments of the present application, determining a regular expression corresponding to a tag based on a key description of the tag includes:

[0017] Determine the verbs and nouns contained in the key description and their corresponding order according to the content of the key description;

[0018] According to the semantics of the verb and the noun, a regular expression is constructed which starts with the verb, ends with the noun and can match multiple arbitrary characters in between.

[0019] In some embodiments of the present application, the key description is generated in the following manner:

[0020] Inputting the second video annotated with the label into the first model, and obtaining a second description about the label through the model input;

[0021] The tag and the second description are input into a second model, key phrases related to the tag in the second description are extracted through the second model, and the key phrases are used as the key description.

[0022] In some embodiments of the present application, the key phrase is used as the key description, including:

[0023] inputting the key phrase and the tag into a second model;

[0024] Through a predetermined prompt word project, it is determined whether the key phrase is related to the tag, and the related key phrase is used as the key description.

[0025] In some embodiments of the present application, determining a label corresponding to a key description matching the first description, and labeling the first video using the label, includes:

[0026] If there are multiple key descriptions matching the first description;

[0027] Then, the priority of the corresponding tag is obtained according to the key description;

[0028] The high-priority tag is used as the tag of the first video, and the first video is labeled by the tag.

[0029] The second aspect of the present application further proposes a video annotation system, comprising:

[0030] a video description generating module, configured to input a first video into a first model to obtain a first description of the first video, wherein the first model includes a multimodal large language model;

[0031] A video classification module, used to determine a preset tag and a key description corresponding to the tag, and determine a key description matching the first description based on a preset matching strategy;

[0032] The video annotation module is used to determine an annotation tag corresponding to a key description matching the first description, and annotate the first video using the annotation tag.

[0033] The third aspect of the present application also proposes a computer device: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; at least one memory connected to the at least one bus, wherein the processor is configured to execute the video annotation method described in any one of the above embodiments.

[0034] The fourth aspect of the present application further proposes a computer storage medium storing computer executable instructions, wherein the computer executable instructions are used to execute the video annotation method of any one of the above-mentioned embodiments.

[0035] The above technical solution provided by the embodiment of the present application has the following advantages over the prior art: The method provided by the embodiment of the present application obtains a natural language description of the video to be annotated by inputting the video to be annotated into a large multimodal model to utilize the multimodal large model's ability to understand and interpret the video, and then matches it with the key description corresponding to the label, and uses the label corresponding to the successfully matched key description as the annotation label of the video to be annotated, and annotates the video to be annotated. Compared with the traditional method of training annotation models, the cognitive ability of the large multimodal model is directly utilized to realize intelligent annotation of the video to be annotated. Manually annotated data is no longer used, and no more cost is spent on training models. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0038] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0039] Figure 1 A schematic diagram of a process flow of a video annotation method provided in an embodiment of the present application;

[0040] Figure 2 A schematic diagram of a process for matching key descriptions and video descriptions provided in an embodiment of the present application;

[0041] Figure 3 A schematic diagram of a process for determining a regular expression based on a key description provided in an embodiment of the present application;

[0042] Figure 4 A schematic diagram of a process for determining a regular expression based on a key description provided in an embodiment of the present application;

[0043] Figure 5 A schematic diagram of a key description generation process provided in an embodiment of the present application;

[0044] Figure 6 A schematic diagram of a key description generation process provided in an embodiment of the present application;

[0045] Figure 7 A schematic diagram of a process of labeling a video to be labeled based on a label tag provided in an embodiment of the present application;

[0046] Figure 8 A schematic diagram of the structure of a video annotation system provided in an embodiment of the present application;

[0047] Fig. 9 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0049] The disclosure below provides many different embodiments or examples to implement different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention. In addition, the present invention can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0050] In order to solve the problem of high cost and low efficiency in traditional video annotation, the present application provides a video annotation solution that can effectively solve the above problem.

[0051] Figure 1 A schematic diagram of a process flow of a video annotation method provided in an embodiment of the present application includes:

[0052] Step S1: input a first video into a first model to obtain a first description of the first video, wherein the first model includes a multimodal large language model;

[0053] Step S2: determining a preset tag and a key description corresponding to the tag, and determining a key description matching the first description based on a preset matching strategy;

[0054] Step S3: determine a label corresponding to the key description matching the first description, and label the first video using the label.

[0055] In an embodiment of the present application, the first video represents a video to be annotated, the first model is a multimodal large model, and the first description refers to the video description output by the multimodal large model for the video to be annotated, that is, the understandable description text output by the multimodal large model based on the content of the video to be annotated. The multimodal large model refers to a large pre-trained machine learning model that can process and integrate a variety of different types of data. In some embodiments of the present application, the ability of the large model to understand video data is mainly used, that is, the ability of the large model to output a text description of the video to be annotated.

[0056] It should be noted that, in some embodiments, the video to be labeled also includes multiple continuous video frames corresponding to a certain picture in the video, that is, in some cases, the multimodal large model outputs the corresponding video description for the corresponding picture in the input video, for example, outputs the corresponding video description for each scene. When the video is subsequently labeled, it is also labeled for different scenes in the video. In this case, the video input to the large model will have multiple labels, and each label corresponds to one or more video frames.

[0057] A preset tag refers to a pre-established classification tag for annotating the video to be annotated. Based on the preset meaning, the video to be annotated with a similar meaning is annotated. A key description refers to an explanatory descriptive sentence or word corresponding to the preset tag, and is similar to the preset meaning of the tag. A classification tag can include multiple key descriptions. Annotation tags are determined tags used to annotate the video to be annotated.

[0058] The matching strategy refers to a matching method for matching the key description with the video description of the video to be annotated, that is, a matching method for determining whether the key description is related to the video description.

[0059] In step S1, the video to be annotated is first input into the multimodal large model, and the corresponding prompt words are used to allow the multimodal large model to output its understanding of the video to be annotated, and output the corresponding video description in the form of text description. In some embodiments of the present application, the multimodal large model used may include mainstream large models such as videochat, videollama, and stllm. Typical prompt words include: "Describe the following video concisely", "Provide a brief description of the given video clip", "Offer a succinctexplanation of the footage presented", etc. to prompt the multimodal large model to understand the content of the picture displayed by the video to be annotated and output the corresponding description.

[0060] In step S2, first determine which tags are used to annotate the video to be annotated, and further determine the key description corresponding to each tag. After obtaining the video description output by the multimodal large model for the video to be annotated, determine whether the video description of the video to be annotated matches the corresponding key description according to the preset matching strategy. In some embodiments of the present application, the large language model can be used to determine which one or more key descriptions match the video description, that is, the video description and the key description are input into the large language model, and then the large language model finds the key description that is close to or the same as the video description.

[0061] In step S3, after determining the key description related to the video description to be annotated, determine the label corresponding to the key description, and use the label as the classification label of the video to be annotated to annotate the video to be annotated.

[0062] like Figure 2 As shown, in some embodiments of the present application, determining a preset tag and a key description corresponding to the tag, and determining a key description matching the first description based on a preset matching strategy includes:

[0063] Step S21, determining a regular expression corresponding to the tag based on the key description of the tag;

[0064] Step S22: searching the first description for a description statement satisfying the regular expression based on the regular expression; if a description statement exists, it is considered that the key description matches the first description;

[0065] Step S23: If the description statement does not exist, it is considered that the key description does not match the first description.

[0066] In this embodiment, the key description and the video description of the video to be marked can be matched quickly by regular matching. The corresponding regular expression is generated by the key description, and each key description corresponds to at least one regular expression. When matching the key description with the video description of the video to be marked, all tags are first determined, and the regular expression of the key description corresponding to the tag is further determined. Then, based on the corresponding regular expression, a description statement that satisfies the regular expression is searched from the video description. If the corresponding regular expression does not find a matching result in the video description, it means that the text in the video description does not match the corresponding regular expression, and further indicates that the corresponding key description does not match or is not related to the video description of the video to be marked. Then, the next regular expression can be replaced and searched from the video description of the video to be marked, and the above process is repeated until all regular expressions are searched from the video description of the video to be marked. If it is determined by regular matching that there are one or more regular expressions that can find matching statements from the video description of the video to be marked, it means that the key description corresponding to one or more regular expressions matches the video description of the video to be marked, and further indicates that the tag corresponding to the key description matches the video description of the video to be marked, and the tag corresponding to the key description can be used to mark the video to be marked. If it is found through regular matching that no regular expression can find a matching statement from the video description of the video to be annotated, it means that no tag matches the video to be annotated.

[0067] like Figure 3As shown, in some embodiments of the present application, determining a regular expression corresponding to a tag based on a key description of the tag includes:

[0068] Step S31, determining the verbs contained in the key description according to the content of the key description;

[0069] Step S32: acquiring different expressions of the verb in corresponding tenses based on the verb;

[0070] Step S33: construct a regular expression that can match different expressions according to the same parts of the verbs in different expressions.

[0071] In this embodiment, when the video description of the video to be annotated is in English output by the multimodal large model. For example, when the video description output by the multimodal large model is more accurate or rich in English, the video description output in English is usually the first choice. In this regard, when determining the regular expression corresponding to the label based on the key description, the tense change of the corresponding verb in the key description needs to be considered.

[0072] Specifically, after determining the key description, obtain the verbs in the key description, and then obtain the different expressions of the corresponding verbs in all tenses. Then, according to the different expressions of different tenses, a regular expression that can match different expressions is constructed. For example, taking the label "hug" as an example, a possible key description may be "Give a hug". The expression forms of the verb "Give" include: Gives (third person singular form), Gave (past tense), Given (past participle form), and Giving (present participle form). The regular expression corresponding to the key description is: "\bGiv[e]?\w*a hug". The "\bGiv[e]?\w*" in the formula is used to match other tense expressions of the word Give.

[0073] Furthermore, in order to be compatible with the past tense, the corresponding regular expression is: "\bGave a hug" is used to match the video description corresponding to the past form of Give alone.

[0074] like Figure 4 As shown, in some embodiments of the present application, determining a regular expression corresponding to a tag based on a key description of the tag includes:

[0075] Step S41, determining the verbs and nouns contained in the key description and their corresponding order according to the content of the key description;

[0076] Step S42: According to the semantics of verbs and nouns, a regular expression is constructed that starts with a verb, ends with a noun, and can match multiple arbitrary characters in between.

[0077] In this embodiment, in the video description output by the multimodal large model, for the same type of transaction or scene, due to the difference of some elements in the video, or the uncertainty of the output content of the multimodal large model, there may be other words in the output video description. For example, the video description related to the label "hug" is also a video description. The multimodal large model outputs the video description of some video content as "Give that girl a big hug". From the semantic point of view, it can be known that the video description also belongs to the label "hug", but its description includes prepositions and nouns. Therefore, for these situations, when determining the regular expression corresponding to the key description, it is necessary to first determine the verbs and nouns in the key description, as well as the order of the two, and then construct a regular expression that starts with the verb and ends with the noun. For example, taking the key description "Give a hug" as an example, combined with the above consideration of the influence of the verb tense, the corresponding regular expression can be: "\bGiv[e]?\w*(\s+\b[\w']+\b){0,3}hug";

[0078] Here, “(\s+\b[\w']+\b)” indicates any word that appears between “Give” and “hug”; “{0,3}” indicates that the maximum number of any words is 3; but this value can be adjusted according to the output characteristics of the specific multimodal large model.

[0079] like Figure 5 As shown, in some embodiments of the present application, the key description is generated in the following manner:

[0080] Step S51: input the second video annotated with the label into the first model, and obtain a second description about the label through the model input;

[0081] Step S52: input the tag and the second description into the second model, extract key phrases related to the tag in the second description through the second model, and use the key phrases as key descriptions.

[0082] In this embodiment, the second video represents a video that has been labeled, the second model is a large language model, and the second description represents a video description output by the multimodal large model for the video that has been labeled.

[0083] In this embodiment, the key description is a text used to match the video description output by the multimodal large model. The key description can be generated by manually understanding and interpreting the tags to generate the corresponding key description. The multimodal large model can also be used to generate video descriptions for the output of the labeled videos.

[0084] Specifically, the labeled video is first input into the multimodal large model, and the multimodal large model outputs the corresponding video description, thereby obtaining the video description corresponding to the label, and realizing the binding of the corresponding label with the understanding ability of the multimodal large model (the video description is the text description output by the multimodal large model for the labeled video content). Then the label and the corresponding video description are input into the large language model, and supplemented by the corresponding prompt word project, the large language model analyzes which words or descriptive terms in the video description are related to the label. In this way, a simplified description that can represent the label is determined as the key description.

[0085] Specifically, taking the label "hug" as an example, when one or more videos labeled "hug" are input into the multimodal large model, the multimodal large model will output content that may include relevant descriptions of different dimensions such as scenes, character behaviors, and human body movements based on different video contents. Therefore, the output video description may be a very long description sentence. Although these description sentences can explain the behavior or status of each element in the video in detail, they are limited to the current corresponding video content in terms of text, and often have limitations when explaining other related video content. Therefore, in order to solve this problem, it is necessary to remove the video description that is only related to the description of individual videos in the video description to find the key description.

[0086] Specifically, the video description output by the multimodal large model can be input into the large language model, and then the prompt words of the large language model can be used, such as: "Read and understand the above text in detail, please make sure to extract the phrases related to the label [hug] as accurately as possible, and the output is strictly required to be the original text" to ask the large language model questions, so that the large language model can understand the meaning of the video description based on its content and determine which words can be expressed as related to "hug". The content output by the large model may be: "Give a hug", "Embrace each other", "Hugging tightly", "Warm embrace", "Bearhug", "Loving embrace", "Gentle hug", "Tight hug", etc.

[0087] Furthermore, after the large language model outputs the corresponding simplified key phrase, the key phrase can be used as the key description.

[0088] like Figure 6 As shown, in some embodiments of the present application, the key phrase is used as the key description, including:

[0089] Step S61, inputting key phrases and tags into the second model;

[0090] Step S62: determine whether the key phrase is related to the tag through a predetermined prompt word project, and use the related key phrase as a key description.

[0091] In this embodiment, after the key description is obtained through the large language model, since all the text of the video description is input into the large language model and screened by the large language model in determining the key description, when the large language model makes a judgment or decision, those ignored words (text content that is not selected in the video description corresponding to the key phrase) may prompt the large language model to make the corresponding key description related to the corresponding key tag, so there may be a situation where a small number of key descriptions are irrelevant to the tag.

[0092] Therefore, the corresponding key phrases and tags need to be input into the large language model again, and then the corresponding prompt words are enriched. For example, "Judge whether {key phrase} is related to {tag}, and reply yes or no." When "{key phrase}" and "{tag}" are input into the large language model, the specific key phrases and tag names will be embedded into the prompt words. For example, "Judge whether Give a hug is related to hugging, and reply yes or no." The key phrases are further screened in the above way.

[0093] After the large language model outputs the corresponding results, the video descriptions related to "hug" are used as key descriptions, and these key descriptions are mapped to the label "hug" and saved. So that when annotating the video to be annotated, after obtaining the video description of the video to be annotated output by the multimodal large model, the classification of the video to be annotated is determined to be "hug" based on the correlation between the key description and the video description of the video to be annotated.

[0094] In some embodiments of the present application, the key descriptions corresponding to the tags may be screened through manual correction, and the key descriptions with poor relevance may be screened out and removed.

[0095] like Figure 7 As shown, in some embodiments of the present application, determining a label corresponding to a key description matching the first description, and labeling the first video by using the label, includes:

[0096] Step S71: if there are multiple key descriptions matching the first description;

[0097] Step S72, obtaining the priority of the corresponding tag according to the key description;

[0098] Step S73: Use the high-priority tag as the annotation tag of the first video, and annotate the first video with the annotation tag.

[0099] In this embodiment, in some cases, there may be multiple key descriptions that match the video description of the video to be annotated. If the multiple key descriptions all belong to the same tag, the tag is directly used to annotate the video to be annotated. If the multiple key descriptions do not belong to the same tag, but belong to multiple tags, the priorities of the corresponding multiple tags are obtained, and the tag with the highest priority is used as the preferred tag to annotate the video to be annotated.

[0100] In some embodiments of the present application, for multiple tags corresponding to multiple key descriptions and using tag priorities to determine the annotation tags, if the priority of a tag is lower than the priority of another tag, but the multiple key descriptions corresponding to the tag all match the video description of the video to be annotated. And in the key description of another high-priority tag, the number of key descriptions that match the video description of the video to be annotated is less than the number of key descriptions of the low-priority tag, the low-priority tag can be selected as the annotation tag of the video to be annotated.

[0101] Furthermore, in some embodiments of the present application, a corresponding reference value can be set. If the number of matches between the key description corresponding to the high-priority tag and the video description of the video to be labeled differs from the number of matches between the key description corresponding to the low-priority tag and the video description of the video to be labeled by more than the reference value, then the low-priority tag is considered to be more appropriate for the classification result of the video to be labeled. The low-priority tag should be used to label the video to be labeled. Conversely, if the corresponding number difference does not exceed the reference value, the high-priority tag should be used to label the video to be labeled.

[0102] The above technical solution provided by the embodiment of the present application has the following advantages over the prior art: The method provided by the embodiment of the present application obtains a natural language description of the video to be annotated by inputting the video to be annotated into a large multimodal model to utilize the multimodal large model's ability to understand and interpret the video, and then matches it with the key description corresponding to the label, and uses the label corresponding to the successfully matched key description as the annotation label of the video to be annotated, and annotates the video to be annotated. Compared with the traditional method of training annotation models, the cognitive ability of the large multimodal model is directly utilized to realize intelligent annotation of the video to be annotated. Manually annotated data is no longer used, and no more cost is spent on training models.

[0103] Furthermore, by generating corresponding regular expressions from key descriptions, the tag can be quickly matched with the video description of the video to be annotated, thereby further improving the efficiency of video annotation.

[0104] like Figure 8 As shown, the second aspect of the present application further proposes a video annotation system, comprising:

[0105] A video description generating module 801 is used to input a first video into a first model to obtain a first description of the first video, wherein the first model includes a multimodal large language model;

[0106] The video classification module 802 is used to determine a preset tag and a key description corresponding to the tag, and determine a key description matching the first description based on a preset matching strategy;

[0107] The video annotation module 803 is used to determine an annotation tag corresponding to the key description matching the first description, and annotate the first video using the annotation tag.

[0108] like Fig. 9 As shown, the third aspect of the present application further proposes a computer device: at least one communication interface 912; at least one bus 914 connected to the at least one communication interface 912; at least one processor 911 connected to the at least one bus 914; at least one memory 913 connected to the at least one bus, wherein the processor 911 is configured to execute the steps of the video annotation method described in any one of the above embodiments:

[0109] Inputting a first video into a first model to obtain a first description of the first video, wherein the first model includes a multimodal large language model;

[0110] Determine a preset tag and a key description corresponding to the tag, and determine a key description matching the first description based on a preset matching strategy;

[0111] Determine a label corresponding to a key description matching the first description, and label the first video using the label.

[0112] In some embodiments of the present application, determining a preset tag and a key description corresponding to the tag, and determining the key description matching the first description based on a preset matching strategy includes:

[0113] Determine the regular expression corresponding to the tag based on the key description of the tag;

[0114] searching the first description for a description statement satisfying the regular expression based on the regular expression, and if the description statement exists, considering that the key description matches the first description;

[0115] If the description statement does not exist, it is considered that the key description does not match the first description.

[0116] In some embodiments of the present application, determining a regular expression corresponding to a tag based on a key description of the tag includes:

[0117] Determining the verbs contained in the key description according to the content of the key description;

[0118] Acquire different expressions of the verb in corresponding tenses based on the verb;

[0119] A regular expression capable of matching different expressions is constructed according to the same parts of the verbs in different expressions.

[0120] In some embodiments of the present application, determining a regular expression corresponding to a tag based on a key description of the tag includes:

[0121] Determine the verbs and nouns contained in the key description and their corresponding order according to the content of the key description;

[0122] According to the semantics of the verb and the noun, a regular expression is constructed which starts with the verb, ends with the noun and can match multiple arbitrary characters in between.

[0123] In some embodiments of the present application, the key description is generated in the following manner:

[0124] Inputting the second video annotated with the label into the first model, and obtaining a second description about the label through the model input;

[0125] The tag and the second description are input into a second model, key phrases related to the tag in the second description are extracted through the second model, and the key phrases are used as the key description.

[0126] In some embodiments of the present application, the key phrase is used as the key description, including:

[0127] inputting the key phrase and the tag into a second model;

[0128] Through a predetermined prompt word project, it is determined whether the key phrase is related to the tag, and the related key phrase is used as the key description.

[0129] In some embodiments of the present application, determining a label corresponding to a key description matching the first description, and labeling the first video using the label, includes:

[0130] If there are multiple key descriptions matching the first description;

[0131] Then, the priority of the corresponding tag is obtained according to the key description;

[0132] The high-priority tag is used as the tag of the first video, and the first video is labeled by the tag.

[0133] The fourth aspect of the present application further provides a computer storage medium storing computer executable instructions, wherein the computer executable instructions are used to execute the steps of the video annotation method in any one of the above embodiments:

[0134] Inputting a first video into a first model to obtain a first description of the first video, wherein the first model includes a multimodal large language model;

[0135] Determine a preset tag and a key description corresponding to the tag, and determine a key description matching the first description based on a preset matching strategy;

[0136] Determine a label corresponding to a key description matching the first description, and label the first video using the label.

[0137] In some embodiments of the present application, determining a preset tag and a key description corresponding to the tag, and determining the key description matching the first description based on a preset matching strategy includes:

[0138] Determine the regular expression corresponding to the tag based on the key description of the tag;

[0139] searching the first description for a description statement satisfying the regular expression based on the regular expression, and if the description statement exists, considering that the key description matches the first description;

[0140] If the description statement does not exist, it is considered that the key description does not match the first description.

[0141] In some embodiments of the present application, determining a regular expression corresponding to a tag based on a key description of the tag includes:

[0142] Determining the verbs contained in the key description according to the content of the key description;

[0143] Acquire different expressions of the verb in corresponding tenses based on the verb;

[0144] A regular expression capable of matching different expressions is constructed according to the same parts of the verbs in different expressions.

[0145] In some embodiments of the present application, determining a regular expression corresponding to a tag based on a key description of the tag includes:

[0146] Determine the verbs and nouns contained in the key description and their corresponding order according to the content of the key description;

[0147] According to the semantics of the verb and the noun, a regular expression is constructed which starts with the verb, ends with the noun and can match multiple arbitrary characters in between.

[0148] In some embodiments of the present application, the key description is generated in the following manner:

[0149] Inputting the second video annotated with the label into the first model, and obtaining a second description about the label through the model input;

[0150] The tag and the second description are input into a second model, key phrases related to the tag in the second description are extracted through the second model, and the key phrases are used as the key description.

[0151] In some embodiments of the present application, the key phrase is used as the key description, including:

[0152] inputting the key phrase and the tag into a second model;

[0153] Through a predetermined prompt word project, it is determined whether the key phrase is related to the tag, and the related key phrase is used as the key description.

[0154] In some embodiments of the present application, determining a label corresponding to a key description matching the first description, and labeling the first video using the label, includes:

[0155] If there are multiple key descriptions matching the first description;

[0156] Then, the priority of the corresponding tag is obtained according to the key description;

[0157] The high-priority tag is used as the tag of the first video, and the first video is labeled by the tag.

[0158] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0159] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0160] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0161] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A video annotation method, characterized in that: include: Inputting a first video into a first model to obtain a first description of the first video, wherein the first model includes a multimodal large language model; Determine a preset tag and a key description corresponding to the tag, and determine a key description matching the first description based on a preset matching strategy; Determine a label corresponding to a key description matching the first description, and label the first video using the label.

2. The method according to claim 1, characterized in that The determining of a preset tag and a key description corresponding to the tag, and determining a key description matching the first description based on a preset matching strategy, includes: Determine the regular expression corresponding to the tag based on the key description of the tag; searching the first description for a description statement satisfying the regular expression based on the regular expression, and if the description statement exists, considering that the key description matches the first description; If the description statement does not exist, it is considered that the key description does not match the first description.

3. The method according to claim 2, characterized in that The step of determining a regular expression corresponding to a tag based on a key description of the tag includes: Determining the verbs contained in the key description according to the content of the key description; Acquire different expressions of the verb in corresponding tenses based on the verb; A regular expression capable of matching different expressions is constructed according to the same parts of the verbs in different expressions.

4. The method according to claim 2, characterized in that: The step of determining a regular expression corresponding to a tag based on a key description of the tag includes: Determine the verbs and nouns contained in the key description and their corresponding order according to the content of the key description; According to the semantics of the verb and the noun, a regular expression is constructed which starts with the verb, ends with the noun and can match multiple arbitrary characters in between.

5. The method according to claim 1, characterized in that The key description is generated in the following manner: Inputting the second video annotated with the label into the first model, and obtaining a second description about the label through the model input; The tag and the second description are input into a second model, key phrases related to the tag in the second description are extracted through the second model, and the key phrases are used as the key description.

6. The method according to claim 5, characterized in that The using the key phrase as the key description includes: inputting the key phrase and the tag into a second model; Through a predetermined prompt word project, it is determined whether the key phrase is related to the tag, and the related key phrase is used as the key description.

7. The method according to claim 1, characterized in that The determining a label corresponding to the key description matching the first description, and labeling the first video by using the label, includes: If there are multiple key descriptions matching the first description; Then, the priority of the corresponding tag is obtained according to the key description; The high-priority tag is used as the tag of the first video, and the first video is labeled by the tag.

8. A video annotation system, characterized in that: include: a video description generating module, configured to input a first video into a first model to obtain a first description of the first video, wherein the first model includes a multimodal large language model; A video classification module, used to determine a preset tag and a key description corresponding to the tag, and determine a key description matching the first description based on a preset matching strategy; The video annotation module is used to determine an annotation tag corresponding to a key description matching the first description, and annotate the first video using the annotation tag.

9. A computer device: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; at least one memory connected to the at least one bus, wherein: The processor is configured to execute the video annotation method according to any one of claims 1 to 7.

10. A computer storage medium storing computer executable instructions, wherein the computer executable instructions are used to execute the video annotation method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • An intention recognition method and a system

    CN109388700A

  • Multi-intention selection method and device and terminal equipment

    CN109948155A

  • Label prediction method, text prediction model training method, device and equipment

    CN115631496A

  • Method and device for generating annotation information, electronic equipment and storage medium

    CN118113869A

  • Image annotation method and apparatus

    WO2024045641A1