Video fine-grained understanding labeling method and device based on multi-modal large model

Abstract

The invention provides a video fine-grained understanding labeling method and device based on a multi-modal large model, relates to the field of video text labeling, and aims to realize automatic video fine-grained understanding labeling. According to the method, fine-grained static information extraction is realized through a plurality of key frames of a target video clip; obtaining a plurality of candidate static fine-grained text labels according to the static fine-grained description information; improving the extraction process of dynamic information in the form of video questions and answers, and obtaining a plurality of candidate dynamic fine-grained text labels according to the answer to the target question and the dynamic fine-grained description information; utilizing an evaluation model to obtain a confidence score corresponding to each candidate static fine-grained text label and a confidence score corresponding to each candidate dynamic fine-grained text label; and finally, taking the plurality of candidate static fine-grained text annotations and the corresponding confidence scores thereof, and the candidate dynamic fine-grained text annotations and the corresponding confidence scores thereof as fine-grained annotation information of the target video clip.

Description

Technical Field

[0001] The present application relates to the technical field of video text annotation, and in particular to a video fine-grained understanding annotation method and device based on a multimodal large model. Background Art

[0002] Due to the lack of high-quality fine-grained video understanding datasets, existing methods for video library clip retrieval are basically limited to coarse-grained understanding of videos and cannot cope well with fine-grained query scenarios.

[0003] The existing coarse-grained ground-truth (GT) annotation cannot mark the differences between similar clips, and the algorithm cannot learn the fine-grained features in the video well, so as to understand the differences between similar videos. However, marking fine-grained information requires high labor costs and requires the annotator to have certain professional capabilities, which is not conducive to scalability.

[0004] Therefore, how to achieve automatic fine-grained video understanding annotation to promote fine-grained video understanding in the field of fine-grained video understanding is a technical problem that needs to be solved urgently. Summary of the invention

[0005] In view of the above problems, an embodiment of the present application provides a method and device for fine-grained video understanding and annotation based on a multimodal large model, so as to overcome the above problems or at least partially solve the above problems.

[0006] In a first aspect of an embodiment of the present application, a video fine-grained understanding and annotation method based on a multimodal large model is disclosed, the method comprising:

[0007] Obtaining a target video segment and a coarse-grained text annotation corresponding to the target video segment;

[0008] Inputting multiple key frames of the target video clip and the coarse-grained text annotations into the image multimodal large model to generate static information to obtain static fine-grained description information, and obtaining multiple candidate static fine-grained text annotations based on the static fine-grained description information;

[0009] Inputting the target question for dynamic content understanding and the target video clip into the video multimodal large model for dynamic information generation, obtaining the answer to the target question and dynamic fine-grained description information, and obtaining multiple candidate dynamic fine-grained text annotations based on the answer to the target question and the dynamic fine-grained description information;

[0010] Using the evaluation model, respectively evaluate the matching degree between the target video segment and the multiple candidate static fine-grained text annotations and the multiple candidate dynamic fine-grained text annotations to obtain a confidence score corresponding to each candidate static fine-grained text annotation and a confidence score corresponding to each candidate dynamic fine-grained text annotation;

[0011] The multiple candidate static fine-grained text annotations and the confidence score corresponding to each candidate static fine-grained text annotation, as well as the multiple candidate dynamic fine-grained text annotations and the confidence score corresponding to each candidate dynamic fine-grained text annotation are used as the fine-grained annotation information of the target video segment.

[0012] Optionally, the method further comprises:

[0013] Filtering a static fine-grained text annotation from the plurality of candidate static fine-grained text annotations according to a confidence score corresponding to each candidate static fine-grained text annotation;

[0014] Filtering a dynamic fine-grained text annotation from the multiple candidate dynamic fine-grained text annotations according to the confidence score corresponding to each candidate dynamic fine-grained text annotation;

[0015] The multiple candidate static fine-grained text annotations and the confidence score corresponding to each candidate static fine-grained text annotation, and the multiple candidate dynamic fine-grained text annotations and the confidence score corresponding to each candidate dynamic fine-grained text annotation are used as the fine-grained annotation information of the target video segment, including:

[0016] The static fine-grained text annotation and the confidence score corresponding to the static fine-grained text annotation, as well as the dynamic fine-grained text annotation and the confidence score corresponding to the dynamic fine-grained text annotation, are used as the fine-grained annotation information of the target video segment.

[0017] Optionally, the evaluation model is constructed by the following steps:

[0018] Constructing a training data set according to the sample coarse-grained annotations corresponding to the sample video clips, wherein each training data in the training data set includes: a sample video clip, a coarse-grained annotation of the sample video clip, and an enhanced sample;

[0019] Constructing a loss value based on the training data set; the loss value is used to represent the matching loss between the video clip and the text annotation;

[0020] The model parameters of the evaluation model are optimized according to the loss value, and the evaluation model is obtained after the optimization result conditions are met.

[0021] Optionally, a training data set is constructed based on the sample coarse-grained annotations corresponding to the sample video clips, including:

[0022] Generate rewritten samples according to the coarse-grained annotation of the samples; the rewritten samples include: semantically unchanged rewritten positive samples, static content perturbation rewritten negative samples, and dynamic content perturbation rewritten negative samples;

[0023] According to the similarity between the rewritten sample and the sample coarse-grained annotation, an enhanced sample is selected from the rewritten sample, wherein the enhanced sample includes: an enhanced positive sample, an enhanced static content perturbation rewritten negative sample, and an enhanced dynamic content perturbation rewritten negative sample;

[0024] The sample video clip, the sample coarse-grained annotation, and the enhanced sample are used as a training data.

[0025] Optionally, the loss value includes a comparison loss value and a matching loss value, and constructing the loss value according to the training data set includes:

[0026] The enhanced static content perturbation rewriting negative samples and the enhanced dynamic content perturbation rewriting negative samples in the training data set are labeled as negative samples, and the coarse-grained annotations of the samples or the enhanced positive samples are labeled as positive samples;

[0027] Constructing a contrast loss value according to the negative sample annotation and the positive sample annotation; the contrast loss value includes: a contrast loss first item loss value and a contrast loss second item loss value; the contrast loss first item loss value is used to characterize the matching between the sample video segment and the negative sample annotation, and the contrast loss second item loss value is used to characterize the matching between the positive sample annotation and the sample video segment;

[0028] According to the negative sample annotation and the positive sample annotation, a matching loss value is constructed, and the matching loss value includes: a first matching loss value, a second matching loss value and a third matching loss value; the first matching loss value is used to characterize the matching situation between the sample video clip and the matched positive sample annotation, the second matching loss value is used to characterize the matching situation between the sample video clip and the unmatched negative sample annotation, and the third matching loss value is used to characterize the matching situation between the positive sample annotation and the unmatched sample video clip.

[0029] Optionally, before inputting the multiple key frames of the target video segment and the coarse-grained text annotations into the image multimodal large model for static information generation, the method further includes:

[0030] According to the content change of the target video segment, the target video segment is divided into a plurality of video sub-segments;

[0031] Using the middle frame of each video sub-segment as the key frame of the video sub-segment;

[0032] A plurality of key frames of the target video segment are obtained according to the key frames of the plurality of video sub-segments.

[0033] Optionally, the static description fine-grained information includes: foreground detail information, background detail information and fine-grained description information;

[0034] According to the static fine-grained description information, a plurality of candidate static fine-grained text annotations are obtained, including:

[0035] The foreground detail information, the background detail information, the fine-grained description information and the coarse-grained text annotation are input into a large language model to extract static attribute information and generate annotations, so as to obtain a plurality of candidate static fine-grained text annotations.

[0036] Optionally, before inputting the target question for dynamic content understanding and the target video clip into the video multimodal large model for dynamic information generation, the method further includes:

[0037] Inputting the coarse-grained text annotation into a large language model to generate questions, thereby obtaining target questions for dynamic content understanding;

[0038] According to the answer to the target question and the dynamic fine-grained description information, a plurality of candidate dynamic fine-grained text annotations are obtained, including:

[0039] The answer to the target question, the dynamic fine-grained description information and the coarse-grained text annotation are input into a large language model to extract dynamic attribute information and generate annotations, so as to obtain a plurality of candidate dynamic fine-grained text annotations.

[0040] According to a second aspect of the embodiment of the present application, a video fine-grained understanding and annotation device based on a multimodal large model is disclosed, the device comprising:

[0041] An acquisition module, used to acquire a target video segment and a coarse-grained text annotation corresponding to the target video segment;

[0042] A static information module, used for inputting a plurality of key frames of the target video segment and the coarse-grained text annotations into the image multimodal large model to generate static information, obtain static fine-grained description information, and obtain a plurality of candidate static fine-grained text annotations according to the static fine-grained description information;

[0043] A dynamic information module is used to input the target question for dynamic content understanding and the target video clip into the video multimodal large model to generate dynamic information, obtain the answer to the target question and dynamic fine-grained description information, and obtain multiple candidate dynamic fine-grained text annotations based on the answer to the target question and the dynamic fine-grained description information;

[0044] An evaluation module, used to evaluate the matching degree between the target video segment and the multiple candidate static fine-grained text annotations and the multiple candidate dynamic fine-grained text annotations respectively by using the evaluation model, and obtain a confidence score corresponding to each candidate static fine-grained text annotation and a confidence score corresponding to each candidate dynamic fine-grained text annotation;

[0045] The annotation module is used to use the multiple candidate static fine-grained text annotations and the confidence score corresponding to each candidate static fine-grained text annotation, as well as the multiple candidate dynamic fine-grained text annotations and the confidence score corresponding to each candidate dynamic fine-grained text annotation as the fine-grained annotation information of the target video segment.

[0046] According to a third aspect of an embodiment of the present application, an electronic device is disclosed, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the video fine-grained understanding and annotation method based on a multimodal large model described in the first aspect of the embodiment of the present application are implemented.

[0047] The fourth aspect of the embodiments of the present application discloses a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the video fine-grained understanding and annotation method based on a multimodal large model described in the first aspect of the embodiments of the present application are implemented.

[0048] The embodiments of the present application include the following advantages:

[0049] In the embodiment of the present application, based on the target video segment and coarse-grained text annotation, automatic video fine-grained understanding annotation is realized based on the multimodal large model. For the static fine-grained information annotation contained in the target video segment, static information extraction is realized through multiple key frames in the target video segment, and the multiple key frames of the target video segment and the coarse-grained text annotation are input into the image multimodal large model for static information generation to obtain static fine-grained description information, thereby obtaining multiple candidate static fine-grained text annotations based on the static fine-grained description information; based on the static fine-grained description information obtained from multiple key frames, instead of obtaining coarse-grained information that describes the target video segment as a whole, the static fine-grained information in the target video segment is annotated according to the static fine-grained description information. For the annotation of dynamic fine-grained information contained in the target video clip, the dynamic information extraction process is improved through the form of video question-answering. The target question for dynamic content understanding and the target video clip are input into the video multimodal large model for dynamic information generation, and the answer to the target question and dynamic fine-grained description information are obtained. Then, based on the answer to the target question and the dynamic fine-grained description information, multiple candidate dynamic fine-grained text annotations are obtained to achieve the annotation of dynamic fine-grained information in the target video clip.

[0050] In addition, the evaluation model is used to evaluate the matching degree between the target video clip and multiple candidate static fine-grained text annotations and multiple candidate dynamic fine-grained text annotations, and the confidence score corresponding to each candidate static fine-grained text annotation and the confidence score corresponding to each candidate dynamic fine-grained text annotation are obtained; according to the confidence score, the credibility of each fine-grained text annotation can be determined to achieve effective control of hallucination annotations.

[0051] In this way, by annotating the static fine-grained information and dynamic fine-grained information contained in the target video clip, multiple candidate static fine-grained text annotations and the confidence scores corresponding to each candidate static fine-grained text annotation, as well as multiple candidate dynamic fine-grained text annotations and the confidence scores corresponding to each candidate dynamic fine-grained text annotation are used as the fine-grained annotation information of the target video clip, to achieve automatic video fine-grained understanding annotation, which promotes fine-grained video understanding in the field of fine-grained video understanding. Through this automatic video fine-grained understanding annotation scheme, a fine-grained video annotation dataset can be quickly constructed, greatly reducing the annotation cost while ensuring the annotation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0053] Figure 1 This is a comparison diagram of retrieval results of coarse-grained text annotation and fine-grained text annotation provided in an embodiment of the present application;

[0054] Figure 2 It is a flowchart of the steps of a video fine-grained understanding and annotation method based on a multimodal large model provided in an embodiment of the present application;

[0055] Figure 3 It is a schematic diagram of the processing result of fine-grained text annotation by an evaluation model provided in an embodiment of the present application;

[0056] Figure 4 It is a flowchart of another method for fine-grained video understanding and annotation based on a multimodal large model provided in an embodiment of the present application;

[0057] Figure 5 This is an overall architecture diagram of a video fine-grained understanding and annotation method based on a multimodal large model provided in an embodiment of the present application;

[0058] Figure 6It is a structural schematic diagram of a video fine-grained understanding and annotation device based on a multimodal large model provided in an embodiment of the present application;

[0059] Figure 7 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0061] There are two main types of existing video-text annotation solutions: manual annotation and automatic annotation. For manual annotation, especially for annotating fine-grained content in videos, video-text annotation requires a lot of work and domain knowledge, making these data sets difficult to scale. For automated annotation solutions, some solutions obtain annotated text through web crawling, voice transcription, etc., but the cross-modal alignment of these annotations is relatively noisy; with the development of multimodal basic models and large language model technologies, automatic annotation of large-scale video-text datasets has become feasible, but the accuracy of existing automatic annotation solutions based on multimodal basic models and large language model technologies is difficult to guarantee, and the annotation is not fine-grained enough, especially for dynamic details in the video (for example, actions and interactive relationships, etc.), or its fine-grained annotations are mostly derived from collected video subtitles or other text tags, and its annotation content cannot well reflect the content of the visual modality.

[0062] Therefore, existing video-text annotation schemes have the following problems: (1) The manual annotation cost of fine-grained annotation is high and difficult to scale; (2) Existing automatic annotation techniques cannot generate accurate fine-grained annotation information, especially lacking dynamic content; and there is a lack of effective control over annotation hallucinations.

[0063] In order to overcome the limitations of related technologies, an embodiment of the present application provides a video fine-grained understanding and annotation method based on a multimodal large model, which realizes automated video fine-grained understanding and annotation based on a multimodal large model. For the annotation of static fine-grained information contained in the target video segment, fine-grained static information extraction is realized through multiple key frames in the target video segment to obtain static fine-grained information; for the annotation of dynamic fine-grained information contained in the target video segment, the extraction process of dynamic information is improved through the form of video question-answering to obtain dynamic fine-grained description information. In addition, the confidence score corresponding to each candidate static fine-grained text annotation and the confidence score corresponding to each candidate dynamic fine-grained text annotation are obtained using the evaluation model.

[0064] In this way, by annotating the static fine-grained information and dynamic fine-grained information contained in the target video clip, multiple candidate static fine-grained text annotations and their corresponding confidence scores, candidate dynamic fine-grained text annotations and their corresponding confidence scores are used as the fine-grained annotation information of the target video clip, thereby realizing automatic video fine-grained understanding annotation, solving the hallucination phenomenon of multimodal large models, and filtering out annotations containing inaccurate content through confidence scores.

[0065] The following is a detailed description of the video fine-grained understanding and annotation method based on a multimodal large model provided in an embodiment of the present application in conjunction with the accompanying drawings.

[0066] In order to better understand the technical solution of the present application, the “coarse-grained text annotation” and “fine-grained text annotation” involved in the embodiments of the present application are first explained.

[0067] Reference Figure 1 As shown, Figure 1 This is a comparison chart of the retrieval results of coarse-grained text annotation and fine-grained text annotation provided in an embodiment of the present application. Among them, coarse-grained text annotation refers to a text description with less information description. When searching the video library based on coarse-grained text annotation, in addition to being able to retrieve the target video segment corresponding to the coarse-grained text annotation, other potential segments that are not annotated can also be retrieved. In other words, it is difficult to accurately retrieve a one-to-one target video segment based on coarse-grained text annotation. Figure 1 As shown in the figure, for a coarse-grained text annotation "a person opens a bag", three video clips can be retrieved in the video library, but when performing text annotation, only one video clip is annotated, and other potential matches in the video library are ignored. As a result, problems will arise in both model training (that is, the training process ignores other positive samples) and reasoning (may only consider some of the videos) based on the video library with coarse-grained text annotations.

[0068] Fine-grained text annotation refers to a more comprehensive text description of information. When searching the video library based on fine-grained text annotation, the one-to-one target video clips can be accurately retrieved. That is, fine-grained text annotation can eliminate the one-to-many matching existing in coarse-grained text annotation and achieve accurate video retrieval. Figure 1 As shown, for a fine-grained text annotation "a person on the sofa opens a colorful bag, revealing a mobile phone", a unique corresponding video clip can be retrieved in the video library. Based on the solution provided by the embodiment of the present application, the fine-grained information contained in the target video clip can be annotated, thereby promoting fine-grained video understanding based on the fine-grained annotation information.

[0069] The following sections will introduce them in Sections 1.1 and 1.2 respectively.

[0070] 1.1 Methods for automatically generating fine-grained text annotations:

[0071] Reference Figure 2 As shown, Figure 2 1 is a flowchart of the steps of a video fine-grained understanding and annotation method based on a multimodal large model provided in an embodiment of the present application. Figure 2 As shown, the video fine-grained understanding and annotation method based on the multimodal large model may include steps S210 to S250:

[0072] Step S210: obtaining a target video segment and a coarse-grained text annotation corresponding to the target video segment.

[0073] The target video segment refers to any video segment in the existing coarse-grained video text annotation database, and the coarse-grained text annotation refers to the text annotation corresponding to the target video segment. The form of coarse-grained text annotation is (V,t s ,t e ,q), respectively representing the target video segment, start time, end time, and query text (the query text is a description text containing less descriptive information).

[0074] In an embodiment of the present application, fine-grained annotation is performed on the basis of an existing coarse-grained video text annotation database; obtaining a target video segment and a coarse-grained text annotation corresponding to the target video segment means: obtaining any video segment as a target video segment from an existing coarse-grained video text annotation database, and obtaining the coarse-grained text annotation corresponding to the target video segment.

[0075] Step S220: multiple key frames of the target video clip and the coarse-grained text annotations are input into the image multimodal large model to generate static information to obtain static fine-grained description information, and multiple candidate static fine-grained text annotations are obtained based on the static fine-grained description information.

[0076] In the embodiment of the present application, static fine-grained description information refers to a type of fine-grained information, and the static fine-grained description information represents the fine-grained description information of the video frame in the target video segment. In order to annotate the static fine-grained information contained in the target video segment, fine-grained static information extraction is implemented based on multiple key frames of the target video segment to obtain static fine-grained information; wherein the key frame of the target video segment can be a video frame in a sub-segment corresponding to different scenes in the target video segment.

[0077] For multiple key frames of the target video clip, multiple key frames and coarse-grained text annotations are input into the image multimodal large model for static information generation to obtain static fine-grained description information; wherein the static fine-grained description information includes the static fine-grained description information corresponding to each key frame. Thus, multiple candidate static fine-grained text annotations are obtained according to the static fine-grained description information; wherein the candidate static fine-grained text annotation refers to a text annotation containing static fine-grained description information.

[0078] The static fine-grained description information is obtained based on multiple key frames, rather than obtaining coarse-grained information that describes the target video segment as a whole. Therefore, the static fine-grained information in the target video segment is labeled according to the static fine-grained description information.

[0079] Step S230: Input the target question for dynamic content understanding and the target video clip into the video multimodal large model for dynamic information generation to obtain the answer to the target question and dynamic fine-grained description information, and obtain multiple candidate dynamic fine-grained text annotations based on the answer to the target question and the dynamic fine-grained description information.

[0080] In an embodiment of the present application, dynamic fine-grained description information refers to a type of information in fine-grained information, and the dynamic fine-grained description information represents fine-grained description information of dynamic changes in scenes between video frames in a target video clip. In order to annotate the dynamic fine-grained information contained in the target video clip, the dynamic information extraction process is improved through the form of video question-answering, and target questions for understanding dynamic content are obtained. The target questions for understanding dynamic content can be a series of questions generated based on coarse-grained text annotations. For example, for a coarse-grained text annotation "the person put down the bag between 4 seconds and 9 seconds", one of the target questions can be "where did the person put the bag?".

[0081] The target question and target video clip for dynamic content understanding are input into the video multimodal large model for dynamic information generation, and the answer to the target question and dynamic fine-grained description information are obtained. The answer to the target question refers to the answer to the target question. For example, for the target question "Where did this person put the bag?", the corresponding answer to the target question may be "This person put the bag on the table." Thus, based on the answer to the target question and the dynamic fine-grained description information, multiple candidate dynamic fine-grained text annotations are obtained to achieve the annotation of dynamic fine-grained information in the target video clip; wherein, the candidate dynamic fine-grained text annotation refers to the text annotation containing dynamic fine-grained description information.

[0082] Step S240: Use the evaluation model to respectively evaluate the target video clip and the matching degree between the multiple candidate static fine-grained text annotations and the multiple candidate dynamic fine-grained text annotations to obtain a confidence score corresponding to each candidate static fine-grained text annotation and a confidence score corresponding to each candidate dynamic fine-grained text annotation.

[0083] In the embodiment of the present application, considering that a large multimodal model (e.g., a large image multimodal model or a large video multimodal model) has hallucination phenomenon, it is easy to generate erroneous information. Therefore, the evaluation model is used to evaluate the confidence of the multiple candidate static fine-grained text annotations obtained in the above step S220 and the multiple candidate dynamic fine-grained text annotations obtained in step S230.

[0084] The evaluation model can judge the degree of match between the target video clip and the fine-grained text annotations (i.e., the candidate static fine-grained text annotations and the candidate dynamic fine-grained text annotations), that is, judge the degree of match between the coarse-grained text annotations and the fine-grained text annotations of the target video clip; for fine-grained text annotations with a high degree of match, the evaluation model outputs a higher confidence score (the first confidence score or the second confidence score), and for fine-grained text annotations with a low degree of match, the evaluation model outputs a lower confidence score, so that fine-grained text annotations of poor quality (containing erroneous information) can be distinguished according to the confidence score.

[0085] For example, Figure 3 It is a schematic diagram of the processing results of fine-grained text annotation by an evaluation model provided in an embodiment of the present application. For example, the evaluation model evaluates the matching degree of the coarse-grained text annotation "a person opens a bag" and the fine-grained text annotation "a person, wearing a green sweatshirt and glasses, sitting on a sofa, opens a colorful bag with both hands, revealing a mobile phone", and obtains a confidence score of 3.82. For another example, the evaluation model evaluates the matching degree of the coarse-grained text annotation "a person opens a bag" and the fine-grained text annotation "a person in the kitchen holds a bag with a cherry pattern and pulls the drawstring to open the bag", and obtains a confidence score of 3.23.

[0086] Specifically, the evaluation model is used to evaluate the matching degree between the target video clip and multiple candidate static fine-grained text annotations, and the confidence score corresponding to each candidate static fine-grained text annotation is obtained; the evaluation model is used to evaluate the matching degree between the target video clip and multiple candidate dynamic fine-grained text annotations, and the confidence score corresponding to each candidate dynamic fine-grained text annotation is obtained.

[0087] Step S250: The multiple candidate static fine-grained text annotations and the confidence score corresponding to each candidate static fine-grained text annotation, as well as the multiple candidate dynamic fine-grained text annotations and the confidence score corresponding to each candidate dynamic fine-grained text annotation, are used as the fine-grained annotation information of the target video segment.

[0088] In the embodiment of the present application, the annotation form of the fine-grained annotation information of the target video segment is: They respectively represent the target video segment, the start time, the end time, multiple candidate static fine-grained text annotations, the confidence scores corresponding to the multiple candidate static fine-grained text annotations, and multiple candidate dynamic fine-grained text annotations, and the confidence scores corresponding to the multiple candidate dynamic fine-grained text annotations.

[0089] Through the above implementation process, the static fine-grained information and dynamic fine-grained information contained in the target video clip are annotated, and multiple candidate static fine-grained text annotations and the confidence scores corresponding to each candidate static fine-grained text annotation, as well as multiple candidate dynamic fine-grained text annotations and the confidence scores corresponding to each candidate dynamic fine-grained text annotation, are used as the fine-grained annotation information of the target video clip, so as to realize the automatic video fine-grained understanding annotation, and promote the fine-grained video understanding in the field of fine-grained video understanding. Through this automatic video fine-grained understanding annotation scheme, a fine-grained video annotation dataset can be quickly constructed, which greatly reduces the annotation cost while ensuring the annotation accuracy.

[0090] The present application is described in detail below in conjunction with the above-mentioned implementation steps.

[0091] In some specific implementations, before inputting the multiple key frames of the target video segment and the coarse-grained text annotations into the image multimodal large model for static information generation in step S220, the following steps A1 to A3 are also included:

[0092] Step A1: dividing the target video segment into a plurality of video sub-segments according to the content change of the target video segment;

[0093] Step A2: taking the middle frame of each video sub-segment as the key frame of the video sub-segment;

[0094] Step A3: obtaining a plurality of key frames of the target video segment according to the key frames of the plurality of video sub-segments.

[0095] In the embodiment of the present application, multiple key frames of the target video segment are obtained according to step A1 to step A3. Specifically, according to the content change of the target video segment, the target video segment is divided into multiple video sub-segments, and the PySceneDetect threshold can be adaptively adjusted to divide the target video segment into multiple video sub-segments (such as L). For example, a threshold is set, and when the RGB value change of the image frame in the target video segment is greater than the threshold, the target video segment is divided. Finally, the middle frame of each video sub-segment is used as the key frame of the video sub-segment, thereby multiple key frames of the target video segment.

[0096] In some specific implementations, the static fine-grained description information includes: foreground detail information, background detail information and fine-grained description information. Among them, the foreground detail information refers to the subject information (people or objects) in the key frame, the background detail information refers to the background information in the key frame, and the fine-grained description information refers to a comprehensive description information that combines the foreground detail information and the background detail information.

[0097] Specifically, in the above step S220, "inputting the multiple key frames of the target video segment and the coarse-grained text annotations into the image multimodal large model to generate static information to obtain static fine-grained description information" can be expressed as:

[0098]

[0099] Among them, ImageLMM represents the image multimodal large model, f i represents a key frame, q represents a coarse-grained text annotation, Represents foreground detail information, Represents background details, D i Represents fine-grained description information.

[0100] Optionally, the "obtaining multiple candidate static fine-grained text annotations based on the static fine-grained description information" in the above step S220 may specifically include: inputting the foreground detail information, the background detail information, the fine-grained description information and the coarse-grained text annotations into a large language model to extract static attribute information and generate annotations, so as to obtain multiple candidate static fine-grained text annotations.

[0101] For example, multiple candidate static fine-grained text annotations can be expressed as:

[0102]

[0103] in, Represents multiple candidate static fine-grained text annotations, with a total of N s candidate static fine-grained text annotations; LLM stands for Large Language Model.

[0104] In some specific implementations, before step S230 of inputting the target question for dynamic content understanding and the target video clip into the video multimodal large model for dynamic information generation, the step further includes: inputting the coarse-grained text annotation into the large language model for question generation to obtain the target question for dynamic content understanding.

[0105] In an embodiment of the present application, in order to annotate the dynamic fine-grained information contained in the target video clip, the process of extracting dynamic information is improved through video question-answering, which prompts the large language model to generate target questions for dynamic content understanding based on coarse-grained text annotations.

[0106] For example, the target problem for dynamic content understanding is expressed as:

[0107]

[0108] in, represents the target problem for dynamic content understanding, with a total of N qa A target question.

[0109] Afterwards, the target question for dynamic content understanding and the target video clip are input into the video multimodal large model for dynamic information generation to obtain the answer to the target question and dynamic fine-grained description information.

[0110] For example, the answer to the target question and the dynamic fine-grained description information can be expressed as:

[0111]

[0112] Among them, VideoLMM represents the video multimodal large model. represents the answer to the target question, each answer to the target question corresponds to a target question, Represents dynamic fine-grained description information.

[0113] Furthermore, the above step S230 of "obtaining multiple candidate dynamic fine-grained text annotations based on the target question answer and the dynamic fine-grained description information" may specifically include: inputting the target question answer, the dynamic fine-grained description information and the coarse-grained text annotation into a large language model to extract dynamic attribute information and generate annotations, so as to obtain multiple candidate dynamic fine-grained text annotations.

[0114] For example, multiple candidate dynamic fine-grained text annotations can be expressed as:

[0115]

[0116] in, Represents multiple candidate dynamic fine-grained text annotations, with a total of N d Candidate dynamic fine-grained text annotations.

[0117] In some specific implementations, the following steps are further included after step S240:

[0118] Step C1: selecting a static fine-grained text annotation from the plurality of candidate static fine-grained text annotations according to a confidence score corresponding to each candidate static fine-grained text annotation;

[0119] Step C2: Filter out a dynamic fine-grained text annotation from the multiple candidate dynamic fine-grained text annotations according to the confidence score corresponding to each candidate dynamic fine-grained text annotation.

[0120] In an embodiment of the present application, the confidence score corresponding to the candidate static fine-grained text annotation can reflect the quality of the candidate static fine-grained text annotation, that is, it can be determined whether the candidate static fine-grained text annotation contains error information based on the confidence score, and thus, based on the confidence score corresponding to each candidate static fine-grained text annotation, the static fine-grained text annotation that does not contain error information is screened out from multiple candidate static fine-grained text annotations. Similarly, the confidence score corresponding to the candidate dynamic fine-grained text annotation can reflect the quality of the candidate dynamic fine-grained text annotation, that is, it can be determined whether the candidate dynamic fine-grained text annotation contains error information based on the confidence score, and thus, based on the confidence score corresponding to each candidate dynamic fine-grained text annotation, the dynamic fine-grained text annotation that does not contain error information is screened out from multiple candidate dynamic fine-grained text annotations.

[0121] Furthermore, in the above step S250, “using the multiple candidate static fine-grained text annotations and the confidence score corresponding to each candidate static fine-grained text annotation, and the multiple candidate dynamic fine-grained text annotations and the confidence score corresponding to each candidate dynamic fine-grained text annotation as the fine-grained annotation information of the target video segment” may specifically include:

[0122] The static fine-grained text annotation and the confidence score corresponding to the static fine-grained text annotation, as well as the dynamic fine-grained text annotation and the confidence score corresponding to the dynamic fine-grained text annotation, are used as the fine-grained annotation information of the target video segment.

[0123] In an embodiment of the present application, inaccurate annotation data can be automatically filtered out through confidence scores. Constructing a fine-grained video annotation dataset according to this method can improve the accuracy of fine-grained video understanding and construct a new benchmark for fine-grained video library segment retrieval.

[0124] 1.2 Evaluation model construction process:

[0125] In a specific embodiment, the evaluation model is constructed by following steps D1 to D3:

[0126] Step D1: construct a training data set according to the sample coarse-grained annotations corresponding to the sample video clips, where each training data in the training data set includes: a sample video clip, the coarse-grained annotations of the sample video clip, and an enhanced sample.

[0127] The sample video clip refers to the video clip used for fine-tuning the evaluation model. Each sample video clip corresponds to a sample coarse-grained annotation, which is a text description with less information. The enhanced sample is a sample annotation generated based on the sample coarse-grained annotation. A training data set is constructed based on the sample coarse-grained annotation corresponding to each sample video clip, and the training data set is used to fine-tune the evaluation model.

[0128] Optionally, a training data set is constructed according to the sample coarse-grained annotations corresponding to the sample video clips, including sub-steps D11 to D13:

[0129] Step D11: Generate rewritten samples according to the coarse-grained annotation of the samples; the rewritten samples include: semantically unchanged rewritten positive samples, static content perturbation rewritten negative samples, and dynamic content perturbation rewritten negative samples.

[0130] Step D12: According to the similarity between the rewritten sample and the coarse-grained annotation of the sample, an enhanced sample is selected from the rewritten sample, and the enhanced sample includes: an enhanced positive sample, an enhanced static content perturbation rewritten negative sample, and an enhanced dynamic content perturbation rewritten negative sample.

[0131] Step D13: The sample video clip, the sample coarse-grained annotation, and the enhanced sample are used as a training data.

[0132] In the embodiment of the present application, for each sample coarse-grained annotation, a rewritten sample is generated according to the large language model, that is, a semantically unchanged rewritten positive sample is generated. Static content perturbation rewrites negative samples Dynamic content perturbation rewrites negative samples

[0133] For example, the generation method of generating rewriting samples is expressed as:

[0134]

[0135] Among them, LLM represents the large language model and q represents the coarse-grained labeling of samples.

[0136] After obtaining the rewritten samples, a similarity calculation model (for example, the Sentence BERT model) can be used to calculate the similarity between the rewritten samples and the coarse-grained annotations of the samples, so as to select enhanced samples from the rewritten samples according to the similarity, that is, to select the most appropriate perturbed positive and negative samples.

[0137] For example, selecting an enhanced sample from the rewritten sample can be expressed as:

[0138]

[0139]

[0140] Among them, SentenceBERT represents a sentence embedding model based on the BERT model, which is mainly used to process sentence-level tasks, such as obtaining a vector representation of a sentence, calculating text semantic similarity, etc. Represents an enhanced positive sample, and the semantically unchanged rewritten positive sample with the lowest similarity to the sample coarse-grained annotation is used as the enhanced positive sample; Represents enhanced rewritten negative samples, including: enhanced static content perturbation rewritten negative samples and enhance dynamic content perturbation to rewrite negative samples Rewrite the negative sample by perturbing the static content that has the highest similarity to the sample’s coarse-grained annotation Rewrite negative samples as enhanced static content perturbations Rewrite the negative sample by perturbing the dynamic content that is most similar to the sample’s coarse-grained annotation Rewriting negative samples as enhanced dynamic content perturbations

[0141] Step D2: construct a loss value based on the training data set; the loss value is used to characterize the matching loss between the video clip and the text annotation.

[0142] In an embodiment of the present application, a training data set is used to fine-tune the evaluation model, and a loss value is constructed using the training data; wherein the constructed loss value includes matching loss and contrast loss.

[0143] Optionally, the loss value includes a comparison loss value and a matching loss value, and the loss value is constructed according to the training data set, including sub-steps D21 to D23:

[0144] Step D21: annotate the enhanced static content perturbation rewriting negative samples and enhanced dynamic content perturbation rewriting negative samples in the training data set as negative samples, and annotate the coarse-grained samples or the enhanced positive samples as positive samples.

[0145] Step D22: construct a contrast loss value based on the negative sample annotation and the positive sample annotation; the contrast loss value includes: a contrast loss first loss value and a contrast loss second loss value; the contrast loss first loss value is used to characterize the matching between the sample video clip and the negative sample annotation, and the contrast loss second loss value is used to characterize the matching between the positive sample annotation and the sample video clip.

[0146] Step D23: Construct a matching loss value based on the negative sample annotation and the positive sample annotation, and the matching loss value includes: a first matching loss value, a second matching loss value and a third matching loss value; the first matching loss value is used to characterize the matching between the sample video clip and the matched positive sample annotation, the second matching loss value is used to characterize the matching between the sample video clip and the unmatched negative sample annotation, and the third matching loss value is used to characterize the matching between the positive sample annotation and the unmatched sample video clip.

[0147] In the embodiment of the present application, the positive sample is annotated as "sample coarse-grained annotation" or "enhanced positive sample". Using "enhanced positive sample" as the positive sample annotation can alleviate the potential harmful deviation that may exist in the text generated by the large language model. Therefore, the contrast loss and matching loss when the positive sample is annotated as "sample coarse-grained annotation" and the contrast loss and matching loss when the positive sample is annotated as "enhanced positive sample" are constructed respectively. Finally, the contrast loss and matching loss corresponding to the two positive sample annotations obtain the loss value.

[0148] For example, the contrast loss l when the positive sample is labeled as “sample coarse-grained labeling” c and matching loss l m Respectively expressed as:

[0149]

[0150] Contrastive loss when positive samples are labeled as “enhanced positive samples” and matching loss Respectively expressed as:

[0151]

[0152] Among them, c is a classifier, N(v i ) and N(q i ) contains v i / q i The corresponding cross-modal negative samples, N(v i ) includes: Enhanced static content perturbation rewriting negative samples Enhanced dynamic content perturbation rewriting negative samples Trivial negative sample qj (j≠i), q j Refers to the sample annotations that do not match the sample video clips; N(q i ) contains: trivial negative samples v j (j≠i), v j Refers to sample video clips that do not match the sample annotations.

[0153] Finally, the loss value l is expressed as:

[0154]

[0155] Among them, λ c represents the contrast loss weight coefficient, λ m Represents the matching loss weight coefficient.

[0156] Step D3: Optimizing the model parameters of the evaluation model according to the loss value, and obtaining the evaluation model after satisfying the optimization result conditions.

[0157] In the embodiment of the present application, the evaluation model after loss value optimization can judge the matching degree between the target video segment and the fine-grained text annotation, thereby obtaining the confidence score of each fine-grained text annotation. In this way, effective control of hallucination annotation is achieved based on the evaluation model.

[0158] The following is a specific example to illustrate the video fine-grained understanding and annotation method based on a multimodal large model provided in an embodiment of the present application.

[0159] Reference Figure 4 As shown, Figure 4 It is a flowchart of another method for fine-grained video understanding and annotation based on a multimodal large model provided in an embodiment of the present application. The method for fine-grained video understanding and annotation based on a multimodal large model includes steps S410 to S470:

[0160] Step S410: obtaining a target video segment and a coarse-grained text annotation corresponding to the target video segment.

[0161] Step S420: Acquire multiple key frames of the target video segment.

[0162] Specifically, according to the content change of the target video segment, the target video segment is divided into multiple video sub-segments; the middle frame of each video sub-segment is used as the key frame of the video sub-segment; and according to the key frames of the multiple video sub-segments, multiple key frames of the target video segment are obtained.

[0163] Step S430: multiple key frames of the target video clip and the coarse-grained text annotations are input into the image multimodal large model to generate static information to obtain static fine-grained description information, and multiple candidate static fine-grained text annotations are obtained based on the static fine-grained description information.

[0164] The static fine-grained description information includes foreground detail information, background detail information and fine-grained description information. The foreground detail information, the background detail information, the fine-grained description information and the coarse-grained text annotation are then input into a large language model for static attribute information extraction and annotation generation, thereby obtaining a plurality of candidate static fine-grained text annotations.

[0165] Step S340: input the coarse-grained text annotation into a large language model to generate questions, and obtain target questions for dynamic content understanding.

[0166] Step S350: Input the target question for dynamic content understanding and the target video clip into the video multimodal large model for dynamic information generation, obtain the answer to the target question and dynamic fine-grained description information, and obtain multiple candidate dynamic fine-grained text annotations based on the target question answer and the dynamic fine-grained description information.

[0167] Step S360: Use the evaluation model to respectively evaluate the target video clip and the matching degree between the multiple candidate static fine-grained text annotations and the multiple candidate dynamic fine-grained text annotations to obtain a confidence score corresponding to each candidate static fine-grained text annotation and a confidence score corresponding to each candidate dynamic fine-grained text annotation.

[0168] Step S370: The multiple candidate static fine-grained text annotations and the confidence score corresponding to each candidate static fine-grained text annotation, as well as the multiple candidate dynamic fine-grained text annotations and the confidence score corresponding to each candidate dynamic fine-grained text annotation, are used as the fine-grained annotation information of the target video segment.

[0169] Optionally, according to the confidence score corresponding to each candidate static fine-grained text annotation, a static fine-grained text annotation is screened out from the multiple candidate static fine-grained text annotations; according to the confidence score corresponding to each candidate dynamic fine-grained text annotation, a dynamic fine-grained text annotation is screened out from the multiple candidate dynamic fine-grained text annotations; thereby, the confidence scores corresponding to the static fine-grained text annotation and the static fine-grained text annotation, as well as the confidence scores corresponding to the dynamic fine-grained text annotation and the dynamic fine-grained text annotation, are used as the fine-grained annotation information of the target video segment.

[0170] Through the above implementation process, the static fine-grained information and dynamic fine-grained information contained in the target video clip are annotated, multiple candidate static fine-grained text annotations and multiple candidate dynamic fine-grained text annotations are annotated, and the confidence score of each candidate static fine-grained text annotation and the confidence score of each candidate dynamic fine-grained text annotation are given at the same time, so that the credibility of each fine-grained text annotation can be determined according to the confidence score, and the hallucination annotation can be effectively controlled. In this way, the automatic video fine-grained understanding annotation is realized, which promotes the fine-grained video understanding in the field of fine-grained video understanding; and, according to this method, the construction of a fine-grained video annotation dataset can improve the accuracy of fine-grained video understanding, and greatly reduce the annotation cost while ensuring the annotation accuracy.

[0171] For example, Figure 5 This is an overall architecture diagram of a video fine-grained understanding and annotation method based on a multimodal large model provided in an embodiment of the present application. The embodiment of the present application is based on the target video segment and coarse-grained text annotation, and implements automated video fine-grained understanding and annotation based on a multimodal large model. For the static fine-grained information annotation contained in the target video segment, static information extraction is implemented through multiple key frames in the target video segment, and multiple key frames and coarse-grained text annotations of the target video segment are input into the image multimodal large model for static information generation to obtain static fine-grained description information, thereby obtaining multiple candidate static fine-grained text annotations based on the static fine-grained description information based on the large language model.

[0172] For the annotation of dynamic fine-grained information contained in the target video clip, the dynamic information extraction process is improved through the form of video question-answering, and the large language model is used to obtain the target question for dynamic content understanding. The target question for dynamic content understanding and the target video clip are input into the video multimodal large model for dynamic information generation to obtain the answer to the target question and the dynamic fine-grained description information. Based on the large language model, multiple candidate dynamic fine-grained text annotations are obtained according to the answer to the target question and the dynamic fine-grained description information, thereby realizing the annotation of dynamic fine-grained information in the target video clip.

[0173] In addition, the existence of hallucination phenomenon in the multimodal large model constructs an evaluation model; thereby, the evaluation model evaluates the matching degree between the target video segment and the multiple candidate static fine-grained text annotations and the multiple candidate dynamic fine-grained text annotations respectively, and obtains the confidence score corresponding to each candidate static fine-grained text annotation and the confidence score corresponding to each candidate dynamic fine-grained text annotation; and uses the confidence scores corresponding to the multiple candidate static fine-grained text annotations and each candidate static fine-grained text annotation, as well as the confidence scores corresponding to the multiple candidate dynamic fine-grained text annotations and each candidate dynamic fine-grained text annotation as the fine-grained annotation information of the target video segment.

[0174] In this way, based on the method provided in the embodiment of the present application, a fine-grained video annotation dataset can be quickly constructed, while greatly reducing the annotation cost while ensuring the annotation accuracy.

[0175] The embodiment of the present application also provides a video fine-grained understanding and annotation device based on a multimodal large model, referring to Figure 6 As shown, Figure 6 : is a structural schematic diagram of a video fine-grained understanding and annotation device based on a multimodal large model provided in an embodiment of the present application, the device comprising:

[0176] An acquisition module 610 is used to acquire a target video segment and a coarse-grained text annotation corresponding to the target video segment;

[0177] The static information module 620 is used to input the multiple key frames of the target video segment and the coarse-grained text annotations into the image multimodal large model to generate static information, obtain static fine-grained description information, and obtain multiple candidate static fine-grained text annotations according to the static fine-grained description information;

[0178] The dynamic information module 630 is used to input the target question for dynamic content understanding and the target video clip into the video multimodal large model to generate dynamic information, obtain the answer to the target question and dynamic fine-grained description information, and obtain multiple candidate dynamic fine-grained text annotations based on the answer to the target question and the dynamic fine-grained description information;

[0179] An evaluation module 640 is used to evaluate the matching degree between the target video segment and the multiple candidate static fine-grained text annotations and the multiple candidate dynamic fine-grained text annotations respectively by using the evaluation model, and obtain a confidence score corresponding to each candidate static fine-grained text annotation and a confidence score corresponding to each candidate dynamic fine-grained text annotation;

[0180] The annotation module 650 is used to use the multiple candidate static fine-grained text annotations and the confidence score corresponding to each candidate static fine-grained text annotation, as well as the multiple candidate dynamic fine-grained text annotations and the confidence score corresponding to each candidate dynamic fine-grained text annotation as the fine-grained annotation information of the target video segment.

[0181] In an optional embodiment, the device further includes:

[0182] A first screening module, configured to screen out a static fine-grained text annotation from the plurality of candidate static fine-grained text annotations according to a confidence score corresponding to each candidate static fine-grained text annotation;

[0183] A second screening module, configured to screen out a dynamic fine-grained text annotation from the plurality of candidate dynamic fine-grained text annotations according to a confidence score corresponding to each candidate dynamic fine-grained text annotation;

[0184] The annotation module is further used to use the static fine-grained text annotation and the confidence score corresponding to the static fine-grained text annotation, as well as the dynamic fine-grained text annotation and the confidence score corresponding to the dynamic fine-grained text annotation, as the fine-grained annotation information of the target video segment.

[0185] In an optional embodiment, the device further includes a construction module for constructing the evaluation model, and the construction module includes:

[0186] A data set construction module is used to construct a training data set according to the sample coarse-grained annotations corresponding to the sample video clips, wherein each training data in the training data set includes: a sample video clip, a coarse-grained annotation of the sample video clip, and an enhanced sample;

[0187] A loss construction module, used to construct a loss value according to the training data set; the loss value is used to represent the matching loss between the video clip and the text annotation;

[0188] The parameter updating module is used to optimize the model parameters of the evaluation model according to the loss value, and obtain the evaluation model after the optimization result conditions are met.

[0189] In an optional embodiment, the data set construction module includes:

[0190] A sample rewriting module, used to generate rewritten samples according to the coarse-grained annotation of the samples; the rewritten samples include: semantically unchanged rewritten positive samples, static content perturbation rewritten negative samples, and dynamic content perturbation rewritten negative samples;

[0191] A sample selection module, configured to select an enhanced sample from the rewritten sample according to the similarity between the rewritten sample and the sample coarse-grained annotation, wherein the enhanced sample includes: an enhanced positive sample, an enhanced static content perturbation rewritten negative sample, and an enhanced dynamic content perturbation rewritten negative sample;

[0192] The training data module is used to use the sample video clip, the sample coarse-grained annotation, and the enhanced sample as a training data.

[0193] In an optional embodiment, the loss value includes a comparison loss value and a matching loss value, and the loss construction module includes:

[0194] A positive and negative sample module, used to annotate the enhanced static content perturbation rewriting negative samples and enhanced dynamic content perturbation rewriting negative samples in the training data set as negative samples, and annotate the coarse-grained samples or enhanced positive samples as positive samples;

[0195] A contrast loss construction module, used to construct a contrast loss value according to the negative sample annotation and the positive sample annotation; the contrast loss value includes: a contrast loss first item loss value and a contrast loss second item loss value; the contrast loss first item loss value is used to characterize the matching between the sample video segment and the negative sample annotation, and the contrast loss second item loss value is used to characterize the matching between the positive sample annotation and the sample video segment;

[0196] A matching loss construction module is used to construct a matching loss value according to the negative sample annotation and the positive sample annotation, and the matching loss value includes: a first matching loss value, a second matching loss value and a third matching loss value; the first matching loss value is used to characterize the matching between the sample video clip and the matched positive sample annotation, the second matching loss value is used to characterize the matching between the sample video clip and the unmatched negative sample annotation, and the third matching loss value is used to characterize the matching between the positive sample annotation and the unmatched sample video clip.

[0197] In an optional embodiment, the device also includes a key frame acquisition module, which is further used to divide the target video segment into multiple video sub-segments according to content changes of the target video segment; use the intermediate frame of each video sub-segment as the key frame of the video sub-segment; and obtain multiple key frames of the target video segment based on the key frames of the multiple video sub-segments.

[0198] In an optional embodiment, the static fine-grained description information includes: foreground detail information, background detail information and fine-grained description information; the first annotation module is also used to input the foreground detail information, the background detail information, the fine-grained description information and the coarse-grained text annotation into a large language model to extract static attribute information and generate annotations to obtain multiple candidate static fine-grained text annotations.

[0199] In an optional embodiment, the device further comprises a target question acquisition module, the target question acquisition module being used to input the coarse-grained text annotation into a large language model for question generation to obtain a target question for dynamic content understanding;

[0200] The second annotation module is also used to input the answer to the target question, the dynamic fine-grained description information and the coarse-grained text annotation into a large language model to extract dynamic attribute information and generate annotations to obtain multiple candidate dynamic fine-grained text annotations.

[0201] The present application also provides an electronic device, referring to Figure 7 , Figure 7 Schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 7 As shown, the electronic device 700 includes: a memory 710 and a processor 720. The memory 710 and the processor 720 are connected via a bus communication. A computer program is stored in the memory 710. The computer program can be run on the processor 720, thereby implementing the steps of the video fine-grained understanding and annotation method based on a multimodal large model described in an embodiment of the present application.

[0202] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the video fine-grained understanding and annotation method based on a multimodal large model described in the embodiment of the present application are implemented.

[0203] The embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the video fine-grained understanding and annotation method based on a multimodal large model described in the embodiment of the present application.

[0204] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0205] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0206] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0207] The above is a detailed introduction to a video fine-grained understanding and annotation method and device based on a multimodal large model provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A video fine-grained understanding and annotation method based on a multimodal large model, characterized in that: The method comprises: Obtaining a target video segment and a coarse-grained text annotation corresponding to the target video segment; Inputting multiple key frames of the target video clip and the coarse-grained text annotations into the image multimodal large model to generate static information to obtain static fine-grained description information, and obtaining multiple candidate static fine-grained text annotations based on the static fine-grained description information; Inputting the target question for dynamic content understanding and the target video clip into the video multimodal large model for dynamic information generation, obtaining the answer to the target question and dynamic fine-grained description information, and obtaining multiple candidate dynamic fine-grained text annotations based on the answer to the target question and the dynamic fine-grained description information; Using the evaluation model, respectively evaluate the matching degree between the target video segment and the multiple candidate static fine-grained text annotations and the multiple candidate dynamic fine-grained text annotations, and obtain a confidence score corresponding to each candidate static fine-grained text annotation and a confidence score corresponding to each candidate dynamic fine-grained text annotation; The multiple candidate static fine-grained text annotations and the confidence score corresponding to each candidate static fine-grained text annotation, as well as the multiple candidate dynamic fine-grained text annotations and the confidence score corresponding to each candidate dynamic fine-grained text annotation are used as the fine-grained annotation information of the target video segment.

2. The method according to claim 1, characterized in that The method further comprises: Filtering a static fine-grained text annotation from the plurality of candidate static fine-grained text annotations according to a confidence score corresponding to each candidate static fine-grained text annotation; Filtering a dynamic fine-grained text annotation from the multiple candidate dynamic fine-grained text annotations according to the confidence score corresponding to each candidate dynamic fine-grained text annotation; The multiple candidate static fine-grained text annotations and the confidence score corresponding to each candidate static fine-grained text annotation, and the multiple candidate dynamic fine-grained text annotations and the confidence score corresponding to each candidate dynamic fine-grained text annotation are used as the fine-grained annotation information of the target video segment, including: The static fine-grained text annotation and the confidence score corresponding to the static fine-grained text annotation, as well as the dynamic fine-grained text annotation and the confidence score corresponding to the dynamic fine-grained text annotation, are used as the fine-grained annotation information of the target video segment.

3. The method according to claim 1, characterized in that The evaluation model is constructed through the following steps: Constructing a training data set according to the sample coarse-grained annotations corresponding to the sample video clips, wherein each training data in the training data set includes: a sample video clip, a coarse-grained annotation of the sample video clip, and an enhanced sample; Constructing a loss value based on the training data set; the loss value is used to represent the matching loss between the video clip and the text annotation; The model parameters of the evaluation model are optimized according to the loss value, and the evaluation model is obtained after the optimization result conditions are met.

4. The method according to claim 3, characterized in that According to the coarse-grained annotation of samples corresponding to the sample video clips, a training dataset is constructed, including: Generate rewritten samples according to the coarse-grained annotation of the samples; the rewritten samples include: semantically unchanged rewritten positive samples, static content perturbation rewritten negative samples, and dynamic content perturbation rewritten negative samples; According to the similarity between the rewritten sample and the sample coarse-grained annotation, an enhanced sample is selected from the rewritten sample, wherein the enhanced sample includes: an enhanced positive sample, an enhanced static content perturbation rewritten negative sample, and an enhanced dynamic content perturbation rewritten negative sample; The sample video clip, the sample coarse-grained annotation, and the enhanced sample are used as a training data.

5. The method according to claim 3, characterized in that: The loss value includes a comparison loss value and a matching loss value. According to the training data set, the loss value is constructed, including: Annotate the enhanced static content perturbation rewritten negative samples and enhanced dynamic content perturbation rewritten negative samples in the training data set as negative samples, and annotate the coarse-grained samples or enhanced positive samples as positive samples; Constructing a contrast loss value according to the negative sample annotation and the positive sample annotation; the contrast loss value includes: a contrast loss first item loss value and a contrast loss second item loss value; the contrast loss first item loss value is used to characterize the matching between the sample video segment and the negative sample annotation, and the contrast loss second item loss value is used to characterize the matching between the positive sample annotation and the sample video segment; According to the negative sample annotation and the positive sample annotation, a matching loss value is constructed, and the matching loss value includes: a first matching loss value, a second matching loss value and a third matching loss value; the first matching loss value is used to characterize the matching situation between the sample video clip and the matched positive sample annotation, the second matching loss value is used to characterize the matching situation between the sample video clip and the unmatched negative sample annotation, and the third matching loss value is used to characterize the matching situation between the positive sample annotation and the unmatched sample video clip.

6. The method according to claim 1, characterized in that Before inputting the multiple key frames of the target video segment and the coarse-grained text annotations into the image multimodal large model for static information generation, the method further includes: According to the content change of the target video segment, the target video segment is divided into a plurality of video sub-segments; Using the middle frame of each video sub-segment as the key frame of the video sub-segment; A plurality of key frames of the target video segment are obtained according to the key frames of the plurality of video sub-segments.

7. The method according to claim 1, characterized in that The static fine-grained description information includes: foreground detail information, background detail information and fine-grained description information; According to the static fine-grained description information, a plurality of candidate static fine-grained text annotations are obtained, including: The foreground detail information, the background detail information, the fine-grained description information and the coarse-grained text annotation are input into a large language model to extract static attribute information and generate annotations, so as to obtain a plurality of candidate static fine-grained text annotations.

8. The method according to claim 1, characterized in that: Before inputting the target problem for dynamic content understanding and the target video clip into the video multimodal large model for dynamic information generation, the method further includes: Inputting the coarse-grained text annotation into a large language model to generate questions, thereby obtaining target questions for dynamic content understanding; According to the answer to the target question and the dynamic fine-grained description information, a plurality of candidate dynamic fine-grained text annotations are obtained, including: The answer to the target question, the dynamic fine-grained description information and the coarse-grained text annotation are input into a large language model to extract dynamic attribute information and generate annotations, so as to obtain a plurality of candidate dynamic fine-grained text annotations.

9. A video fine-grained understanding and annotation device based on a multimodal large model, characterized in that: The device comprises: An acquisition module, used to acquire a target video segment and a coarse-grained text annotation corresponding to the target video segment; A static information module, used for inputting a plurality of key frames of the target video segment and the coarse-grained text annotations into the image multimodal large model to generate static information, obtain static fine-grained description information, and obtain a plurality of candidate static fine-grained text annotations according to the static fine-grained description information; A dynamic information module is used to input the target question for dynamic content understanding and the target video clip into the video multimodal large model to generate dynamic information, obtain the answer to the target question and dynamic fine-grained description information, and obtain multiple candidate dynamic fine-grained text annotations based on the answer to the target question and the dynamic fine-grained description information; An evaluation module, used to evaluate the matching degree between the target video segment and the multiple candidate static fine-grained text annotations and the multiple candidate dynamic fine-grained text annotations respectively by using the evaluation model, and obtain a confidence score corresponding to each candidate static fine-grained text annotation and a confidence score corresponding to each candidate dynamic fine-grained text annotation; The annotation module is used to use the multiple candidate static fine-grained text annotations and the confidence score corresponding to each candidate static fine-grained text annotation, as well as the multiple candidate dynamic fine-grained text annotations and the confidence score corresponding to each candidate dynamic fine-grained text annotation as the fine-grained annotation information of the target video segment.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the video fine-grained understanding and annotation method based on a multimodal large model are implemented as described in any one of claims 1-8.

Images