Video fine-grained understanding and annotation method and device based on multimodal large model

Abstract

The present application provides a video fine-grained understanding and annotation method and device based on a multimodal large model, which relates to the field of video text annotation and aims to realize automated video fine-grained understanding and annotation. The method realizes fine-grained static information extraction through multiple key frames of a target video clip, and obtains multiple candidate static fine-grained text annotations based on the static fine-grained description information; improves the dynamic information extraction process through the form of video question-answering, and obtains multiple candidate dynamic fine-grained text annotations based on the target question answer and the dynamic fine-grained description information; uses an evaluation model to obtain the confidence score corresponding to each candidate static fine-grained text annotation and the confidence score corresponding to each candidate dynamic fine-grained text annotation; and finally uses the multiple candidate static fine-grained text annotations and their corresponding confidence scores, and the candidate dynamic fine-grained text annotations and their corresponding confidence scores as the fine-grained annotation information of the target video clip.

Description

Technical Field

[0001] The present application relates to the technical field of video text annotation, and in particular to a method and device for fine-grained video understanding and annotation based on a multimodal large model. Background Art

[0002] Due to the lack of high-quality fine-grained video understanding datasets, existing video library clip retrieval methods are basically limited to coarse-grained understanding of videos and cannot cope well with fine-grained query scenarios.

[0003] Existing coarse-grained ground-truth (GT) annotation methods fail to accurately identify the differences between similar clips, preventing algorithms from effectively learning the fine-grained features within videos and understanding the differences between them. Labeling fine-grained information also requires high labor costs and specialized expertise, hindering scalability.

[0004] Therefore, how to achieve automated fine-grained video understanding annotation to promote fine-grained video understanding in the field of fine-grained video understanding is a technical problem that needs to be solved urgently. Summary of the Invention

[0005] In view of the above problems, the embodiments of the present application provide a method and device for fine-grained video understanding and annotation based on a multimodal large model to overcome the above problems or at least partially solve the above problems.

[0006] In a first aspect of an embodiment of the present application, a method for fine-grained video understanding and annotation based on a multimodal large model is disclosed, the method comprising:

[0007] Obtaining a target video segment and a coarse-grained text annotation corresponding to the target video segment;

[0008] Inputting multiple key frames of the target video clip and the coarse-grained text annotations into the image multimodal large model to generate static information to obtain static fine-grained description information, and obtaining multiple candidate static fine-grained text annotations based on the static fine-grained description information;

[0009] The target question for dynamic content understanding and the target video clip are input into the video multimodal large model for dynamic information generation, thereby obtaining the answer to the target question and dynamic fine-grained description information. Based on the answer to the target question and the dynamic fine-grained description information, multiple candidate dynamic fine-grained text annotations are obtained.

[0010] Using the evaluation model, respectively evaluate the matching degree between the target video clip and the multiple candidate static fine-grained text annotations and the multiple candidate dynamic fine-grained text annotations, and obtain a confidence score corresponding to each candidate static fine-grained text annotation and a confidence score corresponding to each candidate dynamic fine-grained text annotation;

[0011] The multiple candidate static fine-grained text annotations and the confidence score corresponding to each candidate static fine-grained text annotation, as well as the multiple candidate dynamic fine-grained text annotations and the confidence score corresponding to each candidate dynamic fine-grained text annotation, are used as the fine-grained annotation information of the target video segment.

[0012] Optionally, the method further includes:

[0013] Filtering a static fine-grained text annotation from the plurality of candidate static fine-grained text annotations according to the confidence score corresponding to each candidate static fine-grained text annotation;

[0014] Filtering a dynamic fine-grained text annotation from the multiple candidate dynamic fine-grained text annotations according to the confidence score corresponding to each candidate dynamic fine-grained text annotation;

[0015] The method further comprises: using the plurality of candidate static fine-grained text annotations and the confidence score corresponding to each candidate static fine-grained text annotation, and the plurality of candidate dynamic fine-grained text annotations and the confidence score corresponding to each candidate dynamic fine-grained text annotation as the fine-grained annotation information of the target video segment, including:

[0016] The static fine-grained text annotation and the confidence score corresponding to the static fine-grained text annotation, as well as the dynamic fine-grained text annotation and the confidence score corresponding to the dynamic fine-grained text annotation, are used as the fine-grained annotation information of the target video segment.

[0017] Optionally, the evaluation model is constructed by the following steps:

[0018] Constructing a training data set based on the sample coarse-grained annotations corresponding to the sample video clips, wherein each training data in the training data set includes: a sample video clip, the coarse-grained annotations of the sample video clip, and an enhanced sample;

[0019] Constructing a loss value based on the training data set; the loss value is used to represent the matching loss between the video clip and the text annotation;

[0020] The model parameters of the evaluation model are optimized according to the loss value, and the evaluation model is obtained after the optimization result conditions are met.

[0021] Optionally, a training dataset is constructed based on the coarse-grained annotations of samples corresponding to the sample video clips, including:

[0022] Generate rewritten samples based on the coarse-grained annotation of the samples; the rewritten samples include: semantically unchanged rewritten positive samples, static content perturbation rewritten negative samples, and dynamic content perturbation rewritten negative samples;

[0023] Selecting enhanced samples from the rewritten samples according to the similarity between the rewritten samples and the coarse-grained annotations of the samples, wherein the enhanced samples include: enhanced positive samples, enhanced negative samples rewritten with static content perturbation, and enhanced negative samples rewritten with dynamic content perturbation;

[0024] The sample video clip, the sample coarse-grained annotation, and the enhanced sample are used as a training data.

[0025] Optionally, the loss value includes a comparison loss value and a matching loss value, and constructing the loss value according to the training data set includes:

[0026] Label the enhanced static content perturbation rewriting negative samples and the enhanced dynamic content perturbation rewriting negative samples in the training data set as negative samples, and label the coarse-grained labeled samples or enhanced positive samples as positive samples;

[0027] Constructing a contrast loss value based on the negative sample annotation and the positive sample annotation; the contrast loss value includes: a first contrast loss value and a second contrast loss value; the first contrast loss value is used to characterize the matching between the sample video clip and the negative sample annotation, and the second contrast loss value is used to characterize the matching between the positive sample annotation and the sample video clip;

[0028] According to the negative sample annotation and the positive sample annotation, a matching loss value is constructed, and the matching loss value includes: a first matching loss value, a second matching loss value and a third matching loss value; the first matching loss value is used to characterize the matching between the sample video clip and the matched positive sample annotation, the second matching loss value is used to characterize the matching between the sample video clip and the unmatched negative sample annotation, and the third matching loss value is used to characterize the matching between the positive sample annotation and the unmatched sample video clip.

[0029] Optionally, before inputting the multiple key frames of the target video segment and the coarse-grained text annotations into the image multimodal large model for generating static information, the method further includes:

[0030] dividing the target video segment into a plurality of video sub-segments according to content changes of the target video segment;

[0031] The middle frame of each video sub-segment is used as the key frame of the video sub-segment;

[0032] A plurality of key frames of the target video segment are obtained according to the key frames of the plurality of video sub-segments.

[0033] Optionally, the static description fine-grained information includes: foreground detail information, background detail information and fine-grained description information;

[0034] According to the static fine-grained description information, a plurality of candidate static fine-grained text annotations are obtained, including:

[0035] The foreground detail information, the background detail information, the fine-grained description information and the coarse-grained text annotation are input into a large language model to extract static attribute information and generate annotations, thereby obtaining a plurality of candidate static fine-grained text annotations.

[0036] Optionally, before inputting the target question for dynamic content understanding and the target video clip into the video multimodal large model for dynamic information generation, the method further includes:

[0037] Inputting the coarse-grained text annotations into a large language model to generate questions, thereby obtaining target questions for dynamic content understanding;

[0038] According to the answer to the target question and the dynamic fine-grained description information, multiple candidate dynamic fine-grained text annotations are obtained, including:

[0039] The answer to the target question, the dynamic fine-grained description information and the coarse-grained text annotation are input into a large language model to extract dynamic attribute information and generate annotations, thereby obtaining a plurality of candidate dynamic fine-grained text annotations.

[0040] A second aspect of the embodiments of the present application discloses a video fine-grained understanding and annotation device based on a multimodal large model, the device comprising:

[0041] An acquisition module, configured to acquire a target video segment and a coarse-grained text annotation corresponding to the target video segment;

[0042] a static information module, configured to input multiple key frames of the target video clip and the coarse-grained text annotations into an image multimodal large model to generate static information, thereby obtaining static fine-grained description information, and to obtain multiple candidate static fine-grained text annotations based on the static fine-grained description information;

[0043] A dynamic information module is used to input the target question for dynamic content understanding and the target video clip into the video multimodal large model to generate dynamic information, obtain the answer to the target question and dynamic fine-grained description information, and obtain multiple candidate dynamic fine-grained text annotations based on the target question answer and the dynamic fine-grained description information;

[0044] An evaluation module is configured to use an evaluation model to evaluate the matching degree between the target video segment and the multiple candidate static fine-grained text annotations and the multiple candidate dynamic fine-grained text annotations, respectively, to obtain a confidence score corresponding to each candidate static fine-grained text annotation and a confidence score corresponding to each candidate dynamic fine-grained text annotation;

[0045] The annotation module is used to use the multiple candidate static fine-grained text annotations and the confidence score corresponding to each candidate static fine-grained text annotation, as well as the multiple candidate dynamic fine-grained text annotations and the confidence score corresponding to each candidate dynamic fine-grained text annotation, as the fine-grained annotation information of the target video segment.

[0046] The third aspect of the embodiments of the present application discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, the steps of the video fine-grained understanding and annotation method based on a multimodal large model described in the first aspect of the embodiments of the present application are implemented.

[0047] The fourth aspect of the embodiments of the present application discloses a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the video fine-grained understanding and annotation method based on a multimodal large model described in the first aspect of the embodiments of the present application are implemented.

[0048] The embodiments of the present application include the following advantages:

[0049] In an embodiment of the present application, based on the target video clip and coarse-grained text annotation, an automated video fine-grained understanding annotation is implemented based on a multimodal large model. For the annotation of static fine-grained information contained in the target video clip, static information extraction is implemented through multiple key frames in the target video clip, and the multiple key frames of the target video clip and the coarse-grained text annotation are input into the image multimodal large model for static information generation to obtain static fine-grained description information, thereby obtaining multiple candidate static fine-grained text annotations based on the static fine-grained description information; based on the static fine-grained description information obtained from multiple key frames, rather than obtaining coarse-grained information that describes the target video clip as a whole, the static fine-grained information in the target video clip is annotated based on the static fine-grained description information. For the annotation of dynamic fine-grained information contained in the target video clip, the dynamic information extraction process is improved through the form of video question answering. The target question for dynamic content understanding and the target video clip are input into the video multimodal large model for dynamic information generation, and the answer to the target question and dynamic fine-grained description information are obtained. Based on the answer to the target question and the dynamic fine-grained description information, multiple candidate dynamic fine-grained text annotations are obtained to realize the annotation of dynamic fine-grained information in the target video clip.

[0050] In addition, the evaluation model is used to evaluate the matching degree between the target video clip and multiple candidate static fine-grained text annotations and multiple candidate dynamic fine-grained text annotations, and the confidence score corresponding to each candidate static fine-grained text annotation and the confidence score corresponding to each candidate dynamic fine-grained text annotation are obtained; according to the confidence score, the credibility of each fine-grained text annotation can be determined, thereby achieving effective control of hallucination annotations.

[0051] In this way, by annotating the static and dynamic fine-grained information contained in the target video clip, multiple candidate static fine-grained text annotations and the corresponding confidence scores for each candidate static fine-grained text annotation, as well as multiple candidate dynamic fine-grained text annotations and the corresponding confidence scores for each candidate dynamic fine-grained text annotation, are used as the fine-grained annotation information of the target video clip. This achieves automated video fine-grained understanding annotation, promoting the field of fine-grained video understanding. This automated video fine-grained understanding annotation solution can quickly construct a fine-grained video annotation dataset, significantly reducing annotation costs while ensuring annotation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0053] Figure 1 This is a comparison chart of retrieval results of coarse-grained text annotation and fine-grained text annotation provided in an embodiment of the present application;

[0054] Figure 2 This is a flowchart of the steps of a video fine-grained understanding and annotation method based on a multimodal large model provided in an embodiment of the present application;

[0055] Figure 3 This is a schematic diagram of the processing results of fine-grained text annotation by an evaluation model provided in an embodiment of the present application;

[0056] Figure 4 This is a flowchart of another method for fine-grained video understanding and annotation based on a multimodal large model provided in an embodiment of the present application;

[0057] Figure 5 This is an overall architecture diagram of a video fine-grained understanding and annotation method based on a multimodal large model provided in an embodiment of the present application;

[0058] Figure 6This is a structural diagram of a video fine-grained understanding and annotation device based on a multimodal large model provided in an embodiment of the present application;

[0059] Figure 7 This is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] To make the above-mentioned purposes, features, and advantages of this application more clearly understood, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of this application.

[0061] There are two main types of existing video-text annotation solutions: manual annotation and automated annotation. For manual annotation, especially for annotating fine-grained content in videos, video-text annotation requires a lot of work and domain knowledge, making these datasets difficult to scale. For automated annotation solutions, some solutions obtain annotated text through web crawling, speech transcription, etc., but the cross-modal alignment of these annotations is relatively noisy; with the development of multimodal basic models and large language model technologies, automatic annotation of large-scale video-text datasets has become feasible, but the accuracy of existing automatic annotation solutions based on multimodal basic models and large language model technologies is difficult to guarantee, and the annotation is not fine-grained enough, especially for dynamic details in videos (for example, actions and interactive relationships, etc.), or its fine-grained annotations are mostly derived from collected video subtitles or other text tags, and its annotation content cannot well reflect the content of the visual modality.

[0062] Therefore, existing video-text annotation schemes have the following problems: (1) The manual annotation cost of fine-grained annotation is high and difficult to scale; (2) Existing automatic annotation technologies cannot generate accurate fine-grained annotation information, especially lack dynamic content; and there is a lack of effective control over annotation hallucinations.

[0063] In order to overcome the limitations of related technologies, an embodiment of the present application provides a video fine-grained understanding and annotation method based on a multimodal large model, which realizes automated video fine-grained understanding and annotation based on a multimodal large model. For the annotation of static fine-grained information contained in the target video clip, fine-grained static information extraction is realized through multiple key frames in the target video clip to obtain static fine-grained information; for the annotation of dynamic fine-grained information contained in the target video clip, the extraction process of dynamic information is improved through the form of video question-answering to obtain dynamic fine-grained description information. In addition, the confidence score corresponding to each candidate static fine-grained text annotation and the confidence score corresponding to each candidate dynamic fine-grained text annotation are obtained using the evaluation model.

[0064] In this way, by annotating the static fine-grained information and dynamic fine-grained information contained in the target video clip, multiple candidate static fine-grained text annotations and their corresponding confidence scores, candidate dynamic fine-grained text annotations and their corresponding confidence scores are used as the fine-grained annotation information of the target video clip, thereby realizing automated video fine-grained understanding and annotation, and solving the hallucination phenomenon of multimodal large models. Annotations containing inaccurate content can be filtered out through confidence scores.

[0065] The following is a detailed description of the video fine-grained understanding and annotation method based on a multimodal large model provided in an embodiment of the present application, in conjunction with the accompanying drawings.

[0066] In order to better understand the technical solution of the present application, the “coarse-grained text annotation” and “fine-grained text annotation” involved in the embodiments of the present application are first explained.

[0067] Reference Figure 1 As shown, Figure 1 This is a comparison chart of the retrieval results of coarse-grained text annotation and fine-grained text annotation provided in an embodiment of the present application. Among them, coarse-grained text annotation refers to a text description with less information description. When searching the video library based on coarse-grained text annotation, in addition to being able to retrieve the target video segment corresponding to the coarse-grained text annotation, other potential segments that are not annotated can also be retrieved. In other words, it is difficult to accurately retrieve a one-to-one target video segment based on coarse-grained text annotation. Figure 1 As shown in the figure, for a coarse-grained text annotation "a person opens a bag", three video clips can be retrieved in the video library. However, when performing text annotation, only one video clip is annotated, and other potential matches in the video library are ignored. As a result, problems will arise in both model training (i.e., the training process ignores other positive samples) and inference (may only consider some of the videos) based on the video library with coarse-grained text annotations.

[0068] Fine-grained text annotation refers to a more comprehensive text description of information. When searching a video library based on fine-grained text annotation, it is possible to accurately retrieve one-to-one target video clips. That is, fine-grained text annotation can eliminate the one-to-many matching problem existing in coarse-grained text annotation, thus achieving accurate video retrieval. Figure 1 As shown, for a fine-grained text annotation "A person on the sofa opens a colorful bag, revealing a mobile phone," a unique corresponding video clip can be retrieved from the video library. Based on the solution provided by the embodiment of this application, the fine-grained information contained in the target video clip can be annotated, thereby promoting fine-grained video understanding based on the fine-grained annotation information.

[0069] The following sections introduce them in Sections 1.1 and 1.2 respectively.

[0070] 1.1 Methods for automatically generating fine-grained text annotations:

[0071] Reference Figure 2 As shown, Figure 2 This is a flowchart of the steps of a video fine-grained understanding and annotation method based on a multimodal large model provided in an embodiment of the present application. Figure 2 As shown, the video fine-grained understanding and annotation method based on the multimodal large model may include steps S210 to S250:

[0072] Step S210: Obtain a target video segment and a coarse-grained text annotation corresponding to the target video segment.

[0073] The target video segment refers to any video segment in the existing coarse-grained video text annotation database, and the coarse-grained text annotation refers to the text annotation corresponding to the target video segment. The form of coarse-grained text annotation is (V,t s ,t e ,q), respectively represent the target video segment, start time, end time, and query text (the query text is a description text containing less descriptive information).

[0074] In an embodiment of the present application, fine-grained annotation is performed on the basis of an existing coarse-grained video text annotation database; obtaining a target video segment and a coarse-grained text annotation corresponding to the target video segment means: obtaining any video segment as a target video segment from the existing coarse-grained video text annotation database, and obtaining the coarse-grained text annotation corresponding to the target video segment.

[0075] Step S220: Multiple key frames of the target video clip and the coarse-grained text annotations are input into the image multimodal large model to generate static information to obtain static fine-grained description information, and multiple candidate static fine-grained text annotations are obtained based on the static fine-grained description information.

[0076] In the embodiments of the present application, static fine-grained description information refers to a type of fine-grained information, representing the fine-grained description information of video frames in a target video segment. To annotate the static fine-grained information contained in the target video segment, fine-grained static information is extracted based on multiple key frames of the target video segment to obtain the static fine-grained information. A key frame of the target video segment can be a video frame in a sub-segment corresponding to a different scene in the target video segment.

[0077] For multiple keyframes in the target video clip, the keyframes and coarse-grained text annotations are input into the image multimodal large model for static information generation, resulting in static fine-grained description information. This static fine-grained description information includes the static fine-grained description information corresponding to each keyframe. Based on this static fine-grained description information, multiple candidate static fine-grained text annotations are obtained. These candidate static fine-grained text annotations are text annotations that contain static fine-grained description information.

[0078] Based on the static fine-grained description information obtained from multiple key frames, rather than obtaining coarse-grained information that describes the target video segment as a whole, the static fine-grained information in the target video segment is labeled according to the static fine-grained description information.

[0079] Step S230: Input the target question for dynamic content understanding and the target video clip into the video multimodal large model for dynamic information generation, obtain the answer to the target question and dynamic fine-grained description information, and obtain multiple candidate dynamic fine-grained text annotations based on the target question answer and the dynamic fine-grained description information.

[0080] In the embodiment of the present application, dynamic fine-grained description information refers to a type of fine-grained information, and the dynamic fine-grained description information represents the fine-grained description information of the dynamic changes of the scene between the video frames in the target video clip. In order to annotate the dynamic fine-grained information contained in the target video clip, the dynamic information extraction process is improved in the form of video question answering, and the target questions for understanding dynamic content are obtained. The target questions for understanding dynamic content can be a series of questions generated based on the coarse-grained text annotation. For example, for the coarse-grained text annotation "the person puts down the bag between 4 seconds and 9 seconds", then one of the target questions can be "where did the person put the bag?".

[0081] The target question for dynamic content understanding and the target video clip are input into a large multimodal video model for dynamic information generation, resulting in the target question answer and dynamic fine-grained description information. The target question answer refers to the response to the target question. For example, for the target question "Where did the person put the bag?", the corresponding target question answer might be "The person put the bag on the table." Based on the target question answer and dynamic fine-grained description information, multiple candidate dynamic fine-grained text annotations are generated to annotate the dynamic fine-grained information in the target video clip. The candidate dynamic fine-grained text annotations are text annotations containing dynamic fine-grained description information.

[0082] Step S240: Use the evaluation model to evaluate the matching degree between the target video clip and the multiple candidate static fine-grained text annotations and the multiple candidate dynamic fine-grained text annotations, and obtain the confidence score corresponding to each candidate static fine-grained text annotation and the confidence score corresponding to each candidate dynamic fine-grained text annotation.

[0083] In the embodiment of the present application, considering that large multimodal models (e.g., large image multimodal models or large video multimodal models) may experience hallucinations and may easily generate erroneous information, an evaluation model is used to evaluate the confidence of the multiple candidate static fine-grained text annotations obtained in step S220 and the multiple candidate dynamic fine-grained text annotations obtained in step S230.

[0084] The evaluation model can judge the degree of match between the target video clip and the fine-grained text annotations (i.e., candidate static fine-grained text annotations and candidate dynamic fine-grained text annotations), that is, judge the degree of match between the coarse-grained text annotations and the fine-grained text annotations of the target video clip; for fine-grained text annotations with a high degree of match, the evaluation model outputs a higher confidence score (first confidence score or second confidence score), and for fine-grained text annotations with a low degree of match, the evaluation model outputs a lower confidence score, so that fine-grained text annotations of poor quality (containing incorrect information) can be distinguished based on the confidence score.

[0085] For example, Figure 3 The figure is a schematic diagram of the processing results of fine-grained text annotations by an evaluation model provided in an embodiment of the present application. For example, the evaluation model evaluates the matching degree of the coarse-grained text annotation "A person opens a bag" with the fine-grained text annotation "A person, wearing a green sweatshirt and glasses, sits on a sofa, opens a colorful bag with both hands, revealing a mobile phone", and obtains a confidence score of 3.82. For another example, the evaluation model evaluates the matching degree of the coarse-grained text annotation "A person opens a bag" with the fine-grained text annotation "A person in the kitchen holds a cherry-patterned bag and pulls the drawstring to open the bag", and obtains a confidence score of 3.23.

[0086] Specifically, the evaluation model is used to evaluate the matching degree between the target video clip and multiple candidate static fine-grained text annotations, and the confidence score corresponding to each candidate static fine-grained text annotation is obtained; the evaluation model is used to evaluate the matching degree between the target video clip and multiple candidate dynamic fine-grained text annotations, and the confidence score corresponding to each candidate dynamic fine-grained text annotation is obtained.

[0087] Step S250: The multiple candidate static fine-grained text annotations and the confidence score corresponding to each candidate static fine-grained text annotation, as well as the multiple candidate dynamic fine-grained text annotations and the confidence score corresponding to each candidate dynamic fine-grained text annotation, are used as the fine-grained annotation information of the target video segment.

[0088] In the embodiment of the present application, the annotation form of the fine-grained annotation information of the target video segment is: They respectively represent the target video segment, start time, end time, multiple candidate static fine-grained text annotations, the confidence scores corresponding to the multiple candidate static fine-grained text annotations, multiple candidate dynamic fine-grained text annotations, and the confidence scores corresponding to the multiple candidate dynamic fine-grained text annotations.

[0089] Through the above implementation process, the static and dynamic fine-grained information contained in the target video clip is annotated. Multiple candidate static fine-grained text annotations and the corresponding confidence scores for each candidate static fine-grained text annotation, as well as multiple candidate dynamic fine-grained text annotations and the corresponding confidence scores for each candidate dynamic fine-grained text annotation, are used as the fine-grained annotation information of the target video clip. This achieves automated video fine-grained understanding annotation, promoting fine-grained video understanding in the field. This automated video fine-grained understanding annotation solution can quickly construct a fine-grained video annotation dataset, significantly reducing annotation costs while ensuring annotation accuracy.

[0090] The present application will be described in detail below in conjunction with the above implementation steps.

[0091] In some specific implementations, before inputting the multiple key frames of the target video segment and the coarse-grained text annotations into the image multimodal large model for static information generation in step S220, the following steps A1 to A3 are further included:

[0092] Step A1: dividing the target video segment into multiple video sub-segments according to content changes of the target video segment;

[0093] Step A2: taking the middle frame of each video sub-segment as the key frame of the video sub-segment;

[0094] Step A3: obtaining multiple key frames of the target video segment according to the multiple key frames of the video sub-segments.

[0095] In an embodiment of the present application, multiple key frames of the target video segment are obtained according to steps A1 to A3. Specifically, according to the content changes of the target video segment, the target video segment is divided into multiple video sub-segments, and the PySceneDetect threshold can be adaptively adjusted to divide the target video segment into multiple video sub-segments (such as L). For example, a threshold is set, and when the RGB value change of the image frame in the target video segment is greater than the threshold, the target video segment is divided. Finally, the middle frame of each video sub-segment is used as the key frame of the video sub-segment, thereby obtaining multiple key frames of the target video segment.

[0096] In some specific implementations, the static fine-grained description information includes foreground detail information, background detail information, and fine-grained description information. The foreground detail information refers to subject information (people or objects) in a keyframe, the background detail information refers to background information in a keyframe, and the fine-grained description information refers to a comprehensive description information that integrates the foreground detail information and the background detail information.

[0097] Specifically, the above step S220 of "inputting the multiple key frames of the target video segment and the coarse-grained text annotations into the image multimodal large model to generate static information to obtain static fine-grained description information" can be expressed as:

[0098]

[0099] Among them, ImageLMM represents the image multimodal large model, f i represents a key frame, q represents a coarse-grained text annotation, Represents foreground detail information, Represents background details, D i Represents fine-grained description information.

[0100] Optionally, the "obtaining multiple candidate static fine-grained text annotations based on the static fine-grained description information" in the above step S220 may specifically include: inputting the foreground detail information, the background detail information, the fine-grained description information and the coarse-grained text annotations into a large language model to extract static attribute information and generate annotations to obtain multiple candidate static fine-grained text annotations.

[0101] For example, multiple candidate static fine-grained text annotations can be represented as:

[0102]

[0103] in, Represents multiple candidate static fine-grained text annotations, with a total of N s candidate static fine-grained text annotations; LLM stands for Large Language Model.

[0104] In some specific implementations, before step S230 of inputting the target question for dynamic content understanding and the target video clip into the video multimodal large model for dynamic information generation, the step further includes: inputting the coarse-grained text annotation into the large language model for question generation to obtain the target question for dynamic content understanding.

[0105] In an embodiment of the present application, in order to annotate the dynamic fine-grained information contained in the target video clip, the dynamic information extraction process is improved through the form of video question answering, prompting the large language model to generate target questions for dynamic content understanding based on coarse-grained text annotations.

[0106] For example, the target problem for dynamic content understanding is expressed as:

[0107]

[0108] in, Represents the target problem for dynamic content understanding, with a total of N qa A target problem.

[0109] Afterwards, the target question for dynamic content understanding and the target video clip are input into the video multimodal large model for dynamic information generation to obtain the answer to the target question and dynamic fine-grained description information.

[0110] For example, the target question answer and dynamic fine-grained description information can be expressed as:

[0111]

[0112] Among them, VideoLMM represents the video multimodal large model, Represents the answer to the target question, each target question answer corresponds to a target question, Represents dynamic fine-grained description information.

[0113] Furthermore, the above-mentioned step S230 of "obtaining multiple candidate dynamic fine-grained text annotations based on the target question answer and the dynamic fine-grained description information" may specifically include: inputting the target question answer, the dynamic fine-grained description information and the coarse-grained text annotation into a large language model to extract dynamic attribute information and generate annotations, so as to obtain multiple candidate dynamic fine-grained text annotations.

[0114] For example, multiple candidate dynamic fine-grained text annotations can be expressed as:

[0115]

[0116] in, Represents multiple candidate dynamic fine-grained text annotations, with a total of N d Candidate dynamic fine-grained text annotations.

[0117] In some specific implementations, the following steps are further included after step S240:

[0118] Step C1: selecting a static fine-grained text annotation from the plurality of candidate static fine-grained text annotations according to the confidence score corresponding to each candidate static fine-grained text annotation;

[0119] Step C2: Filtering a dynamic fine-grained text annotation from the multiple candidate dynamic fine-grained text annotations according to the confidence score corresponding to each candidate dynamic fine-grained text annotation.

[0120] In an embodiment of the present application, the confidence score corresponding to the candidate static fine-grained text annotation can reflect the quality of the candidate static fine-grained text annotation, that is, it can be determined whether the candidate static fine-grained text annotation contains error information based on the confidence score, and thus, based on the confidence score corresponding to each candidate static fine-grained text annotation, static fine-grained text annotations that do not contain error information are screened out from multiple candidate static fine-grained text annotations. Similarly, the confidence score corresponding to the candidate dynamic fine-grained text annotation can reflect the quality of the candidate dynamic fine-grained text annotation, that is, it can be determined whether the candidate dynamic fine-grained text annotation contains error information based on the confidence score, and thus, based on the confidence score corresponding to each candidate dynamic fine-grained text annotation, dynamic fine-grained text annotations that do not contain error information are screened out from multiple candidate dynamic fine-grained text annotations.

[0121] Furthermore, in step S250 above, "using the multiple candidate static fine-grained text annotations and the confidence score corresponding to each candidate static fine-grained text annotation, and the multiple candidate dynamic fine-grained text annotations and the confidence score corresponding to each candidate dynamic fine-grained text annotation as the fine-grained annotation information of the target video segment" may specifically include:

[0122] The static fine-grained text annotation and the confidence score corresponding to the static fine-grained text annotation, as well as the dynamic fine-grained text annotation and the confidence score corresponding to the dynamic fine-grained text annotation, are used as the fine-grained annotation information of the target video segment.

[0123] In the embodiment of the present application, inaccurate annotation data can be automatically filtered through confidence scores. Constructing a fine-grained video annotation dataset based on this method can improve the accuracy of fine-grained video understanding and establish a new benchmark for fine-grained video library segment retrieval.

[0124] 1.2 Evaluation model construction process:

[0125] In a specific embodiment, the evaluation model is constructed by following steps D1 to D3:

[0126] Step D1: constructing a training data set based on the sample coarse-grained annotations corresponding to the sample video clips, where each training data in the training data set includes: a sample video clip, the coarse-grained annotations of the sample video clip, and an enhanced sample.

[0127] Sample video clips are the video clips used to fine-tune the evaluation model. Each sample video clip corresponds to a coarse-grained sample annotation, which is a text description with minimal information. Enhanced samples are sample annotations generated based on the coarse-grained sample annotations. A training dataset is constructed based on the coarse-grained sample annotations corresponding to each sample video clip, which is used to fine-tune the evaluation model.

[0128] Optionally, a training dataset is constructed based on the coarse-grained annotations of samples corresponding to the sample video clips, including sub-steps D11 to D13:

[0129] Step D11: Generate rewritten samples based on the coarse-grained annotation of the samples; the rewritten samples include: semantically unchanged rewritten positive samples, static content perturbation rewritten negative samples, and dynamic content perturbation rewritten negative samples.

[0130] Step D12: Select enhanced samples from the rewritten samples based on the similarity between the rewritten samples and the coarse-grained annotations of the samples, wherein the enhanced samples include: enhanced positive samples, enhanced static content perturbation rewritten negative samples, and enhanced dynamic content perturbation rewritten negative samples.

[0131] Step D13: The sample video clip, the sample coarse-grained annotation, and the enhanced sample are used as a training data.

[0132] In the embodiment of the present application, for each sample coarse-grained annotation, a rewritten sample is generated according to the large language model, that is, a semantically unchanged rewritten positive sample is generated. Static content perturbation rewriting negative samples Dynamic content perturbation rewriting negative samples

[0133] For example, the generation method of generating rewriting samples is expressed as:

[0134]

[0135] Among them, LLM represents the large language model and q represents the coarse-grained annotation of samples.

[0136] After obtaining the rewritten samples, a similarity calculation model (for example, the Sentence BERT model) can be used to calculate the similarity between the rewritten samples and the coarse-grained annotations of the samples, so as to select enhanced samples from the rewritten samples based on the similarity, that is, to select the most appropriate perturbed positive and negative samples.

[0137] For example, selecting the enhanced sample from the rewritten sample can be expressed as:

[0138]

[0139]

[0140] SentenceBERT is a sentence embedding model based on the BERT model, which is mainly used to process sentence-level tasks, such as obtaining a vector representation of a sentence and calculating text semantic similarity. Represents an enhanced positive sample, and the semantically unchanged rewritten positive sample with the lowest similarity to the sample coarse-grained annotation is used as the enhanced positive sample; Represents enhanced rewritten negative samples, including: enhanced static content perturbation rewritten negative samples and enhance dynamic content perturbation to rewrite negative samples Perturb the static content that has the highest similarity to the sample's coarse-grained annotation to rewrite the negative sample Rewrite negative samples as enhanced static content perturbations Rewrite the negative sample by perturbing the dynamic content that has the highest similarity to the sample's coarse-grained annotation Rewriting negative samples as enhanced dynamic content perturbations

[0141] Step D2: constructing a loss value based on the training data set; the loss value is used to represent the matching loss between the video clip and the text annotation.

[0142] In an embodiment of the present application, a training data set is used to fine-tune the evaluation model, and a loss value is constructed using the training data; wherein the constructed loss value includes matching loss and contrast loss.

[0143] Optionally, the loss value includes a comparison loss value and a matching loss value. Constructing the loss value according to the training data set includes sub-steps D21 to D23:

[0144] Step D21: label the enhanced static content perturbation rewriting negative samples and enhanced dynamic content perturbation rewriting negative samples in the training data set as negative samples, and label the coarse-grained samples or enhanced positive samples as positive samples.

[0145] Step D22: Construct a contrast loss value based on the negative sample annotation and the positive sample annotation; the contrast loss value includes: a first contrast loss value and a second contrast loss value; the first contrast loss value is used to characterize the matching between the sample video clip and the negative sample annotation, and the second contrast loss value is used to characterize the matching between the positive sample annotation and the sample video clip.

[0146] Step D23: Construct a matching loss value based on the negative sample annotation and the positive sample annotation, and the matching loss value includes: a first matching loss value, a second matching loss value, and a third matching loss value; the first matching loss value is used to characterize the matching between the sample video clip and the matched positive sample annotation, the second matching loss value is used to characterize the matching between the sample video clip and the unmatched negative sample annotation, and the third matching loss value is used to characterize the matching between the positive sample annotation and the unmatched sample video clip.

[0147] In the embodiments of this application, positive samples are labeled as "coarse-grained sample annotation" or "enhanced positive sample." Using "enhanced positive sample" as the positive sample annotation can alleviate potential harmful biases that may exist in the text generated by large language models. Therefore, the contrast loss and matching loss are constructed for the case where the positive sample is labeled as "coarse-grained sample annotation," and the contrast loss and matching loss are constructed for the case where the positive sample is labeled as "enhanced positive sample." Finally, the contrast loss and matching loss corresponding to the two positive sample annotations are used to obtain the loss value.

[0148] For example, the contrast loss l when the positive sample is labeled as “sample coarse-grained labeling” c and matching loss l m Respectively expressed as:

[0149]

[0150] Contrast loss when the positive sample is labeled as "enhanced positive sample" and matching loss Respectively expressed as:

[0151]

[0152] Among them, c is a classifier, N(v i ) and N(q i ) contains v i / q i The corresponding cross-modal negative samples, N(v i ) includes: Enhanced static content perturbation rewriting negative samples Enhanced dynamic content perturbation rewriting negative samples trivial negative sample qj (j≠i),q j Refers to the sample annotations that do not match the sample video clips; N(q i ) contains: trivial negative samples v j (j≠i), v j Refers to sample video clips that do not match the sample annotations.

[0153] Finally, the loss value l is expressed as:

[0154]

[0155] Among them, λ c represents the contrast loss weight coefficient, λ m Represents the matching loss weight coefficient.

[0156] Step D3: Optimizing the model parameters of the evaluation model according to the loss value, and obtaining the evaluation model after satisfying the optimization result conditions.

[0157] In this embodiment of the present application, the loss-optimized evaluation model can determine the degree of match between the target video clip and the fine-grained text annotation, thereby obtaining a confidence score for each fine-grained text annotation. In this way, effective control of hallucination annotations is achieved based on the evaluation model.

[0158] The following is a specific example to illustrate the video fine-grained understanding and annotation method based on a multimodal large model provided in the embodiment of the present application.

[0159] Reference Figure 4 As shown, Figure 4 This is a flowchart of another method for fine-grained video understanding and annotation based on a multimodal large model provided in an embodiment of the present application. The method for fine-grained video understanding and annotation based on a multimodal large model includes steps S410 to S470:

[0160] Step S410: Obtain a target video segment and a coarse-grained text annotation corresponding to the target video segment.

[0161] Step S420: Acquire multiple key frames of the target video segment.

[0162] Specifically, according to the content changes of the target video segment, the target video segment is divided into multiple video sub-segments; the middle frame of each video sub-segment is used as the key frame of the video sub-segment; and based on the key frames of the multiple video sub-segments, multiple key frames of the target video segment are obtained.

[0163] Step S430: Multiple key frames of the target video clip and the coarse-grained text annotations are input into the image multimodal large model to generate static information to obtain static fine-grained description information, and multiple candidate static fine-grained text annotations are obtained based on the static fine-grained description information.

[0164] The static fine-grained description information includes foreground detail information, background detail information, and fine-grained description information. The foreground detail information, background detail information, fine-grained description information, and coarse-grained text annotations are then input into a large language model for static attribute information extraction and annotation generation, resulting in multiple candidate static fine-grained text annotations.

[0165] Step S340: Input the coarse-grained text annotations into a large language model to generate questions, and obtain target questions for dynamic content understanding.

[0166] Step S350: Input the target question for dynamic content understanding and the target video clip into the video multimodal large model for dynamic information generation, obtain the answer to the target question and dynamic fine-grained description information, and obtain multiple candidate dynamic fine-grained text annotations based on the target question answer and the dynamic fine-grained description information.

[0167] Step S360: Use the evaluation model to evaluate the matching degree between the target video clip and the multiple candidate static fine-grained text annotations and the multiple candidate dynamic fine-grained text annotations, and obtain the confidence score corresponding to each candidate static fine-grained text annotation and the confidence score corresponding to each candidate dynamic fine-grained text annotation.

[0168] Step S370: The multiple candidate static fine-grained text annotations and the confidence score corresponding to each candidate static fine-grained text annotation, as well as the multiple candidate dynamic fine-grained text annotations and the confidence score corresponding to each candidate dynamic fine-grained text annotation, are used as the fine-grained annotation information of the target video segment.

[0169] Optionally, according to the confidence score corresponding to each candidate static fine-grained text annotation, a static fine-grained text annotation is screened out from the multiple candidate static fine-grained text annotations; according to the confidence score corresponding to each candidate dynamic fine-grained text annotation, a dynamic fine-grained text annotation is screened out from the multiple candidate dynamic fine-grained text annotations; thereby the confidence scores corresponding to the static fine-grained text annotation and the static fine-grained text annotation, as well as the confidence scores corresponding to the dynamic fine-grained text annotation and the dynamic fine-grained text annotation, are used as the fine-grained annotation information of the target video segment.

[0170] Through the above implementation process, the static and dynamic fine-grained information contained in the target video clip is annotated, and multiple candidate static fine-grained text annotations and multiple candidate dynamic fine-grained text annotations are annotated. A confidence score is also given for each candidate static fine-grained text annotation, thereby determining the credibility of each fine-grained text annotation based on the confidence score, effectively controlling hallucination annotations. In this way, automated video fine-grained understanding annotation is achieved, promoting fine-grained video understanding in the field. Furthermore, constructing a fine-grained video annotation dataset based on this method can improve the accuracy of fine-grained video understanding, significantly reducing annotation costs while ensuring annotation accuracy.

[0171] For example, Figure 5 This is an overall architecture diagram of a video fine-grained understanding and annotation method based on a multimodal large model provided in an embodiment of the present application. The embodiment of the present application is based on the target video segment and coarse-grained text annotation, and realizes automated video fine-grained understanding and annotation based on a multimodal large model. For the static fine-grained information annotation contained in the target video segment, static information extraction is realized through multiple key frames in the target video segment, and multiple key frames and coarse-grained text annotations of the target video segment are input into the image multimodal large model for static information generation to obtain static fine-grained description information, thereby obtaining multiple candidate static fine-grained text annotations based on the static fine-grained description information based on the large language model.

[0172] For the annotation of dynamic fine-grained information contained in the target video clip, the dynamic information extraction process is improved through the form of video question answering, and the large language model is used to obtain the target question for dynamic content understanding. The target question for dynamic content understanding and the target video clip are input into the video multimodal large model for dynamic information generation, and the target question answer and dynamic fine-grained description information are obtained. Based on the large language model, according to the target question answer and the dynamic fine-grained description information, multiple candidate dynamic fine-grained text annotations are obtained to realize the annotation of dynamic fine-grained information in the target video clip.

[0173] Moreover, the existence of hallucination phenomenon in the multimodal large model constructs an evaluation model; thereby, the evaluation model evaluates the matching degree between the target video segment and the multiple candidate static fine-grained text annotations and the multiple candidate dynamic fine-grained text annotations respectively, and obtains the confidence score corresponding to each candidate static fine-grained text annotation and the confidence score corresponding to each candidate dynamic fine-grained text annotation; and uses the confidence scores corresponding to the multiple candidate static fine-grained text annotations and each candidate static fine-grained text annotation, as well as the confidence scores corresponding to the multiple candidate dynamic fine-grained text annotations and each candidate dynamic fine-grained text annotation as the fine-grained annotation information of the target video segment.

[0174] In this way, based on the method provided in the embodiments of the present application, a fine-grained video annotation dataset can be quickly constructed, while greatly reducing the annotation cost while ensuring the annotation accuracy.

[0175] The embodiment of the present application also provides a video fine-grained understanding and annotation device based on a multimodal large model, referring to Figure 6 As shown, Figure 6 : This is a structural diagram of a video fine-grained understanding and annotation device based on a multimodal large model provided in an embodiment of the present application, the device comprising:

[0176] An acquisition module 610 is configured to acquire a target video segment and a coarse-grained text annotation corresponding to the target video segment;

[0177] A static information module 620 is configured to input multiple key frames of the target video segment and the coarse-grained text annotations into an image multimodal large model to generate static information, thereby obtaining static fine-grained description information, and to obtain multiple candidate static fine-grained text annotations based on the static fine-grained description information;

[0178] The dynamic information module 630 is configured to input the target question for dynamic content understanding and the target video clip into the video multimodal large model to generate dynamic information, obtain the target question answer and dynamic fine-grained description information, and obtain multiple candidate dynamic fine-grained text annotations based on the target question answer and the dynamic fine-grained description information;

[0179] An evaluation module 640 is configured to use an evaluation model to evaluate the matching degree between the target video segment and the multiple candidate static fine-grained text annotations and the multiple candidate dynamic fine-grained text annotations, respectively, to obtain a confidence score corresponding to each candidate static fine-grained text annotation and a confidence score corresponding to each candidate dynamic fine-grained text annotation;

[0180] The annotation module 650 is used to use the multiple candidate static fine-grained text annotations and the confidence scores corresponding to each candidate static fine-grained text annotation, as well as the multiple candidate dynamic fine-grained text annotations and the confidence scores corresponding to each candidate dynamic fine-grained text annotation, as the fine-grained annotation information of the target video segment.

[0181] In an optional embodiment, the device further includes:

[0182] A first screening module is configured to screen out a static fine-grained text annotation from the plurality of candidate static fine-grained text annotations according to a confidence score corresponding to each candidate static fine-grained text annotation;

[0183] A second screening module is configured to screen out a dynamic fine-grained text annotation from the plurality of candidate dynamic fine-grained text annotations according to a confidence score corresponding to each candidate dynamic fine-grained text annotation;

[0184] The annotation module is further configured to use the static fine-grained text annotation and the confidence score corresponding to the static fine-grained text annotation, as well as the dynamic fine-grained text annotation and the confidence score corresponding to the dynamic fine-grained text annotation, as fine-grained annotation information of the target video segment.

[0185] In an optional embodiment, the apparatus further includes a construction module for constructing the evaluation model, wherein the construction module includes:

[0186] A data set construction module is used to construct a training data set based on the sample coarse-grained annotations corresponding to the sample video clips, wherein each training data in the training data set includes: a sample video clip, the coarse-grained annotations of the sample video clip, and an enhanced sample;

[0187] A loss construction module, configured to construct a loss value based on the training data set; the loss value is used to represent the matching loss between the video clip and the text annotation;

[0188] The parameter updating module is used to optimize the model parameters of the evaluation model according to the loss value, and obtain the evaluation model after the optimization result conditions are met.

[0189] In an optional embodiment, the data set construction module includes:

[0190] A sample rewriting module is used to generate rewritten samples based on the coarse-grained annotation of the samples; the rewritten samples include: semantically unchanged rewritten positive samples, static content perturbation rewritten negative samples, and dynamic content perturbation rewritten negative samples;

[0191] A sample selection module is configured to select enhanced samples from the rewritten samples based on the similarity between the rewritten samples and the coarse-grained annotations of the samples, wherein the enhanced samples include: enhanced positive samples, enhanced negative samples rewritten with static content perturbation, and enhanced negative samples rewritten with dynamic content perturbation;

[0192] The training data module is used to take the sample video clip, the sample coarse-grained annotation, and the enhanced sample as a training data.

[0193] In an optional embodiment, the loss value includes a comparison loss value and a matching loss value, and the loss construction module includes:

[0194] A positive and negative sample module is used to mark the enhanced static content perturbation rewriting negative samples and the enhanced dynamic content perturbation rewriting negative samples in the training data set as negative samples, and mark the coarse-grained samples or enhanced positive samples as positive samples;

[0195] A contrast loss construction module is used to construct a contrast loss value based on the negative sample annotation and the positive sample annotation; the contrast loss value includes: a first contrast loss value and a second contrast loss value; the first contrast loss value is used to characterize the matching between the sample video segment and the negative sample annotation, and the second contrast loss value is used to characterize the matching between the positive sample annotation and the sample video segment;

[0196] A matching loss construction module is used to construct a matching loss value based on the negative sample annotation and the positive sample annotation, and the matching loss value includes: a first matching loss value, a second matching loss value and a third matching loss value; the first matching loss value is used to characterize the matching between the sample video clip and the matched positive sample annotation, the second matching loss value is used to characterize the matching between the sample video clip and the unmatched negative sample annotation, and the third matching loss value is used to characterize the matching between the positive sample annotation and the unmatched sample video clip.

[0197] In an optional embodiment, the device also includes a key frame acquisition module, which is further used to divide the target video segment into multiple video sub-segments according to content changes of the target video segment; use the intermediate frame of each video sub-segment as the key frame of the video sub-segment; and obtain multiple key frames of the target video segment based on the key frames of the multiple video sub-segments.

[0198] In an optional embodiment, the static fine-grained description information includes: foreground detail information, background detail information and fine-grained description information; the first annotation module is also used to input the foreground detail information, the background detail information, the fine-grained description information and the coarse-grained text annotation into a large language model for static attribute information extraction and annotation generation to obtain multiple candidate static fine-grained text annotations.

[0199] In an optional embodiment, the device further includes a target question acquisition module, the target question acquisition module being configured to input the coarse-grained text annotations into a large language model to generate a question, thereby obtaining a target question for dynamic content understanding;

[0200] The second annotation module is further used to input the answer to the target question, the dynamic fine-grained description information and the coarse-grained text annotation into a large language model to extract dynamic attribute information and generate annotations to obtain multiple candidate dynamic fine-grained text annotations.

[0201] The present application also provides an electronic device, Figure 7 , Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 7 As shown, the electronic device 700 includes: a memory 710 and a processor 720. The memory 710 and the processor 720 are connected via a bus communication. A computer program is stored in the memory 710, and the computer program can be run on the processor 720, thereby realizing the steps of the video fine-grained understanding and annotation method based on a multimodal large model described in the embodiment of the present application.

[0202] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the video fine-grained understanding and annotation method based on a multimodal large model described in the embodiment of the present application are implemented.

[0203] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the video fine-grained understanding and annotation method based on a multimodal large model described in the embodiment of the present application.

[0204] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0205] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0206] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0207] The above is a detailed introduction to the video fine-grained understanding and annotation method and device based on a multimodal large model provided by this application. This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method of this application and its core idea; at the same time, for general technical personnel in this field, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on this application.

Claims

1. A video fine-grained understanding and annotation method based on a multimodal large model, characterized by: The method comprises: Obtaining a target video segment and a coarse-grained text annotation corresponding to the target video segment; Inputting multiple key frames of the target video clip and the coarse-grained text annotations into the image multimodal large model to generate static information to obtain static fine-grained description information, and obtaining multiple candidate static fine-grained text annotations based on the static fine-grained description information; The target question for dynamic content understanding and the target video clip are input into the video multimodal large model for dynamic information generation, thereby obtaining the answer to the target question and dynamic fine-grained description information. Based on the answer to the target question and the dynamic fine-grained description information, multiple candidate dynamic fine-grained text annotations are obtained. Using the evaluation model, respectively evaluate the matching degree between the target video clip and the multiple candidate static fine-grained text annotations and the multiple candidate dynamic fine-grained text annotations, and obtain a confidence score corresponding to each candidate static fine-grained text annotation and a confidence score corresponding to each candidate dynamic fine-grained text annotation; The multiple candidate static fine-grained text annotations and the confidence score corresponding to each candidate static fine-grained text annotation, as well as the multiple candidate dynamic fine-grained text annotations and the confidence score corresponding to each candidate dynamic fine-grained text annotation, are used as the fine-grained annotation information of the target video segment.

2. The method according to claim 1, characterized in that The method further comprises: Filtering a static fine-grained text annotation from the plurality of candidate static fine-grained text annotations according to the confidence score corresponding to each candidate static fine-grained text annotation; Filtering a dynamic fine-grained text annotation from the multiple candidate dynamic fine-grained text annotations according to the confidence score corresponding to each candidate dynamic fine-grained text annotation; The method further comprises: using the plurality of candidate static fine-grained text annotations and the confidence score corresponding to each candidate static fine-grained text annotation, and the plurality of candidate dynamic fine-grained text annotations and the confidence score corresponding to each candidate dynamic fine-grained text annotation as the fine-grained annotation information of the target video segment, including: The static fine-grained text annotation and the confidence score corresponding to the static fine-grained text annotation, as well as the dynamic fine-grained text annotation and the confidence score corresponding to the dynamic fine-grained text annotation, are used as the fine-grained annotation information of the target video segment.

3. The method according to claim 1, characterized in that The evaluation model is constructed through the following steps: Constructing a training data set based on the sample coarse-grained annotations corresponding to the sample video clips, wherein each training data in the training data set includes: a sample video clip, the coarse-grained annotations of the sample video clip, and an enhanced sample; Constructing a loss value based on the training data set; the loss value is used to represent the matching loss between the video clip and the text annotation; The model parameters of the evaluation model are optimized according to the loss value, and the evaluation model is obtained after the optimization result conditions are met.

4. The method according to claim 3, characterized in that Based on the coarse-grained annotations of the sample video clips, a training dataset is constructed, including: Generate rewritten samples based on the coarse-grained annotation of the samples; the rewritten samples include: semantically unchanged rewritten positive samples, static content perturbation rewritten negative samples, and dynamic content perturbation rewritten negative samples; Selecting enhanced samples from the rewritten samples according to the similarity between the rewritten samples and the coarse-grained annotations of the samples, wherein the enhanced samples include: enhanced positive samples, enhanced negative samples rewritten with static content perturbation, and enhanced negative samples rewritten with dynamic content perturbation; The sample video clip, the sample coarse-grained annotation, and the enhanced sample are used as a training data.

5. The method according to claim 3, characterized in that The loss value includes a comparison loss value and a matching loss value. Constructing the loss value based on the training data set includes: Label the enhanced static content perturbation rewriting negative samples and the enhanced dynamic content perturbation rewriting negative samples in the training data set as negative samples, and label the coarse-grained labeled samples or enhanced positive samples as positive samples; Constructing a contrast loss value based on the negative sample annotation and the positive sample annotation; the contrast loss value includes: a first contrast loss value and a second contrast loss value; the first contrast loss value is used to characterize the matching between the sample video clip and the negative sample annotation, and the second contrast loss value is used to characterize the matching between the positive sample annotation and the sample video clip; According to the negative sample annotation and the positive sample annotation, a matching loss value is constructed, and the matching loss value includes: a first matching loss value, a second matching loss value and a third matching loss value; the first matching loss value is used to characterize the matching between the sample video clip and the matched positive sample annotation, the second matching loss value is used to characterize the matching between the sample video clip and the unmatched negative sample annotation, and the third matching loss value is used to characterize the matching between the positive sample annotation and the unmatched sample video clip.

6. The method according to claim 1, characterized in that Before inputting the multiple key frames of the target video segment and the coarse-grained text annotations into the image multimodal large model for static information generation, the method further includes: dividing the target video segment into a plurality of video sub-segments according to content changes of the target video segment; The middle frame of each video sub-segment is used as the key frame of the video sub-segment; A plurality of key frames of the target video segment are obtained according to the key frames of the plurality of video sub-segments.

7. The method according to claim 1, characterized in that The static fine-grained description information includes: foreground detail information, background detail information and fine-grained description information; According to the static fine-grained description information, a plurality of candidate static fine-grained text annotations are obtained, including: The foreground detail information, the background detail information, the fine-grained description information and the coarse-grained text annotation are input into a large language model to extract static attribute information and generate annotations, thereby obtaining a plurality of candidate static fine-grained text annotations.

8. The method according to claim 1, characterized in that Before inputting the target problem for dynamic content understanding and the target video clip into the video multimodal large model for dynamic information generation, the method further includes: Inputting the coarse-grained text annotations into a large language model to generate questions, thereby obtaining target questions for dynamic content understanding; According to the answer to the target question and the dynamic fine-grained description information, multiple candidate dynamic fine-grained text annotations are obtained, including: The answer to the target question, the dynamic fine-grained description information and the coarse-grained text annotation are input into a large language model to extract dynamic attribute information and generate annotations, thereby obtaining a plurality of candidate dynamic fine-grained text annotations.

9. A video fine-grained understanding and annotation device based on a multimodal large model, characterized by: The device comprises: An acquisition module, configured to acquire a target video segment and a coarse-grained text annotation corresponding to the target video segment; a static information module, configured to input multiple key frames of the target video clip and the coarse-grained text annotations into an image multimodal large model to generate static information, thereby obtaining static fine-grained description information, and to obtain multiple candidate static fine-grained text annotations based on the static fine-grained description information; A dynamic information module is used to input the target question for dynamic content understanding and the target video clip into the video multimodal large model to generate dynamic information, obtain the answer to the target question and dynamic fine-grained description information, and obtain multiple candidate dynamic fine-grained text annotations based on the target question answer and the dynamic fine-grained description information; An evaluation module is configured to use an evaluation model to evaluate the matching degree between the target video segment and the multiple candidate static fine-grained text annotations and the multiple candidate dynamic fine-grained text annotations, respectively, to obtain a confidence score corresponding to each candidate static fine-grained text annotation and a confidence score corresponding to each candidate dynamic fine-grained text annotation; The annotation module is used to use the multiple candidate static fine-grained text annotations and the confidence score corresponding to each candidate static fine-grained text annotation, as well as the multiple candidate dynamic fine-grained text annotations and the confidence score corresponding to each candidate dynamic fine-grained text annotation, as the fine-grained annotation information of the target video segment.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the video fine-grained understanding and annotation method based on a multimodal large model are implemented as described in any one of claims 1-8.

Images