Video labeling method and device, storage medium and equipment
Through the OVSeg model and graph neural network, the video is divided into multiple video clips, and text descriptions are generated in combination with the multimodal model, which solves the problem that the multimodal model cannot accurately describe the video features, improves the labeling quality and efficiency, and adapts to complex videos content.
Patent Information
- Application Number
- CN202411137592.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-11-29
AI Technical Summary
The existing multimodal model cannot accurately describe all the features in the video picture, and hallucinations are prone to occur during the processing, affecting the quality of the labeling.
The OVSeg model is used to extract the object feature information, category information and position information in the video, and use the graph neural network to divide the video into multiple video segments. Combined with the multimodal model, it generates text based on user prompt words and domain prompt words. Description, adjust the description through the scoring model to improve accuracy.
It improves the quality and efficiency of video labeling, enhances flexibility and adaptability to complex and variable video content, and ensures the accuracy of labeling results and the timeliness of user feedback.
Smart Images

Figure CN119052599A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of deep learning technology, and in particular to a video annotation method, apparatus, storage medium and device. Background Art
[0002] By manually annotating videos, we can generate text descriptions of videos. First, we manually slice the video to get multiple video clips. Then, we manually summarize these video clips and express the content of the video in natural language. This video annotation method relies on manual observation and understanding capabilities, which requires a lot of manpower and time.
[0003] In the related art, a multimodal model can be used to annotate videos to obtain text descriptions. However, the current multimodal model cannot accurately describe all features in the video screen, and hallucinations are prone to occur during the processing, which affects the annotation quality. Summary of the invention
[0004] The present application provides a video annotation method, device, storage medium and equipment, which are used to solve the problem that the multimodal model cannot accurately describe all the features in the video screen, and the problem of hallucination is prone to occur during the processing, which affects the annotation quality. The technical solution is as follows:
[0005] According to a first aspect of the present application, a video annotation method is provided, the method comprising:
[0006] Obtaining a video to be annotated and a user prompt word, wherein the user prompt word is used to indicate the user's focus on the video;
[0007] Extracting feature information, category information, and location information of objects in the video using an open semantic segmentation OVSeg model;
[0008] Extracting video key frames from the video, using a graph neural network to construct nodes of a graph structure according to the video key frames and feature information of each object in the video key frames, constructing edges of a graph structure according to category relationships, spatial position relationships, and time relationships of each object in the video key frames, and dividing the video into multiple video segments according to the graph structure, each video segment including multiple video frames that are continuous and belong to the same field;
[0009] For each video clip, generating a domain prompt word according to the category information of the object in the video clip and the user prompt word, wherein the domain prompt word is used to indicate the domain to which the video clip belongs;
[0010] A multimodal model is used to generate a text description of the video segment according to the user prompt word, the field prompt word and the corresponding video segment.
[0011] In a possible implementation, the generating of the domain prompt word according to the category information of the object in the video clip and the user prompt word includes:
[0012] Determining a domain label of the video clip according to category information of the object in the video clip;
[0013] Converting the domain label into a first feature vector;
[0014] Converting the user prompt word into a second feature vector;
[0015] The first feature vector and the second feature vector are processed using a prompt word generation model to obtain a domain prompt word.
[0016] In a possible implementation, the method further includes:
[0017] For each video segment, extracting a domain label of the video segment;
[0018] Calculate the similarity between the user prompt word and each field label using a semantic similarity model;
[0019] The domain tags associated with the user prompt word are filtered according to the similarity.
[0020] In a possible implementation, the step of constructing a node of a graph structure according to the video key frame and feature information of each object in the video key frame includes:
[0021] For each video key frame, the feature information of the object in the video key frame is fused with the timestamp information of the video key frame to obtain a fused feature;
[0022] The fused features are constructed as nodes in the graph structure.
[0023] In a possible implementation, the step of constructing edges of a graph structure according to the category relationship, spatial position relationship, and time relationship of each object in the video key frame includes:
[0024] Obtaining the category of each object in each video key frame, determining video key frames having objects of the same category, and constructing edges between nodes corresponding to the video key frames;
[0025] Obtaining the position of each object in each video key frame, determining the relative position of each object in different video frames, and constructing edges between nodes corresponding to the video key frames according to the relative positions;
[0026] The timestamp information of each video key frame is obtained, and edges are constructed between nodes corresponding to the video key frames according to the timestamp information.
[0027] In a possible implementation, the method further includes:
[0028] Scoring the matching degree between the text description and the video clip using a scoring model;
[0029] Determining whether to regenerate the text description according to the score;
[0030] If the text description needs to be regenerated, feedback information for the text description is obtained, and the text description is adjusted according to the feedback information.
[0031] In a possible implementation manner, adjusting the text description according to the feedback information includes:
[0032] When the feedback information is a newly added user prompt word, a text description of the video segment is generated using a multimodal model according to the added user prompt word, the field prompt word and the corresponding video segment;
[0033] When the feedback information is a modification suggestion, a text description of the video segment is generated according to the user prompt word, the domain prompt word, the modification suggestion and the corresponding video segment using a multimodal model.
[0034] According to a second aspect of the present application, a video annotation device is provided, the device comprising:
[0035] An acquisition module, used to acquire a video to be annotated and a user prompt word, wherein the user prompt word is used to indicate a user's focus on the video;
[0036] An extraction module, used to extract feature information, category information and location information of objects in the video using an open semantic segmentation OVSeg model;
[0037] A segmentation module, configured to extract video key frames from the video, construct nodes of a graph structure according to the video key frames and feature information of each object in the video key frames using a graph neural network, construct edges of a graph structure according to a category relationship, a spatial position relationship, and a time relationship of each object in the video key frames, and divide the video into a plurality of video segments according to the graph structure, wherein each video segment includes a plurality of continuous video frames belonging to the same field;
[0038] A generating module, configured to generate, for each video clip, a domain prompt word according to category information of an object in the video clip and the user prompt word, wherein the domain prompt word is used to indicate the domain to which the video clip belongs;
[0039] The generation module is also used to generate a text description of the video clip based on the user prompt word, the field prompt word and the corresponding video clip by using a multimodal model.
[0040] According to a third aspect of the present application, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement the video annotation method as described above.
[0041] According to a fourth aspect of the present application, a computer device is provided, wherein the computer device includes the above-mentioned video annotation device.
[0042] The beneficial effects of the technical solution provided by this application include at least:
[0043] The OVSeg model is used to extract feature information, category information, and location information of objects in the video. The graph neural network is used to divide the video into multiple video segments according to the field based on the extracted information, and field prompt words are generated for each video segment. The multimodal model is then used to generate text descriptions based on user prompt words, field prompt words, and corresponding video segments. The OVSeg model, graph neural network, and multimodal model can be used to jointly generate text descriptions of video segments, thereby improving the annotation quality.
[0044] The matching degree between the text description and the video clip is scored through the scoring model, and whether the text description needs to be regenerated is determined based on the score. If the text description needs to be regenerated, the text description is adjusted according to the feedback information. Errors can be corrected or annotation results can be optimized in time based on user feedback, thereby improving the quality and efficiency of annotation.
[0045] By extracting the category information of objects in video clips, determining the domain label of the video clip according to the category information of the object, converting the domain label into a first feature vector, converting the user prompt word into a second feature vector, and using the prompt word generation model to process the first feature vector and the second feature vector to obtain the domain prompt word, different objects and fields can be differentially labeled, which can improve the flexibility and adaptability of labeling when faced with complex and changeable video content. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0047] Figure 1is a flow chart of a video annotation method provided by an embodiment of the present application;
[0048] Figure 2 is a flow chart of a video annotation method provided by an embodiment of the present application;
[0049] Figure 3 is a schematic diagram of a graph structure provided by an embodiment of the present application;
[0050] Figure 4 It is a schematic diagram of evaluating a text description provided by an embodiment of the present application;
[0051] Figure 5 is a schematic diagram of a video annotation method provided by an embodiment of the present application;
[0052] Figure 6 This is a structural block diagram of a video annotation device provided by an embodiment of the present application. DETAILED DESCRIPTION
[0053] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.
[0054] like Figure 1 As shown, it shows a method flow chart of a video annotation method provided by an embodiment of the present application, and the video annotation method can be applied to a computer device. The video annotation method may include:
[0055] Step 101: obtain a video to be annotated and a user prompt word, where the user prompt word is used to indicate the user's focus on the video.
[0056] The user prompt word is a prompt word input by the user, which can indicate the user's focus on the video. The user prompt word can be a keyword, a word, a sentence, etc., which is not limited in this embodiment.
[0057] In one example, the user prompt may be: "I hope you will pay attention to the content in the field of movies and describe the plot information in detail."
[0058] Step 102: Use the OVSeg model to extract feature information, category information, and location information of objects in the video.
[0059] The Open-Vocabulary Segmentation (OVSeg) model can extract the feature information of objects in the video, and then generate the category information and location information of the objects based on the feature information.
[0060] Step 103: extract video key frames from the video, use a graph neural network to construct nodes of a graph structure according to the video key frames and feature information of each object in the video key frames, construct edges of the graph structure according to the category relationship, spatial position relationship and time relationship of each object in the video key frames, and divide the video into multiple video clips according to the graph structure, each video clip includes multiple continuous video frames belonging to the same field.
[0061] The video key frames are extracted as nodes in the graph structure at fixed intervals, and the interval depends on the computing power and the requirements for segmentation effect. That is, the larger the interval, the fewer video frames need to be processed, the less computing power is required, and the worse the segmentation effect.
[0062] After obtaining the video key frames, obtain the feature information of the objects extracted by the OVSeg model, and construct the nodes of the graph structure according to the video key frames and the feature information; obtain the category information of the objects extracted by the OVSeg model, and generate the category relationship of each object according to the category information; obtain the position relationship of the objects extracted by the OVSeg model, and generate the spatial position relationship of each object according to the position relationship; obtain the timestamp information of the video key frames, and generate the time relationship of each video key frame according to the timestamp information; construct the edges of the graph structure according to the category relationship, spatial position relationship and time relationship to obtain the graph structure.
[0063] After the graph structure is built, the graph neural network is used for domain prediction. First, the graph neural network aggregates the neighbor information of each node to capture the contextual information of the local domain; then, the graph neural network inputs the aggregated features into the classifier (which can be part of the graph neural network) to predict the domain category to which each video key frame belongs; finally, the continuous video frames belonging to the same domain category are divided into a video clip.
[0064] Step 104: for each video clip, a domain prompt word is generated according to the category information of the object in the video clip and the user prompt word, and the domain prompt word is used to indicate the domain to which the video clip belongs.
[0065] In this embodiment, the same video clip may contain different fields, and prompt words with different focus points need to be generated according to user prompt words to meet user needs, thereby improving the utilization rate of the video clip and the adaptability of annotation.
[0066] For example, when the user prompt is "I hope you will pay attention to the content in the film field and describe the plot information in detail", the final generated field prompt is "This video belongs to the film field. The picture contains two men and there is food on the table. Please describe the specific plot in the picture."
[0067] Step 105: Generate a text description of the video clip according to the user prompt word, the domain prompt word and the corresponding video clip using a multimodal model.
[0068] The multimodal model can be a Large Language and Vision Assistant (LLaVA-1.6), which can simultaneously receive text and images and generate corresponding natural language descriptions. The multimodal model LLaVA-1.6 can combine user prompts and domain prompts to generate text descriptions of video clips that match user needs.
[0069] To summarize, the video annotation method provided in the embodiment of the present application extracts feature information, category information and location information of objects in the video through the OVSeg model, divides the video into multiple video segments according to fields based on the extracted information through the graph neural network, generates field prompt words for each video segment, and then uses the multimodal model to generate a text description based on user prompt words, field prompt words and corresponding video segments. The text description of the video segment can be jointly generated based on the OVSeg model, the graph neural network and the multimodal model, thereby improving the annotation quality.
[0070] By extracting the category information of objects in video clips, determining the domain label of the video clip according to the category information of the object, converting the domain label into a first feature vector, converting the user prompt word into a second feature vector, and using the prompt word generation model to process the first feature vector and the second feature vector to obtain the domain prompt word, different objects and fields can be differentially labeled, which can improve the flexibility and adaptability of labeling when faced with complex and changeable video content.
[0071] like Figure 2 As shown, it shows a flow chart of a video annotation method provided by an embodiment of the present application, and the video annotation method can be applied to a computer device. The video annotation method may include:
[0072] Step 201: obtain a video to be annotated and a user prompt word, where the user prompt word is used to indicate the user's focus on the video.
[0073] The user prompt word is a prompt word input by the user, which can indicate the user's focus on the video. The user prompt word can be a keyword, a word, a sentence, etc., which is not limited in this embodiment.
[0074] In one example, the user prompt may be: "I hope you will pay attention to the content in the field of movies and describe the plot information in detail."
[0075] Step 202: Extract feature information, category information, and location information of objects in the video using the OVSeg model.
[0076] The OVSeg model can extract the feature information of objects in the video, and then generate the category information and location information of the objects based on the feature information.
[0077] Step 203: extract video key frames from the video, use a graph neural network to construct nodes of a graph structure according to the video key frames and feature information of each object in the video key frames, construct edges of the graph structure according to the category relationship, spatial position relationship and time relationship of each object in the video key frames, and divide the video into multiple video segments according to the graph structure, each video segment includes multiple continuous video frames belonging to the same field.
[0078] The video key frames are extracted as nodes in the graph structure at fixed intervals, and the interval depends on the computing power and the requirements for segmentation effect. That is, the larger the interval, the fewer video frames need to be processed, the less computing power is required, and the worse the segmentation effect.
[0079] Specifically, constructing nodes of a graph structure based on video key frames and feature information of each object in the video key frames may include: for each video key frame, fusing the feature information of the object in the video key frame with the timestamp information of the video key frame to obtain a fused feature; and constructing the fused feature as a node in the graph structure.
[0080] For each video key frame, we fuse the feature information of the object with the timestamp information to form a new fused feature, which not only contains the static properties of the object, but also incorporates the dynamic information of the object changing over time; then, each fused feature is used as a node in the graph structure.
[0081] We also need to set up edges between nodes. The edges are constructed based on the following information:
[0082] 1) Physical category similarity: If objects in two video frames belong to the same physical category, an edge between them is established.
[0083] 2) Spatial position relationship: Establish spatial relationship edges based on the relative positions of the same object in the video frame.
[0084] 3) Temporal continuity: Based on the timestamp information of the video frames, temporal continuity edges are established to capture the changes of objects over time.
[0085] Correspondingly, constructing the edges of the graph structure according to the category relationship, spatial position relationship and time relationship of each object in the video key frame may include: obtaining the category of each object in each video key frame, determining the video key frames with objects of the same category, and constructing edges between the nodes corresponding to the video key frames; obtaining the position of each object in each video key frame, determining the relative position of each object in different video frames, and constructing edges between the nodes corresponding to the video key frames according to the relative positions; obtaining the timestamp information of each video key frame, and constructing edges between the nodes corresponding to the video key frames according to the timestamp information.
[0086] like Figure 3 As shown, the categories of object 2 in video key frame 1 and video key frame 2 are similar, the positions of object 1 in video key frame 2 and video key frame 3 are close, the colors of multiple objects in video key frame 1 and video key frame n are similar, and video key frame 3 and video key frame n have other feature connections.
[0087] After the graph structure is built, the graph neural network is used for domain prediction. First, the graph neural network aggregates the neighbor information of each node to capture the contextual information of the local domain; then, the graph neural network inputs the aggregated features into the classifier (which can be part of the graph neural network) to predict the domain category to which each video key frame belongs; finally, the continuous video frames belonging to the same domain category are divided into a video clip.
[0088] Step 204: for each video clip, a domain prompt word is generated according to the category information of the object in the video clip and the user prompt word, and the domain prompt word is used to indicate the domain to which the video clip belongs.
[0089] In this embodiment, the same video clip may contain different fields, and prompt words with different focus points need to be generated according to user prompt words to meet user needs, thereby improving the utilization rate of the video clip and the adaptability of annotation.
[0090] For example, when the user prompt is "I hope you will pay attention to the content in the film field and describe the plot information in detail", the final generated field prompt is "This video belongs to the film field. The picture contains two men and there is food on the table. Please describe the specific plot in the picture."
[0091] Specifically, generating domain prompt words based on the category information of objects in the video clip and user prompt words can include: determining the domain label of the video clip based on the category information of the objects in the video clip; converting the domain label into a first feature vector; converting the user prompt words into a second feature vector; and using a prompt word generation model to process the first feature vector and the second feature vector to obtain the domain prompt words.
[0092] The prompt word generation model is a Transformer-based model that can generate domain prompt words that match user needs based on video clips, domain labels, and user prompt words. In this embodiment, a word embedding model or a Transformer-based bidirectional encoder representation (Bidirectional Encoder Representations from Transformer, BERT) model can be used to encode the domain label to obtain a first feature vector, and the user prompt word is encoded using the word embedding model to obtain a second feature vector. The first feature vector and the second feature vector are input into the prompt word generation model for processing to obtain a domain prompt word that better meets user needs, which is used to guide the multimodal model to generate a text description of the video clip.
[0093] Optionally, after obtaining the domain labels, the domain labels can be filtered based on the confidence mechanism to optimize the accuracy of the segmentation results. Specifically, for each video clip, the domain label of the video clip is extracted; the similarity between the user prompt word and each domain label is calculated using the semantic similarity model; and the domain label associated with the user prompt word is filtered according to the similarity.
[0094] Users can input user prompt words to clarify their focus or needs. The semantic similarity model calculates the semantic similarity between the domain labels in the extracted video clips and the user prompt words to evaluate the degree of association between the domain labels and the user prompt words. Then, several domain labels that are most relevant to the user's needs are screened out according to the preset threshold as input for subsequent operations. If the user prompt word requires information about the film field, then the matching confidence of the domain prompt words that are not related to movies in the domain prompt words will be extremely low and will be screened out. For example, if the video contains the objects "two men, table, and food", it may output "film field" or it may belong to "life and drama field", but the user prompt words specify movie-related content, then the matching confidence of "life and drama field" is low and will be screened out.
[0095] Step 205: Generate a text description of the video clip according to the user prompt word, the domain prompt word and the corresponding video clip using the multimodal model.
[0096] The multimodal model may be LLaVA-1.6, which is a large multimodal model capable of simultaneously receiving text and images and generating corresponding natural language descriptions. The multimodal model LLaVA-1.6 may combine user prompt words and domain prompt words to generate a text description of a video clip that matches user needs.
[0097] Step 206, using the scoring model to score the matching degree between the text description and the video clip; judging whether the text description needs to be regenerated based on the score; if the text description needs to be regenerated, obtaining feedback information for the text description, and adjusting the text description based on the feedback information.
[0098] In this embodiment, a scoring threshold may be set. If the score is greater than or equal to the scoring threshold, it is determined that the text description needs to be regenerated. If the score is less than the scoring threshold, it is determined that the text description does not need to be regenerated and the text description is directly output. Alternatively, the computer device may display the score to the user, and the user may use the score as a reference to determine whether the text description needs to be regenerated. If the text description needs to be regenerated, feedback information is input. If the text description does not need to be generated, the text description is not generated.
[0099] Specifically, adjusting the text description according to the feedback information may include:
[0100] (1) When the feedback information is a newly added user prompt word, a multimodal model is used to generate a text description of the video clip according to the added user prompt word, domain prompt word and the corresponding video clip.
[0101] For example, if the text description does not describe the background of the video clip, a user prompt word "background description" may be added.
[0102] (2) When the feedback information is a modification suggestion, a multimodal model is used to generate a text description of the video clip based on the user prompt words, domain prompt words, modification suggestions and corresponding video clips.
[0103] For example, the text description is "The background of the video is in a dimly lit room, full of a serious atmosphere." However, this part may be inconsistent with the actual scene, so you can give a modification suggestion "Although the light in the room is dim, the atmosphere is relatively warm. Please make modifications."
[0104] like Figure 4 As shown, after the multimodal model generates a text description based on the video clip, it determines whether the text description meets the user's needs. If it does not meet the user's needs, user prompt words are added, and the domain prompt words and the added user prompt words are input into the multimodal model to regenerate the text description; if it meets the user's needs, the text description is directly output.
[0105] like Figure 5As shown in the figure, video frames are first extracted from the video, and the categories of objects in the video frames are extracted using the OVSeg model. The video is segmented based on the video frames and object categories using a graph neural network, and domain labels are generated according to the object categories. Domain prompt words are generated according to the domain labels and user prompt words. The multimodal model generates text descriptions based on user prompt words, domain prompt words, and video clips, and the text descriptions are comprehensively scored. If manual feedback is required, user prompt words can be added, and the text description is regenerated based on the added user prompt words and domain prompt words.
[0106] To summarize, the video annotation method provided in the embodiment of the present application extracts feature information, category information and location information of objects in the video through the OVSeg model, divides the video into multiple video segments according to fields based on the extracted information through the graph neural network, generates field prompt words for each video segment, and then uses the multimodal model to generate a text description based on user prompt words, field prompt words and corresponding video segments. The text description of the video segment can be jointly generated based on the OVSeg model, the graph neural network and the multimodal model, thereby improving the annotation quality.
[0107] The matching degree between the text description and the video clip is scored through the scoring model, and whether the text description needs to be regenerated is determined based on the score. If the text description needs to be regenerated, the text description is adjusted according to the feedback information. Errors can be corrected or annotation results can be optimized in time based on user feedback, thereby improving the quality and efficiency of annotation.
[0108] By extracting the category information of objects in video clips, determining the domain label of the video clip according to the category information of the object, converting the domain label into a first feature vector, converting the user prompt word into a second feature vector, and using the prompt word generation model to process the first feature vector and the second feature vector to obtain the domain prompt word, different objects and fields can be differentially labeled, which can improve the flexibility and adaptability of labeling when faced with complex and changeable video content.
[0109] like Figure 6 As shown, it shows a structural block diagram of a video annotation device provided by an embodiment of the present application, and the video annotation device can be applied to a computer device. The video annotation device may include:
[0110] An acquisition module 610 is used to acquire a video to be annotated and a user prompt word, where the user prompt word is used to indicate the user's focus on the video;
[0111] An extraction module 620, for extracting feature information, category information, and location information of objects in the video using the OVSeg model;
[0112] A segmentation module 630 is used to extract video key frames from a video, construct nodes of a graph structure according to the video key frames and feature information of each object in the video key frames using a graph neural network, construct edges of the graph structure according to the category relationship, spatial position relationship and time relationship of each object in the video key frames, and divide the video into multiple video segments according to the graph structure, each video segment includes multiple continuous video frames belonging to the same field;
[0113] A generating module 640, configured to generate, for each video clip, a domain prompt word according to category information of an object in the video clip and a user prompt word, wherein the domain prompt word is used to indicate the domain to which the video clip belongs;
[0114] The generation module 640 is further used to generate a text description of the video segment according to the user prompt word, the domain prompt word and the corresponding video segment using the multimodal model.
[0115] In an optional embodiment, the generating module 640 is further configured to:
[0116] Determine a domain label of the video clip according to category information of the object in the video clip;
[0117] Convert the domain label into the first feature vector;
[0118] Convert the user prompt word into a second feature vector;
[0119] The first feature vector and the second feature vector are processed using a prompt word generation model to obtain a domain prompt word.
[0120] In an optional embodiment, the generating module 640 is further configured to:
[0121] For each video segment, extract the domain label of the video segment;
[0122] The semantic similarity model is used to calculate the similarity between the user prompt words and the labels of various fields;
[0123] Filter the domain labels associated with the user prompt words based on similarity.
[0124] In an optional embodiment, the partitioning module 630 is further configured to:
[0125] For each video key frame, the feature information of the object in the video key frame is fused with the timestamp information of the video key frame to obtain a fused feature;
[0126] The fused features are constructed as nodes in a graph structure.
[0127] In an optional embodiment, the partitioning module 630 is configured to:
[0128] Obtain the category of each object in each video key frame, determine the video key frames with objects of the same category, and build edges between the nodes corresponding to the video key frames;
[0129] Obtain the position of each object in each video key frame, determine the relative position of each object in different video frames, and build edges between the nodes corresponding to the video key frames according to the relative positions;
[0130] The timestamp information of each video key frame is obtained, and edges are constructed between the nodes corresponding to the video key frames according to the timestamp information.
[0131] In an optional embodiment, the generating module 640 is further configured to:
[0132] Use the scoring model to score the matching degree between the text description and the video clip;
[0133] Determine whether the text description needs to be regenerated based on the score;
[0134] If the text description needs to be regenerated, feedback information on the text description is obtained, and the text description is adjusted according to the feedback information.
[0135] In an optional embodiment, the generating module 640 is further configured to:
[0136] When the feedback information is a newly added user prompt word, a text description of the video segment is generated using the multimodal model according to the added user prompt word, the domain prompt word and the corresponding video segment;
[0137] When the feedback information is a modification suggestion, a multimodal model is used to generate a text description of the video clip according to the user prompt word, the domain prompt word, the modification suggestion and the corresponding video clip.
[0138] To summarize, the video annotation device provided in the embodiment of the present application extracts feature information, category information, and location information of objects in the video through the OVSeg model, divides the video into multiple video segments according to fields based on the extracted information through the graph neural network, generates field prompt words for each video segment, and then uses the multimodal model to generate a text description based on user prompt words, field prompt words and corresponding video segments. The text description of the video segment can be jointly generated based on the OVSeg model, the graph neural network and the multimodal model, thereby improving the annotation quality.
[0139] The matching degree between the text description and the video clip is scored through the scoring model, and whether the text description needs to be regenerated is determined based on the score. If the text description needs to be regenerated, the text description is adjusted according to the feedback information. Errors can be corrected or annotation results can be optimized in time based on user feedback, thereby improving the quality and efficiency of annotation.
[0140] By extracting the category information of objects in video clips, determining the domain label of the video clip according to the category information of the object, converting the domain label into a first feature vector, converting the user prompt word into a second feature vector, and using the prompt word generation model to process the first feature vector and the second feature vector to obtain the domain prompt word, different objects and fields can be differentially labeled, which can improve the flexibility and adaptability of labeling when faced with complex and changeable video content.
[0141] An embodiment of the present application provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the video annotation method as described above.
[0142] An embodiment of the present application provides a computer device, wherein the computer device includes the above-mentioned device for annotating any video.
[0143] It should be noted that: the video annotation device provided in the above embodiment only uses the division of the above functional modules as an example when annotating the video. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the video annotation device is divided into different functional modules to complete all or part of the functions described above. In addition, the video annotation device provided in the above embodiment and the video annotation method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.
[0144] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.
[0145] The above description is not intended to limit the embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the protection scope of the embodiments of the present application.
Claims
1. A video annotation method, characterized in that: The method comprises: Obtaining a video to be annotated and a user prompt word, wherein the user prompt word is used to indicate the user's focus on the video; Extracting feature information, category information, and location information of objects in the video using an open semantic segmentation OVSeg model; Extracting video key frames from the video, using a graph neural network to construct nodes of a graph structure according to the video key frames and feature information of each object in the video key frames, constructing edges of a graph structure according to category relationships, spatial position relationships, and time relationships of each object in the video key frames, and dividing the video into multiple video segments according to the graph structure, each video segment including multiple video frames that are continuous and belong to the same field; For each video clip, generating a domain prompt word according to the category information of the object in the video clip and the user prompt word, wherein the domain prompt word is used to indicate the domain to which the video clip belongs; A multimodal model is used to generate a text description of the video segment according to the user prompt word, the field prompt word and the corresponding video segment.
2. The video annotation method according to claim 1, characterized in that: The generating of the domain prompt word according to the category information of the object in the video clip and the user prompt word comprises: Determining a domain label of the video clip according to category information of the object in the video clip; Converting the domain label into a first feature vector; Converting the user prompt word into a second feature vector; The first feature vector and the second feature vector are processed using a prompt word generation model to obtain a domain prompt word.
3. The video annotation method according to claim 2, characterized in that: The method further comprises: For each video segment, extracting a domain label of the video segment; Calculate the similarity between the user prompt word and each field label using a semantic similarity model; The domain tags associated with the user prompt word are filtered according to the similarity.
4. The video annotation method according to claim 1, characterized in that: The node of the graph structure is constructed according to the video key frame and the feature information of each object in the video key frame, including: For each video key frame, the feature information of the object in the video key frame is fused with the timestamp information of the video key frame to obtain a fused feature; The fused features are constructed as nodes in the graph structure.
5. The video annotation method according to claim 1, characterized in that: The step of constructing the edges of the graph structure according to the category relationship, spatial position relationship and time relationship of each object in the video key frame includes: Obtaining the category of each object in each video key frame, determining video key frames having objects of the same category, and constructing edges between nodes corresponding to the video key frames; Obtaining the position of each object in each video key frame, determining the relative position of each object in different video frames, and constructing edges between nodes corresponding to the video key frames according to the relative positions; The timestamp information of each video key frame is obtained, and edges are constructed between nodes corresponding to the video key frames according to the timestamp information.
6. The video annotation method according to any one of claims 1 to 5, characterized in that: The method further comprises: Scoring the matching degree between the text description and the video clip using a scoring model; Determining whether to regenerate the text description according to the score; If the text description needs to be regenerated, feedback information for the text description is obtained, and the text description is adjusted according to the feedback information.
7. The video annotation method according to claim 6, characterized in that: The adjusting the text description according to the feedback information includes: When the feedback information is a newly added user prompt word, a text description of the video segment is generated using a multimodal model according to the added user prompt word, the field prompt word and the corresponding video segment; When the feedback information is a modification suggestion, a text description of the video segment is generated according to the user prompt word, the domain prompt word, the modification suggestion and the corresponding video segment using a multimodal model.
8. A video annotation device, characterized in that: The device comprises: An acquisition module, used to acquire a video to be annotated and a user prompt word, wherein the user prompt word is used to indicate a user's focus on the video; An extraction module, used to extract feature information, category information and location information of objects in the video using an open semantic segmentation OVSeg model; A segmentation module, configured to extract video key frames from the video, construct nodes of a graph structure according to the video key frames and feature information of each object in the video key frames using a graph neural network, construct edges of a graph structure according to a category relationship, a spatial position relationship, and a time relationship of each object in the video key frames, and divide the video into a plurality of video segments according to the graph structure, wherein each video segment includes a plurality of continuous video frames belonging to the same field; A generating module, configured to generate, for each video clip, a domain prompt word according to category information of an object in the video clip and the user prompt word, wherein the domain prompt word is used to indicate the domain to which the video clip belongs; The generation module is also used to generate a text description of the video clip based on the user prompt word, the field prompt word and the corresponding video clip by using a multimodal model.
9. A computer-readable storage medium, characterized in that: At least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by the processor to implement the video labeling method according to any one of claims 1 to 7.
10. A computer device, characterized in that: The computer device comprises: the video annotation device described in claim 8.
Citation Information
Cited By
Video labeling method and device, electronic equipment, storage medium and product
CN121353994A
A video annotation method, apparatus, electronic device, storage medium, and product.
CN121353994B
