Video labeling model training method and device, equipment and storage medium
By combining graph generation models and target language models, a video annotation model is trained, which solves the problem of the simplicity of existing video annotation methods, realizes advanced semantic annotation, and supports deeper video understanding and analysis.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SOUNDAI TECH CO LTD
- Filing Date
- 2022-11-17
- Publication Date
- 2026-07-14
AI Technical Summary
Existing video annotation methods are relatively simple and cannot achieve deeper understanding and analysis, nor can they provide rich information for subsequent intelligent decision-making.
By using a graph generation model, multiple label sets are transformed into multiple views. The graph structure is optimized using graph convolutional layers and graph pooling layers. Combined with the target language model, a video annotation model is trained to achieve advanced semantic annotation.
It achieves advanced semantic annotation of videos, which can provide rich information for subsequent intelligent decision-making and support deeper video understanding and analysis.
Smart Images
Figure CN115861874B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer vision, and in particular to a training method, apparatus, device, and storage medium for a video annotation model. Background Technology
[0002] Computer vision technology has been applied to many fields, such as recorders, which are electronic products based on computer vision technology.
[0003] In detail, a recorder is a portable device that integrates real-time audio and video recording, photography, positioning, face and license plate recognition, and face and license plate comparison functions according to actual needs. In practical applications, the recorder annotates the captured videos using a video annotation model. Video annotation refers to the process of adding tags to the videos.
[0004] Current video annotation technologies typically involve adding a few simple category labels to the video. Because this method is relatively simple, it prevents researchers from gaining a deeper understanding and analysis of the video based on the added labels. In other words, this method fails to provide rich information for subsequent intelligent decision-making. Summary of the Invention
[0005] This application provides a training method, apparatus, device, and storage medium for a video annotation model, enabling high-level semantic annotation of videos. This allows for deeper understanding and analysis of videos based on high-level semantic annotation; in other words, this video annotation method can provide rich information for subsequent intelligent decision-making. The technical solution is as follows:
[0006] On the one hand, a training method for a video annotation model is provided, the method comprising:
[0007] Obtain multiple tag sets; wherein, one of the tag sets includes a category tag that describes the video content;
[0008] Based on the graph generation model in the graph module, the multiple label sets are transformed into multiple views; wherein, the multiple views use category labels as nodes to describe the relationship between category labels from different perspectives;
[0009] Based on the graph convolutional layer and graph pooling layer in the graph module, the graph structure of the multi-view is optimized to obtain the prediction graph;
[0010] A reference graph for the multiple tag sets is obtained based on the target language model;
[0011] The video annotation model is trained based on the predicted graph and the reference graph; wherein the video annotation model includes the graph module and the target language model.
[0012] In some embodiments, obtaining multiple tag sets includes:
[0013] Acquire sample video data;
[0014] Target detection is performed on video frames in the sample video data; a set of person class tags is generated based on the detected targets;
[0015] Perform action recognition on the detected targets; generate a set of action class labels based on the recognized actions;
[0016] Perform emotion recognition on the detected targets; generate a set of emotion category labels based on the recognized emotions.
[0017] In some embodiments, the graph generation model in the graph module transforms the plurality of label sets into multiple views, including:
[0018] The category labels included in the multiple label sets are used as nodes of the multiple views;
[0019] For the nth view in the multi-view, based on the Gaussian graph generation model in the graph module, the initial node representation of each node in the nth view is encoded as a Gaussian distribution;
[0020] In the nth view, the weight of the edge between node i and node j is determined based on the Gaussian distribution corresponding to node i and the Gaussian distribution corresponding to node j.
[0021] The edges of the multi-view are initialized based on the weights of the edges determined in each view;
[0022] Where n, i, and j are all positive integers; a pair of nodes in the multi-view are connected by multiple edges, and one edge corresponds to one view in the multi-view.
[0023] In some embodiments, optimizing the graph structure of the multi-view based on the graph convolutional layer and graph pooling layer in the graph module to obtain a predicted graph includes:
[0024] During each iteration, the node representation of each node in each view is updated through the current graph convolutional layer in the form of message propagation;
[0025] Based on the updated node representation of the multi-view, the attention score set of each view in the multi-view is obtained through the current graph pooling layer; wherein, one of the attention score sets includes the attention weight of each node in the view;
[0026] Based on the attention score set of each view, determine the nodes that should be retained in each view; determine the joint set of nodes that should be retained in each view as the nodes of the graph created in this graph pooling process;
[0027] The prediction graph is generated based on the nodes of the intermediate graph created during multiple graph pooling processes.
[0028] In some embodiments, obtaining the reference graph of the plurality of tag sets based on the target language model includes:
[0029] Based on external knowledge and a self-encoding language model, the multiple tag sets are transformed into the reference graph;
[0030] The external knowledge includes triples, which are used to represent the relationships between entities.
[0031] In some embodiments, training the video annotation model based on the predicted map and the reference map includes:
[0032] Construct the target loss function;
[0033] Based on the target loss function, the error value between the predicted map and the reference map is obtained;
[0034] The error value is propagated backward from the output layer of the video annotation model to the hidden layer, until it reaches the input layer of the video annotation model;
[0035] During backpropagation, the network parameters of the video annotation model are adjusted according to the error value until the video annotation model converges.
[0036] In some embodiments, the video annotation model further includes a classifier; the method further includes:
[0037] The keyframes to be labeled are input into the video labeling model to obtain semantic labels output by the classifier; wherein, the semantic labels are used to describe the semantic relationships between the category labels extracted in the keyframes; the classifier takes the output of the target language model and the graph module as input.
[0038] In some embodiments, the method further includes:
[0039] Obtain the video data to be labeled, and extract features from the video data to be labeled;
[0040] Based on the extracted features, the inter-frame similarity of the video data to be labeled is obtained;
[0041] Based on the inter-frame similarity, the video frames included in the video data to be labeled are clustered using the first clustering method to obtain the initial clustering result;
[0042] The cluster centers in the initial clustering result are used as the initial cluster centers of the second clustering method, and the initial clustering result is optimized using the second clustering method to obtain the target clustering result;
[0043] Determine the cluster center of each class in the target clustering result, and take the video frame in each class that is closest to the cluster center as the key frame to be labeled.
[0044] On the other hand, a training device for a video annotation model is provided, the device comprising:
[0045] The first acquisition module is configured to acquire multiple tag sets; wherein, one of the tag sets includes a category tag that describes the video content;
[0046] The first processing module is configured to transform the multiple label sets into multiple views based on the graph generation model in the graph module; wherein, the multiple views use category labels as nodes to describe the relationship between category labels from different perspectives;
[0047] The second processing module is configured to optimize the graph structure of the multi-view based on the graph convolutional layer and graph pooling layer in the graph module to obtain a prediction graph.
[0048] The second acquisition module is configured to acquire a reference graph of the multiple tag sets based on the target language model;
[0049] The training module is configured to train the video annotation model based on the predicted graph and the reference graph; wherein the video annotation model includes the graph module and the target language model.
[0050] In some embodiments, the first acquisition module is configured to:
[0051] Acquire sample video data;
[0052] Target detection is performed on video frames in the sample video data; a set of person class tags is generated based on the detected targets;
[0053] Perform action recognition on the detected targets; generate a set of action class labels based on the recognized actions;
[0054] Perform emotion recognition on the detected targets; generate a set of emotion category labels based on the recognized emotions.
[0055] In some embodiments, the first processing module is configured to:
[0056] The category labels included in the multiple label sets are used as nodes of the multiple views;
[0057] For the nth view in the multi-view, based on the Gaussian graph generation model in the graph module, the initial node representation of each node in the nth view is encoded as a Gaussian distribution;
[0058] In the nth view, the weight of the edge between node i and node j is determined based on the Gaussian distribution corresponding to node i and the Gaussian distribution corresponding to node j.
[0059] The edges of the multi-view are initialized based on the weights of the edges determined in each view;
[0060] Where n, i, and j are all positive integers; a pair of nodes in the multi-view are connected by multiple edges, and one edge corresponds to one view in the multi-view.
[0061] In some embodiments, the second processing module is configured to:
[0062] During each iteration, the node representation of each node in each view is updated through the current graph convolutional layer in the form of message propagation;
[0063] Based on the updated node representation of the multi-view, the attention score set of each view in the multi-view is obtained through the current graph pooling layer; wherein, one of the attention score sets includes the attention weight of each node in the view;
[0064] Based on the attention score set of each view, determine the nodes that should be retained in each view; determine the joint set of nodes that should be retained in each view as the nodes of the graph created in this graph pooling process;
[0065] The prediction graph is generated based on the nodes of the intermediate graph created during multiple graph pooling processes.
[0066] In some embodiments, the second acquisition module is configured to:
[0067] Based on external knowledge and a self-encoding language model, the multiple tag sets are transformed into the reference graph;
[0068] The external knowledge includes triples, which are used to represent the relationships between entities.
[0069] In some embodiments, the training module is configured as follows:
[0070] Construct the target loss function;
[0071] Based on the target loss function, the error value between the predicted map and the reference map is obtained;
[0072] The error value is propagated backward from the output layer of the video annotation model to the hidden layer, until it reaches the input layer of the video annotation model;
[0073] During backpropagation, the network parameters of the video annotation model are adjusted according to the error value until the video annotation model converges.
[0074] In some embodiments, the video annotation model further includes a classifier; the application process of the video annotation model includes:
[0075] The keyframes to be labeled are input into the video labeling model to obtain semantic labels output by the classifier; wherein, the semantic labels are used to describe the semantic relationships between the category labels extracted in the keyframes; the classifier takes the output of the target language model and the graph module as input.
[0076] In some embodiments, the application process of the video annotation model further includes:
[0077] Obtain the video data to be labeled, and extract features from the video data to be labeled;
[0078] Based on the extracted features, the inter-frame similarity of the video data to be labeled is obtained;
[0079] Based on the inter-frame similarity, the video frames included in the video data to be labeled are clustered using the first clustering method to obtain the initial clustering result;
[0080] The cluster centers in the initial clustering result are used as the initial cluster centers of the second clustering method, and the initial clustering result is optimized using the second clustering method to obtain the target clustering result;
[0081] Determine the cluster center of each class in the target clustering result, and take the video frame in each class that is closest to the cluster center as the key frame to be labeled.
[0082] On the other hand, a computer device is provided, the device including a processor and a memory, the memory storing at least one piece of program code, the at least one piece of program code being loaded and executed by the processor to implement the above-described training method for the video annotation model.
[0083] On the other hand, a computer-readable storage medium is provided, wherein at least one piece of program code is stored in the storage medium, the at least one piece of program code being loaded and executed by a processor to implement the above-described training method for the video annotation model.
[0084] On the other hand, a computer program product or computer program is provided, which includes computer program code stored in a computer-readable storage medium. A processor of a computer device reads the computer program code from the computer-readable storage medium and executes the computer program code, causing the computer device to perform the above-described training method for the video annotation model.
[0085] This application provides a novel video annotation model training scheme that enables high-level semantic annotation of videos based on a trained video annotation model. Specifically, the model training process involves obtaining multiple label sets, each including a category label describing the video content. Then, based on the graph generation model in the graph module, the multiple label sets are transformed into multiple views, where each view uses category labels as nodes to describe the relationships between category labels from different perspectives. Next, based on the graph convolutional layer and graph pooling layer in the graph module, the graph structure of the multiple views is optimized to obtain a prediction graph. Additionally, this scheme also obtains reference graphs for multiple label sets based on a target language model. Finally, the video annotation model is trained based on the predicted graphs and the reference graphs.
[0086] Because this scheme models and analyzes the correlations between different category labels in the video during training, and maps the category labels to a higher-level semantic classification network through a graph neural network, advanced semantic annotation can be achieved based on the trained video annotation model. This allows researchers to gain a deeper understanding and analysis of the video based on advanced semantic annotation; in other words, this video annotation method can provide rich information for subsequent intelligent decision-making. Attached Figure Description
[0087] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0088] Figure 1 This is a schematic diagram of the implementation environment involved in a training method for a video annotation model provided in an embodiment of this application;
[0089] Figure 2 This is a flowchart of a training method for a video annotation model provided in an embodiment of this application;
[0090] Figure 3 This is a schematic diagram of the structure of a video annotation model provided in an embodiment of this application;
[0091] Figure 4 This is a schematic diagram of the structure of a training device for a video annotation model provided in an embodiment of this application;
[0092] Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation
[0093] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.
[0094] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items that have essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor does it limit the quantity or execution order. It should also be understood that although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by the terms.
[0095] These terms are simply used to distinguish one element from another. For example, without departing from the various examples, the first element can be referred to as the second element, and similarly, the second element can be referred to as the first element. Both the first and second elements can be elements, and in some cases, they can be separate and distinct elements.
[0096] "At least one" refers to one or more elements. For example, at least one element can be one element, two elements, three elements, or any integer number of elements greater than or equal to one. "Multiple" refers to two or more elements. For example, multiple elements can be two elements, three elements, or any integer number of elements greater than or equal to two.
[0097] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.
[0098] Figure 1 This is a schematic diagram of the implementation environment involved in a training method for a video annotation model provided in an embodiment of this application.
[0099] See Figure 1 The implementation environment includes a model training device 101 and a video annotation device 102. Exemplarily, in this embodiment, the video annotation device refers to a recorder.
[0100] The model training device 101 is used to train the video annotation model, that is, to execute the training method of the video annotation model provided in the embodiments of this application; the video annotation device 102 is used to annotate the video based on the trained video annotation model.
[0101] The model training device 101 and the video annotation device 102 are computer devices with machine learning capabilities. In some embodiments, the model training device 101 and the video annotation device 102 may be the same device, or they may be different devices. For example, when the model training device 101 and the video annotation device 102 are different devices, the model training device 101 may be a fixed computer device such as a personal computer or server, and the video annotation device 102 may be a mobile computer device, such as a smart wearable device; this application does not impose any limitations on this. Alternatively, when the model training device 101 and the video annotation device 102 are the same device, they may be smart wearable devices; this application also does not impose any limitations on this.
[0102] The following describes the application scenarios of the training method for the video annotation model provided in the embodiments of this application.
[0103] As a type of smart wearable device, dash cams are not only numerous but also record massive amounts of video data. In practical applications, the video data recorded by dash cams can suffer from quality issues due to environmental factors and user characteristics, such as severe shaking, making it difficult for humans to accurately analyze and process the video data. Therefore, how to enable dash cams to automatically analyze and process the recorded video data has become a pressing issue that needs to be addressed.
[0104] One video annotation method in related technologies is to classify videos into simple tag items, that is, to add a few simple category tags to the video. This video annotation method cannot achieve higher semantic understanding and annotation.
[0105] A graph is a data structure that models a set of objects (nodes) and their relationships (edges). Graph analysis is applied to non-Euclidean data structures in machine learning, focusing on tasks such as node classification, link prediction, and clustering. Graph neural networks, a deep learning-based method operating on graph domains, have become a widely used graph analysis approach.
[0106] This application's embodiments model and analyze the correlation between different category labels in videos. By using a graph neural network to map the category labels to a higher-level semantic classification network, advanced semantic annotation can be achieved based on the trained video annotation model. This allows relevant personnel to gain a deeper understanding and analysis of videos based on advanced semantic annotation; in other words, this video annotation method can provide rich information for subsequent intelligent decision-making.
[0107] For example, the recorder may be applied to traffic scenarios involving autonomous driving or assisted driving, or to security scenarios involving video surveillance, and this application does not impose any limitations.
[0108] Figure 2 This is a flowchart illustrating a training method for a video annotation model provided in an embodiment of this application. The method is executed by a computer device, such as... Figure 1 The model training equipment in the middle.
[0109] In the embodiments of this application, the training process of the video annotation model includes, but is not limited to, the following steps: designing the graph structure, designing classes and class labels, designing the graph neural network, designing the loss function, and model training.
[0110] Designing graph structures refers to identifying graph structures by analyzing application scenarios and problems. Graph structures are divided into structured and unstructured scenarios. In structured scenarios, the graph structure is explicit in the application, such as in molecular applications, physical system applications, and knowledge graphs. In unstructured scenarios, the graph structure is implicit in the application and needs to be constructed from the task, such as building a word graph for text or a scene graph for images.
[0111] Current methods for video annotation primarily focus on learning frame-level features or category labels, neglecting higher-level semantic understanding and annotation. Advanced semantic annotation allows for better understanding and analysis of videos, providing rich information for subsequent intelligent decision-making. Therefore, this application proposes a novel video annotation model training method. It first extracts basic category labels from video data, and then maps these labels to a higher-level semantic classification network using a graph neural network. In this application, nodes in the graph structure represent the aforementioned category labels, and edges between different nodes represent relationships between different category labels. Furthermore, the graph structure designed in this application does not include self-connections.
[0112] Below, based on Figure 2 This section provides a detailed introduction to the steps involved in designing classes and class labels, design graph neural networks, design loss functions, and model training. See also... Figure 2 The method flow provided in this application embodiment includes:
[0113] 201. A computer device acquires multiple tag sets; wherein one tag set includes a category tag that describes the video content.
[0114] This step corresponds to the design class and class tag mentioned above.
[0115] For example, taking video annotation of video data collected by a recorder as an example, in most scenarios, the focus is usually on human action data in the video, such as pedestrians falling, violent resistance to law enforcement, and drunk driving tests. To effectively interpret this data, this application embodiment sets tags such as "person," "emotion," and "action." Each specific person under the "person" tag corresponds to a category tag. That is, each "person" category tag contains specific tag content such as person, emotion, and action. Accordingly, this application embodiment will obtain three tag sets, respectively... Figure 3 The set of tags for people (U), emotions (V), and actions (W) is shown. In some embodiments, the set of tags for people (U), emotions (V), and actions (W) can be obtained in the following manner:
[0116] The process involves: acquiring sample video data; performing target detection on video frames within the sample video data; generating a set of person-type tags based on the detected targets; performing action recognition on the detected targets; generating a set of action-type tags based on the recognized actions; performing emotion recognition on the detected targets; and generating a set of emotion-type tags based on the recognized emotions. The sample video data can be raw video data collected by a recorder, and the detected targets refer to people.
[0117] 202. The computer device uses the graph generation model in the graph module to transform multiple label sets into multiple views; wherein, the multiple views use category labels as nodes to describe the relationship between category labels from different perspectives.
[0118] In this embodiment, the graph module includes a graph generation model, multiple graph convolutional layers, and multiple graph pooling layers. For example, the graph generation model is a Gaussian graph generation model, and the graph pooling layers are DTWPool (Dynamic Time Warping Pool) layers.
[0119] See Figure 3 The original labels, namely the set of person labels U, the set of emotion labels V, and the set of action labels W, are used to generate multiple views through a Gaussian graph generation model. The graph structure of the multiple views is then optimized through multiple interactions between graph convolutional layers and DTWPool layers. In the person label set U, the person labels are the target nodes; in other words, other nodes are connected around the target nodes. That is, the person labels are the target nodes, and the specific label content such as person, emotion, and action all point to the target nodes.
[0120] In some embodiments, based on the graph generation model in the graph module, multiple label sets are transformed into multiple views, including but not limited to the following methods:
[0121] 2021, using category labels included in multiple label sets as nodes in a multi-view.
[0122] 2022. For the nth view in a multi-view, based on the Gaussian graph generation model in the graph module, the initial node representation of each node in the nth view is encoded as a Gaussian distribution.
[0123] Assuming that a multi-view contains M nodes, then for the nth view, the Gaussian graph generation model encodes the initial node representations of the M nodes in the nth view into M Gaussian distributions.
[0124] 2023. In the nth view, based on the Gaussian distributions corresponding to node i and node j, determine the weight of the edge between node i and node j; based on the edge weights determined in each view, initialize the edges of the multi-view. For example, retain the edge with the largest weight between two nodes.
[0125] For example, this application embodiment models the relationship between edges between different nodes based on the KL divergence between Gaussian distributions, and this application does not impose any limitations on this. Here, n, i, and j are all positive integers; a pair of nodes in this multi-view is connected by multiple edges, and one edge corresponds to one view in the multi-view.
[0126] 203. The computer device optimizes the graph structure of multiple views based on the graph convolutional layer and graph pooling layer in the graph module to obtain the prediction graph.
[0127] In this embodiment, the graph convolutional layer and the DTWPool layer optimize the graph structure of the multiple views through multiple iterative operations. That is, after the graph convolutional layer updates the node representation through message propagation, the DTWPool layer refines the multiple views. In other words, during each iteration, the DTWPool layer adaptively refines the graph using the union set of remaining nodes obtained from different views.
[0128] Furthermore, during each iteration, a graph can be created based on the joint set of remaining nodes obtained from different views. Since the graph convolutional layers and DTWPool layers undergo multiple iterations, a final graph sequence is obtained. The number of graphs in this sequence is the same as the number of DTWPool layers. In some embodiments, to minimize information loss, the node representations of the intermediate graphs created during graph pooling can be connected to obtain the final graph (referred to as the prediction graph in this document), similar to the residual connections in the learned graph.
[0129] In summary, based on the graph convolutional layer and graph pooling layer in the graph module, the graph structure of multiple views is optimized to obtain a predicted graph, including but not limited to the following methods: In each iteration, the node representation of each node in each view is updated via message propagation through the current graph convolutional layer; based on the updated node representations of the multiple views, the attention score set of each view in the multiple views is obtained through the current graph pooling layer; wherein, an attention score set includes the attention weight of each node in a view; based on the attention score set of each view, the nodes that should be retained in each view are determined; the joint set of nodes that should be retained in each view is determined as the nodes of the graph created in this graph pooling process; based on the nodes of the intermediate graphs created in multiple graph pooling processes, a predicted graph is generated. For example, node retention can be based on a TopN mechanism, which is not limited in this application.
[0130] 204. Reference diagram of computer equipment obtaining multiple tag sets based on target language model.
[0131] For example, the target language model is the self-encoding language model BERT (Bidirectional Encoder Representation from Transformers), and this application does not limit it. Figure 3 As shown, the BERT model generates the aforementioned reference graph by incorporating external knowledge based on the already acquired sets of person, action, and emotion labels. This external knowledge originates from the input layer of the BERT model, explicitly embedding entity relationship triples into the input layer to obtain the BERT model's input. Correspondingly, the reference graph for multiple label sets is obtained based on the target language model, including but not limited to the following methods: transforming multiple label sets into a reference graph based on external knowledge and an autoencoder language model; wherein the external knowledge includes triples, which represent the relationships between entities.
[0132] 205. The computer equipment trains a video annotation model based on the prediction graph and the reference graph; the video annotation model includes a graph module, a target language model, and a classifier.
[0133] In some embodiments, training a video annotation model based on a predicted image and a reference image includes: constructing a target loss function; obtaining the error value between the predicted image and the reference image based on the target loss function; backpropagating the error value from the output layer to the hidden layer of the video annotation model, up to the input layer; and adjusting the network parameters of the video annotation model according to the error value during backpropagation until the video annotation model converges. In other words, during model training, the model convergence is determined based on the loss function; if converged, training stops; if not converged, the model's accuracy continues to be improved.
[0134] It should be noted that preserving important information is crucial during graph pooling. Nodes in a graph embed rich contextual information, so summarizing this context into the pooling nodes is highly beneficial. For this purpose, embodiments of this application use SoftDTW to guide the graph pooling operation. That is, the target loss function is SoftDTW. Here, SoftDTW is a differentiable loss function used to find the possible optimal alignment between two sequences of different lengths.
[0135]
[0136] Here, M represents all paths, L1 and L2 are two sequences, and Δ(L1,L2) is a differentiable cost function (such as Euclidean distance). To align the matrix, the inner product calculation calculates the cost sum for a given path.
[0137] In summary, such as Figure 3 As shown, this embodiment of the application captures various possible relationships between category labels by constructing multiple views, and then further optimizes these multiple views to select nodes important for relationship extraction. The final output of the optimization process is concatenated with the output based on the target language model, and then a final relationship extraction is performed to generate high-level semantics, resulting in labels with high-level semantics. For example, labels with high-level semantics can indicate relationships between objects, etc.
[0138] In other embodiments, the video annotation model further includes a classifier; after the video annotation model is trained, it can be applied to video annotation. During application, the keyframes to be annotated are input into the video annotation model, and the semantic labels output by the classifier of the video annotation model are obtained; wherein, the semantic labels are used to describe the semantic relationships between the category labels extracted from the keyframes; and, as... Figure 3 As shown, the classifier of the video annotation model takes the output of the concatenation of the target language model and the graph module as input.
[0139] In other embodiments, the keyframes described above can be extracted in the following manner:
[0140] The process involves acquiring video data to be labeled and extracting features from it. Based on the extracted features, the inter-frame similarity of the video data to be labeled is obtained. Based on the inter-frame similarity, a first clustering method is used to cluster the video frames included in the video data to be labeled, resulting in an initial clustering result. The cluster centers in the initial clustering result are used as the initial cluster centers for a second clustering method, which is then used to optimize the initial clustering result, resulting in a target clustering result. The cluster centers of each class in the target clustering result are determined, and the video frames in each class that are closest to the cluster centers are selected as keyframes to be labeled.
[0141] For example, the first clustering method described above is a keyframe extraction algorithm based on video clustering, and the second clustering method described above is the K-means algorithm. The features of the extracted video frames consist of the information entropy of each image block in the video frame, which is not limited herein.
[0142] This application provides a novel video annotation model training scheme that enables high-level semantic annotation of videos based on a trained video annotation model. Because the scheme models and analyzes the correlations between different category labels in the video during training, and maps the category labels to a higher-level semantic classification network through a graph neural network, high-level semantic annotation can be achieved based on the trained video annotation model. This allows relevant personnel to gain a deeper understanding and analysis of videos based on high-level semantic annotation; in other words, this video annotation method can provide rich information for subsequent intelligent decision-making.
[0143] Figure 4 This is a schematic diagram of the structure of a training device for a video annotation model provided in an embodiment of this application. See also... Figure 4 The device includes:
[0144] The first acquisition module 401 is configured to acquire multiple tag sets; wherein, one of the tag sets includes a category tag that describes the video content;
[0145] The first processing module 402 is configured to transform the multiple label sets into multiple views based on the graph generation model in the graph module; wherein, the multiple views use category labels as nodes to describe the relationship between category labels from different perspectives;
[0146] The second processing module 403 is configured to optimize the graph structure of the multi-view based on the graph convolutional layer and graph pooling layer in the graph module to obtain a prediction graph.
[0147] The second acquisition module 404 is configured to acquire a reference graph of the multiple tag sets based on the target language model;
[0148] Training module 405 is configured to train the video annotation model based on the predicted graph and the reference graph; wherein the video annotation model includes the graph module and the target language model.
[0149] This application provides a novel video annotation model training scheme that enables high-level semantic annotation of videos based on a trained video annotation model. Specifically, the model training process involves obtaining multiple label sets, each including a category label describing the video content. Then, based on the graph generation model in the graph module, the multiple label sets are transformed into multiple views, where each view uses category labels as nodes to describe the relationships between category labels from different perspectives. Next, based on the graph convolutional layer and graph pooling layer in the graph module, the graph structure of the multiple views is optimized to obtain a prediction graph. Additionally, this scheme also obtains reference graphs for multiple label sets based on a target language model. Finally, the video annotation model is trained based on the predicted graphs and the reference graphs.
[0150] Because this scheme models and analyzes the correlations between different category labels in the video during training, and maps the category labels to a higher-level semantic classification network through a graph neural network, advanced semantic annotation can be achieved based on the trained video annotation model. This allows researchers to gain a deeper understanding and analysis of the video based on advanced semantic annotation; in other words, this video annotation method can provide rich information for subsequent intelligent decision-making.
[0151] In some embodiments, the first acquisition module is configured to:
[0152] Acquire sample video data;
[0153] Target detection is performed on video frames in the sample video data; a set of person class tags is generated based on the detected targets;
[0154] Perform action recognition on the detected targets; generate a set of action class labels based on the recognized actions;
[0155] Perform emotion recognition on the detected targets; generate a set of emotion category labels based on the recognized emotions.
[0156] In some embodiments, the first processing module is configured to:
[0157] The category labels included in the multiple label sets are used as nodes of the multiple views;
[0158] For the nth view in the multi-view, based on the Gaussian graph generation model in the graph module, the initial node representation of each node in the nth view is encoded as a Gaussian distribution;
[0159] In the nth view, the weight of the edge between node i and node j is determined based on the Gaussian distribution corresponding to node i and the Gaussian distribution corresponding to node j.
[0160] The edges of the multi-view are initialized based on the weights of the edges determined in each view;
[0161] Where n, i, and j are all positive integers; a pair of nodes in the multi-view are connected by multiple edges, and one edge corresponds to one view in the multi-view.
[0162] In some embodiments, the second processing module is configured to:
[0163] During each iteration, the node representation of each node in each view is updated through the current graph convolutional layer in the form of message propagation;
[0164] Based on the updated node representation of the multi-view, the attention score set of each view in the multi-view is obtained through the current graph pooling layer; wherein, one of the attention score sets includes the attention weight of each node in the view;
[0165] Based on the attention score set of each view, determine the nodes that should be retained in each view; determine the joint set of nodes that should be retained in each view as the nodes of the graph created in this graph pooling process;
[0166] The prediction graph is generated based on the nodes of the intermediate graph created during multiple graph pooling processes.
[0167] In some embodiments, the second acquisition module is configured to:
[0168] Based on external knowledge and an autoencoder language model, the multiple tag sets are transformed into the reference graph; wherein, the external knowledge includes triples, which are used to represent the relationships between entities.
[0169] In some embodiments, the training module is configured as follows:
[0170] Construct the target loss function;
[0171] Based on the target loss function, the error value between the predicted map and the reference map is obtained;
[0172] The error value is propagated backward from the output layer of the video annotation model to the hidden layer, until it reaches the input layer of the video annotation model;
[0173] During backpropagation, the network parameters of the video annotation model are adjusted according to the error value until the video annotation model converges.
[0174] In some embodiments, the video annotation model further includes a classifier; the application process of the video annotation model includes:
[0175] The keyframes to be labeled are input into the video labeling model to obtain semantic labels output by the classifier; wherein, the semantic labels are used to describe the semantic relationships between the category labels extracted in the keyframes; the classifier takes the output of the target language model and the graph module as input.
[0176] In some embodiments, the application process of the video annotation model further includes:
[0177] Obtain the video data to be labeled, and extract features from the video data to be labeled;
[0178] Based on the extracted features, the inter-frame similarity of the video data to be labeled is obtained;
[0179] Based on the inter-frame similarity, the video frames included in the video data to be labeled are clustered using the first clustering method to obtain the initial clustering result;
[0180] The cluster centers in the initial clustering result are used as the initial cluster centers of the second clustering method, and the initial clustering result is optimized using the second clustering method to obtain the target clustering result;
[0181] Determine the cluster center of each class in the target clustering result, and take the video frame in each class that is closest to the cluster center as the key frame to be labeled.
[0182] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.
[0183] It should be noted that the video annotation model training device provided in the above embodiments is only illustrated by the division of the above functional modules when training the video annotation model. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the video annotation model training device and the video annotation model training method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.
[0184] Figure 5 A structural block diagram of a computer device 500 provided in an exemplary embodiment of this application is shown. The computer device 500 may be a portable mobile terminal.
[0185] Typically, computer device 500 includes a processor 501 and a memory 502.
[0186] Processor 501 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 501 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 501 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 501 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 501 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.
[0187] The memory 502 may include one or more computer-readable storage media, which may be non-transitory. The memory 502 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 502 are used to store at least one program code, which is executed by the processor 501 to implement the training method for the video annotation model provided in the method embodiments of this application.
[0188] In some embodiments, the computer device 500 may also optionally include a peripheral device interface 503 and at least one peripheral device. The processor 501, memory 502, and peripheral device interface 503 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 503 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 504, a display screen 505, a camera assembly 506, an audio circuit 507, a positioning assembly 508, and a power supply 509.
[0189] Peripheral device interface 503 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 501 and memory 502. In some embodiments, processor 501, memory 502 and peripheral device interface 503 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 501, memory 502 and peripheral device interface 503 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.
[0190] The radio frequency (RF) circuit 504 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 504 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 504 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 504 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 504 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 504 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.
[0191] Display screen 505 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 505 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 501 for processing. In this case, display screen 505 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 505, disposed on the front panel of computer device 500; in other embodiments, there may be at least two display screens, disposed on different surfaces of computer device 500 or in a folded design; in still other embodiments, display screen 505 may be a flexible display screen, disposed on a curved or folded surface of computer device 500. Furthermore, display screen 505 may be configured as a non-rectangular irregular shape, i.e., a non-rectangular screen. Display screen 505 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).
[0192] The camera assembly 506 is used to acquire images or videos. Optionally, the camera assembly 506 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 506 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.
[0193] The audio circuit 507 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 501 for processing, or input to the radio frequency circuit 504 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located in a different part of the computer device 500. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert the electrical signals from the processor 501 or the radio frequency circuit 504 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 507 may also include a headphone jack.
[0194] The positioning component 508 is used to locate the current geographical location of the computer device 500 in order to enable navigation or LBS (Location Based Service).
[0195] Power supply 509 is used to supply power to the various components in computer device 500. Power supply 509 can be AC power, DC power, a disposable battery, or a rechargeable battery. When power supply 509 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, and a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.
[0196] In some embodiments, the computer device 500 further includes one or more sensors 510. The one or more sensors 510 include, but are not limited to: an accelerometer 511, a gyroscope 512, a pressure sensor 513, a fingerprint sensor 514, an optical sensor 515, and a proximity sensor 516.
[0197] Accelerometer 511 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by computer device 500. For example, accelerometer 511 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 501 can control display screen 505 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 511. Accelerometer 511 can also be used for games or for acquiring user motion data.
[0198] The gyroscope sensor 512 can detect the orientation and rotation angle of the computer device 500. The gyroscope sensor 512, in conjunction with the accelerometer sensor 511, can collect 3D motion data from the user on the computer device 500. Based on the data collected by the gyroscope sensor 512, the processor 501 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.
[0199] The pressure sensor 513 can be disposed on the side bezel of the computer device 500 and / or on the lower layer of the display screen 505. When the pressure sensor 513 is disposed on the side bezel of the computer device 500, it can detect the user's grip signal on the computer device 500, and the processor 501 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 513. When the pressure sensor 513 is disposed on the lower layer of the display screen 505, the processor 501 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 505. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.
[0200] The fingerprint sensor 514 is used to collect the user's fingerprint. The processor 501 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 514, or the fingerprint sensor 514 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as trusted, the processor 501 authorizes the user to perform relevant sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings. The fingerprint sensor 514 can be located on the front, back, or side of the computer device 500. When the computer device 500 has physical buttons or a manufacturer's logo, the fingerprint sensor 514 can be integrated with the physical buttons or manufacturer's logo.
[0201] An optical sensor 515 is used to collect ambient light intensity. In one embodiment, the processor 501 can control the display brightness of the display screen 505 based on the ambient light intensity collected by the optical sensor 515. Specifically, when the ambient light intensity is high, the display brightness of the display screen 505 is increased; when the ambient light intensity is low, the display brightness of the display screen 505 is decreased. In another embodiment, the processor 501 can also dynamically adjust the shooting parameters of the camera assembly 506 based on the ambient light intensity collected by the optical sensor 515.
[0202] The proximity sensor 516, also known as a distance sensor, is typically located on the front panel of the computer device 500. The proximity sensor 516 is used to detect the distance between the user and the front of the computer device 500. In one embodiment, when the proximity sensor 516 detects that the distance between the user and the front of the computer device 500 is gradually decreasing, the processor 501 controls the display screen 505 to switch from a screen-on state to a screen-off state; when the proximity sensor 516 detects that the distance between the user and the front of the computer device 500 is gradually increasing, the processor 501 controls the display screen 505 to switch from a screen-off state to a screen-on state.
[0203] Those skilled in the art will understand that Figure 5 The structure shown does not constitute a limitation on the computer device 500, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.
[0204] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including program code that can be executed by a processor in a computer device to complete the training method of the video annotation model in the above embodiments. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.
[0205] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer program code stored in a computer-readable storage medium. A processor of a computer device reads the computer program code from the computer-readable storage medium and executes the computer program code, causing the computer device to perform the training method of the video annotation model described above.
[0206] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.
[0207] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. A training method for a video annotation model, characterized in that, The method includes: Obtain multiple tag sets; wherein, one of the tag sets includes a category tag that describes the video content; Based on the graph generation model in the graph module, the multiple label sets are transformed into multiple views; wherein, the multiple views use category labels as nodes to describe the relationship between category labels from different perspectives; Based on the graph convolutional layer and graph pooling layer in the graph module, the graph structure of the multi-view is optimized to obtain the prediction graph; A reference graph for the multiple tag sets is obtained based on the target language model; The video annotation model is trained based on the predicted graph and the reference graph; wherein the video annotation model includes the graph module, the target language model, and a classifier; the classifier takes the output of the concatenated target language model and the graph module as input; The method further includes: Obtain the video data to be labeled, and extract features from the video data to be labeled; Based on the extracted features, the inter-frame similarity of the video data to be labeled is obtained; Based on the inter-frame similarity, the video frames included in the video data to be labeled are clustered using the first clustering method to obtain the initial clustering result; The cluster centers in the initial clustering result are used as the initial cluster centers of the second clustering method, and the initial clustering result is optimized using the second clustering method to obtain the target clustering result; Determine the cluster center of each class in the target clustering result, and take the video frame in each class that is closest to the cluster center as the key frame to be labeled; The keyframes to be labeled are input into the video labeling model to obtain semantic labels output by the classifier; wherein, the semantic labels are used to describe the semantic relationships between the category labels extracted from the keyframes.
2. The method according to claim 1, characterized in that, The process of obtaining multiple tag sets includes: Acquire sample video data; Target detection is performed on video frames in the sample video data; a set of person class tags is generated based on the detected targets; Perform action recognition on the detected targets; generate a set of action class labels based on the recognized actions; Perform emotion recognition on the detected targets; generate a set of emotion category labels based on the recognized emotions.
3. The method according to claim 1, characterized in that, The graph generation model based on the graph module transforms the multiple label sets into multiple views, including: The category labels included in the multiple label sets are used as nodes of the multiple views; For the nth view in the multi-view, based on the Gaussian graph generation model in the graph module, the initial node representation of each node in the nth view is encoded as a Gaussian distribution; In the nth view, the weight of the edge between node i and node j is determined based on the Gaussian distribution corresponding to node i and the Gaussian distribution corresponding to node j. The edges of the multi-view are initialized based on the weights of the edges determined in each view; Where n, i, and j are all positive integers; a pair of nodes in the multi-view are connected by multiple edges, and one edge corresponds to one view in the multi-view.
4. The method according to claim 1, characterized in that, The optimization of the graph structure of the multi-view based on the graph convolutional layer and graph pooling layer in the graph module to obtain the prediction graph includes: During each iteration, the node representation of each node in each view is updated through the current graph convolutional layer in the form of message propagation; Based on the updated node representation of the multi-view, the attention score set of each view in the multi-view is obtained through the current graph pooling layer; wherein, one of the attention score sets includes the attention weight of each node in the view; Based on the attention score set of each view, determine the nodes that should be retained in each view; determine the joint set of nodes that should be retained in each view as the nodes of the graph created in this graph pooling process; The prediction graph is generated based on the nodes of the intermediate graph created during multiple graph pooling processes.
5. The method according to claim 1, characterized in that, The reference graph for obtaining the multiple tag sets based on the target language model includes: Based on external knowledge and a self-encoding language model, the multiple tag sets are transformed into the reference graph; The external knowledge includes triples, which are used to represent the relationships between entities.
6. The method according to claim 1, characterized in that, The step of training the video annotation model based on the predicted map and the reference map includes: Construct the target loss function; Based on the target loss function, the error value between the predicted map and the reference map is obtained; The error value is propagated backward from the output layer of the video annotation model to the hidden layer, until it reaches the input layer of the video annotation model; During backpropagation, the network parameters of the video annotation model are adjusted according to the error value until the video annotation model converges.
7. A training device for a video annotation model, characterized in that, The device includes: The first acquisition module is configured to acquire multiple tag sets; wherein, one of the tag sets includes a category tag that describes the video content; The first processing module is configured to transform the multiple label sets into multiple views based on the graph generation model in the graph module; wherein, the multiple views use category labels as nodes to describe the relationship between category labels from different perspectives; The second processing module is configured to optimize the graph structure of the multi-view based on the graph convolutional layer and graph pooling layer in the graph module to obtain a prediction graph. The second acquisition module is configured to acquire a reference graph of the multiple tag sets based on the target language model; The training module is configured to train the video annotation model based on the predicted graph and the reference graph; wherein the video annotation model includes the graph module, the target language model, and a classifier; the classifier takes the output of the concatenation of the target language model and the graph module as input; The application process of the video annotation model includes: Obtain the video data to be labeled, and extract features from the video data to be labeled; Based on the extracted features, the inter-frame similarity of the video data to be labeled is obtained; Based on the inter-frame similarity, the video frames included in the video data to be labeled are clustered using the first clustering method to obtain the initial clustering result; The cluster centers in the initial clustering result are used as the initial cluster centers of the second clustering method, and the initial clustering result is optimized using the second clustering method to obtain the target clustering result; Determine the cluster center of each class in the target clustering result, and take the video frame in each class that is closest to the cluster center as the key frame to be labeled; The keyframes to be labeled are input into the video labeling model to obtain semantic labels output by the classifier; wherein, the semantic labels are used to describe the semantic relationships between the category labels extracted from the keyframes.
8. A computer device, characterized in that, The device includes a processor and a memory, the memory storing at least one line of program code, the at least one line of program code being loaded and executed by the processor to implement the training method for the video annotation model as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The storage medium stores at least one piece of program code, which is loaded and executed by a processor to implement the training method of the video annotation model as described in any one of claims 1 to 6.
10. A computer program product or computer program, characterized in that, The computer program product or computer program includes computer program code stored in a computer-readable storage medium, a processor of a computer device reads the computer program code from the computer-readable storage medium, and the processor executes the computer program code, causing the computer device to perform the training method for the video annotation model as described in any one of claims 1 to 6.