Video labeling method and device, equipment and medium

Abstract

The invention discloses a video labeling method and device, equipment and a medium. The method comprises the steps of obtaining a to-be-labeled original video, and preprocessing the original video to obtain an image frame sequence; for each image frame in the image frame sequence, entity structured data of each entity contained in the image frame is extracted to obtain an entity recognition sequence, and the entity recognition sequence comprises a structured data set corresponding to each image frame; recognizing a scene label corresponding to each image frame and a behavior state corresponding to each contained entity in combination with each structured data set in the entity recognition sequence to obtain a space-time knowledge graph; and generating a video annotation file in combination with the entity recognition sequence and the space-time knowledge graph. Therefore, manual annotation is not needed, the annotation cost is reduced, the video annotation efficiency is improved, the entity association relationship between different image frames is expressed by combining the space-time knowledge graph constructed by the structured data sets of the multiple image frames, the accuracy of the video annotation data is improved, and the quality of the video annotation data is ensured.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a video annotation method, apparatus, device and medium. Background Technology

[0002] Video data, as a dynamic record of the real world, carries continuous spatiotemporal behavioral patterns, environmental changes, and multimodal interaction information, serving as a crucial entry point for artificial intelligence (AI) models to understand complex physical laws and human actions. Video annotation, through structured analysis of video content (such as target category recognition, target action trajectory annotation, and action temporal type annotation), provides learnable foundational data for downstream task models.

[0003] In related technologies, video data is labeled frame by frame manually, or a single model is used to detect and label the video, providing learnable basic data for downstream task models.

[0004] However, manual annotation methods in related technologies are inefficient and costly. Furthermore, the subjective bias inherent in manual annotation can lead to inconsistent annotation data. On the other hand, automatic annotation methods based on single-frame detection can result in false or missed detections in video frames that are far away, occluded, or dynamically blurred, thereby reducing the accuracy and quality of the video annotation data. Summary of the Invention

[0005] This application provides a video annotation method, apparatus, device, and medium that can improve the annotation efficiency of video data, enhance the accuracy of video annotation data, and ensure the quality of video annotation data.

[0006] Firstly, this application provides a video annotation method, including: Obtain the original video to be labeled, and preprocess the original video to obtain an image frame sequence; For each image frame in the image frame sequence, the entity structured data of each entity contained in the image frame is extracted to obtain an entity recognition sequence, wherein the entity recognition sequence contains the structured dataset corresponding to each image frame; By combining each structured dataset in the entity recognition sequence, the scene label corresponding to each image frame and the behavioral state corresponding to each entity contained therein are identified, thus obtaining a spatiotemporal knowledge graph; By combining the entity recognition sequence and the spatiotemporal knowledge graph, a video annotation file is generated.

[0007] Secondly, this application provides a video annotation device, comprising: The acquisition unit is used to acquire the original video to be labeled and preprocess the original video to obtain an image frame sequence; The extraction unit is used to extract the entity structured data of each entity contained in each image frame in the image frame sequence to obtain an entity recognition sequence, wherein the entity recognition sequence contains the structured dataset corresponding to each image frame. The construction unit is used to combine each structured dataset in the entity recognition sequence to identify the scene label corresponding to each image frame and the behavioral state corresponding to each entity contained therein, so as to obtain a spatiotemporal knowledge graph. The generation unit is used to combine the entity recognition sequence and the spatiotemporal knowledge graph to generate a video annotation file.

[0008] In some embodiments, the extraction unit is further configured to: The category is used to obtain the entity recognition result for each entity in each image frame; For each image frame in the image frame sequence, pixel-level instance segmentation is performed on each entity within the image frame to obtain the target mask information corresponding to each entity in each image frame; For each image frame in the image frame sequence, pose recognition is performed on the target entity within the image frame to obtain the key point coordinates corresponding to each entity in each image frame. Based on the entity bounding box, entity category label, target mask information and key point coordinates corresponding to each entity in each image frame, the entity structured data corresponding to each entity in each image frame is obtained, and the entity recognition sequence is constructed by combining the entity structured data corresponding to each entity in each image frame.

[0009] In some embodiments, the extraction unit is further configured to: For each image frame in the image frame sequence, feature extraction processing is performed on the image frame to obtain the initial image features corresponding to each image frame; A preset query vector is obtained, and the preset query vector and the initial image features corresponding to each image frame are calculated through a cross-attention mechanism to obtain the intermediate image features corresponding to each image frame. Based on the intermediate image features corresponding to each image frame, the entity bounding box and entity category label corresponding to each entity are identified.

[0010] In some embodiments, the extraction unit is further configured to: The convolutional neural network is used to perform feature extraction processing on each image frame in the image frame sequence to obtain the underlying visual features corresponding to each image frame; A multi-scale feature map is obtained by decoding the underlying visual features using a pixel-level decoder. Obtain a preset entity query vector, input the preset entity query vector and the multi-scale feature map into the adaptive decoder to perform multi-layer cross-attention calculation, and obtain at least one entity category and an entity mask vector associated with each entity category; A high-resolution feature map is generated based on the multi-scale feature map, and a dot product operation is performed between each entity mask vector and the high-resolution feature map to obtain the target mask information corresponding to each entity in each image frame.

[0011] In some embodiments, the extraction unit is further configured to: Each image frame in the image frame sequence is input into a high-resolution network; Each image frame is represented by a high-resolution backbone network in the high-resolution network to obtain high-resolution features corresponding to each image frame. Each image frame is also represented by a low-resolution branch network in parallel with the high-resolution backbone network to obtain low-resolution features corresponding to each image frame. The high-resolution features and low-resolution features corresponding to each image frame are then fused to obtain high-resolution fused features corresponding to each image frame. Obtain predefined key point information, and generate a heatmap corresponding to each preset key point by combining the high-resolution fusion feature and the predefined key point information through a convolutional network. The heatmap contains the pixel value corresponding to each key pixel. The coordinates of key points are determined by combining the pixel values ​​of each key pixel in each heatmap.

[0012] In some embodiments, the building unit is further configured to: Based on each structured dataset in the entity recognition sequence, the behavioral state corresponding to each entity contained in each image frame and the scene label corresponding to each image frame are identified. By combining the scene label corresponding to each image frame and the behavior state corresponding to each entity contained therein, an event node corresponding to each image frame is constructed; Determine the node temporal relationship between multiple event nodes corresponding to multiple image frames, and calculate the causal weight between any two temporally adjacent event nodes according to the node temporal relationship; Based on the temporal relationship between the nodes and the causal weight between any two temporally adjacent event nodes, two temporally adjacent event nodes are connected in series to obtain a spatiotemporal knowledge graph.

[0013] In some embodiments, the generating unit is further configured to: The entity recognition sequence and the spatiotemporal knowledge graph are verified using labeled data to obtain the verification results. When the verification result is successful, the entity recognition sequence and the spatiotemporal knowledge graph are identified as verified labeled data. The verified annotation data is encapsulated to obtain a video annotation file.

[0014] In some embodiments, the generating unit is further configured to: The Monte Carlo algorithm is used to determine the first confidence level of each structured dataset in the entity recognition sequence and the second confidence level of the spatiotemporal knowledge graph. The entity recognition sequence is cross-referenced with the spatiotemporal knowledge graph to obtain the comparison result. When both the first confidence level and the second confidence level are greater than or equal to the preset confidence threshold, and the information between the entity recognition sequence and the spatiotemporal knowledge graph is determined to be consistent based on the comparison result, the verification result is determined to be successful. When either the first confidence level or the second confidence level is less than the preset confidence threshold, or when the comparison result determines that the information between the entity recognition sequence and the spatiotemporal knowledge graph is inconsistent, the verification result is determined to be verification failure.

[0015] In some embodiments, the generating unit is further configured to: When the verification result is that the verification fails, the abnormal time node corresponding to the abnormal data in the target data is determined, and the target data is at least one of the entity recognition sequence and the spatiotemporal knowledge graph. At least one target image frame from other time nodes adjacent to the abnormal time node is acquired from the original video; Entity recognition is performed on the target image frame to obtain the target structured dataset corresponding to the target image frame, and the target behavior state corresponding to each entity is identified based on the target structured dataset; The target structured dataset and the target behavior state are validated to obtain the validation results; When the verification result is that the target structured dataset and the target behavior state pass the verification, the verification result is determined to be a successful verification.

[0016] Furthermore, this application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described video annotation method.

[0017] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute the aforementioned video annotation method.

[0018] In this embodiment, the original video to be labeled is first preprocessed to obtain an image frame sequence. Then, the entity structured data of each entity contained in each image frame in the image frame sequence is extracted to obtain an entity recognition sequence. The entity recognition sequence contains a structured dataset corresponding to each image frame. In this way, each entity in each image frame is labeled to represent each image frame through each structured dataset. Next, the scene label corresponding to each image frame and the behavior state corresponding to each entity contained in each image frame are identified by combining each structured dataset in the entity recognition sequence to construct a spatiotemporal knowledge graph. In this way, the spatiotemporal knowledge graph is used to represent the relationship between entities between different image frames, avoiding the problem of false detection or missed detection in the background of image frames with long distance, occlusion, or dynamic blur caused by single image frame detection. Finally, the entity recognition sequence and the spatiotemporal knowledge graph are combined to generate a video annotation file.

[0019] As can be seen from the above, compared to the inefficient and costly manual annotation methods in related technologies, which suffer from inconsistencies due to subjective biases, and the single-model automatic annotation method which relies on single image frame detection, thus reducing the accuracy and quality of video annotation data, this application extracts entity structured data from the image frames corresponding to the original video and combines it with each structured dataset in the entity recognition sequence to extract the scene label and behavioral state of each entity corresponding to each image frame, thereby constructing a spatiotemporal knowledge graph. Then, the video annotation file is generated by combining the entity recognition sequence and the spatiotemporal knowledge graph. This eliminates the need for manual annotation, reduces annotation costs, improves video annotation efficiency, and the spatiotemporal knowledge graph constructed by combining structured datasets from multiple image frames to represent the entity relationships between different image frames effectively avoids false detections or missed detections in distant, occluded, or dynamically blurred image frames, improving the accuracy of video annotation data and ensuring its quality. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1This is a schematic diagram of a video annotation system provided in an embodiment of this application; Figure 2 This is a flowchart illustrating the steps of the video annotation method provided in the embodiments of this application; Figure 3 This is an example diagram of the video annotation process provided in the embodiments of this application; Figure 4 Example diagrams of video annotation scenarios provided in embodiments of this application; Figure 5 This is a schematic diagram of the structure of the video annotation device provided in the embodiments of this application; Figure 6 This is a schematic diagram of the terminal structure provided in the embodiments of this application; Figure 7 This is a partial structural block diagram of the server provided in an embodiment of this application. Detailed Implementation

[0022] To enable those skilled in the art to better understand the solutions of this application, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, other embodiments obtained by those skilled in the art without creative effort are all within the scope of protection of this application.

[0023] It is understood that in the specific implementation of this application, data such as original videos, image frame sequences, entity recognition sequences, structured datasets, spatiotemporal knowledge graphs, and video annotation files are involved. When the above embodiments of this application are applied to specific products or technologies, permission or consent from the subject is required, and the collection, use, and processing of related data must comply with relevant laws, regulations, and standards.

[0024] Furthermore, when this application embodiment needs to obtain relevant data, it will obtain separate permission or separate consent for the original video, image frame sequence, entity recognition sequence, structured dataset, spatiotemporal knowledge graph, video annotation file, and other related data through pop-up windows or redirection to a confirmation page. Only after clearly obtaining separate permission or separate consent for the original video, image frame sequence, entity recognition sequence, structured dataset, spatiotemporal knowledge graph, video annotation file, and other related data will it obtain the necessary data for enabling the application embodiment to operate normally.

[0025] It should be noted that while some processes described in the specification, claims, and accompanying drawings include multiple steps appearing in a specific order, it should be clearly understood that these steps may not be performed in the order they appear herein, or may be performed in parallel. The step numbers are merely used to distinguish different steps and do not represent any execution order. Furthermore, descriptions such as "first," "second," or "objective" in this document are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, other embodiments obtained by those skilled in the art without creative effort are all within the scope of protection of this application.

[0027] This application provides a video annotation method, apparatus, device, and medium. Specifically, the spatiotemporal game-based collaborative decision-making method of this application can be implemented in a computer device, which can be a server or a terminal device. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The user terminal device can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, smart home appliance, vehicle terminal, intelligent voice interaction device, aircraft, drone, etc., but is not limited to these.

[0028] For ease of understanding, this application will describe the implementation process of the video annotation method through multiple embodiments, as follows: This application provides a video annotation method. First, the original video to be annotated is preprocessed to obtain an image frame sequence. Then, from each image frame in the image frame sequence, the structured data of each entity contained therein is extracted to obtain an entity recognition sequence. This entity recognition sequence contains a structured dataset corresponding to each image frame. Thus, by performing basic annotation on each entity within each image frame, each image frame is represented by a structured dataset. Next, by combining each structured dataset from the entity recognition sequence, the scene label corresponding to each image frame and the behavioral state corresponding to each entity are identified to construct a spatiotemporal knowledge graph. This spatiotemporal knowledge graph represents the relationships between entities in different image frames, avoiding false detections or missed detections caused by single-image-frame detection in distant, occluded, or dynamically blurred image frames. Finally, the entity recognition sequence and the spatiotemporal knowledge graph are combined to generate a video annotation file. Please refer to the following specific embodiments for details.

[0029] It should be noted that this video annotation method can be executed by the terminal device alone, by the server alone, or by both the terminal and the server together.

[0030] For example, the process can be executed independently by a terminal device, which may have a target application (client) installed on it for video annotation. Specifically, in executing the video annotation method, the terminal can acquire the original video to be annotated and preprocess it to obtain an image frame sequence. For each image frame in the image frame sequence, the entity structured data of each entity contained within the image frame is extracted to obtain an entity recognition sequence, which contains a structured dataset corresponding to each image frame. Combining each structured dataset in the entity recognition sequence, the scene label corresponding to each image frame and the behavioral state corresponding to each entity contained therein are identified to obtain a spatiotemporal knowledge graph. Combining the entity recognition sequence and the spatiotemporal knowledge graph, a video annotation file is generated.

[0031] For example, if executed by the server alone, the server can retrieve the original video to be labeled from the database and preprocess the original video to obtain an image frame sequence; for each image frame in the image frame sequence, extract the entity structured data of each entity contained in the image frame to obtain an entity recognition sequence, which contains the structured dataset corresponding to each image frame; combine each structured dataset in the entity recognition sequence to identify the scene label corresponding to each image frame and the behavioral state corresponding to each entity contained therein, to obtain a spatiotemporal knowledge graph; combine the entity recognition sequence and the spatiotemporal knowledge graph to generate a video annotation file.

[0032] For example, it can be executed jointly by the terminal and the server, see [link to relevant documentation]. Figure 1This is a schematic diagram of a video annotation system provided in an embodiment of this application. The system includes a terminal 110 and a server 120.

[0033] The terminal 110 may have a target application, i.e., a client, installed on it, which is used to send the original video to be labeled to the server 120. The server 120 may be a single service node, a distributed system composed of multiple service nodes, or a service node in a distributed system.

[0034] Specifically, the server 120 executes the following steps for the video annotation method: It acquires the original video to be annotated and preprocesses it to obtain an image frame sequence; for each image frame in the image frame sequence, it extracts the entity structured data of each entity contained within the image frame to obtain an entity recognition sequence, which contains the structured dataset corresponding to each image frame; combining each structured dataset in the entity recognition sequence, it identifies the scene label corresponding to each image frame and the behavioral state corresponding to each entity contained therein, thus obtaining a spatiotemporal knowledge graph; and combining the entity recognition sequence and the spatiotemporal knowledge graph, it generates a video annotation file.

[0035] Therefore, compared to the inefficient and costly manual annotation methods in related technologies, which suffer from inconsistencies due to subjective biases and reliance on single-frame automatic annotation (which reduces accuracy and quality), this application extracts structured entity data from the image frames of the original video and combines it with the structured dataset from the entity recognition sequence to extract scene labels and behavioral states for each image frame, thus constructing a spatiotemporal knowledge graph. This is then combined with the entity recognition sequence and the spatiotemporal knowledge graph to generate a video annotation file. This eliminates the need for manual annotation, reduces annotation costs, and improves video annotation efficiency. Furthermore, the spatiotemporal knowledge graph, constructed by combining structured datasets from multiple image frames, effectively avoids false or missed detections in distant, occluded, or dynamically blurred image frames, improving the accuracy and quality of the video annotation data.

[0036] To facilitate understanding, the steps of the video annotation method will be explained in detail below. It should be noted that the order of the following embodiments is not intended to limit the preferred order of the embodiments.

[0037] See Figure 2 , Figure 2This is a flowchart illustrating the steps of the video annotation method provided in this embodiment. In this embodiment, the video annotation method can be executed by a computer device, such as a terminal acting as the first industrial device, to execute the video annotation method. The specific process is as follows: 101. Obtain the original video to be labeled, and preprocess the original video to obtain an image frame sequence.

[0038] Video data, as a dynamic record of the real world, carries continuous spatiotemporal behavioral patterns, environmental changes, and multimodal interaction information, serving as a crucial entry point for artificial intelligence (AI) models to understand complex physical laws and human actions. Video annotation, through structured analysis of video content (such as target category recognition, target action trajectory annotation, and action temporal type annotation), provides learnable foundational data for downstream task models.

[0039] However, related technologies either manually annotate video data frame by frame or use a single model to detect and annotate videos, providing learnable foundational data for downstream task models. However, manual annotation is inefficient and costly, and subjective biases can lead to inconsistent annotation data. Meanwhile, single-model automatic annotation relies on single-frame detection, which can result in false or missed detections in distant, occluded, or dynamically blurred video frames, reducing the accuracy and quality of the annotated data and causing a sharp drop in the recognition accuracy of downstream task models.

[0040] To address the above issues, this embodiment first preprocesses the original video to be labeled to obtain an image frame sequence. Then, it extracts the entity structured data of each entity contained in each image frame from the image frame sequence to obtain an entity recognition sequence. This entity recognition sequence contains a structured dataset corresponding to each image frame. In this way, by performing basic annotation on each entity in each image frame, each image frame is represented by each structured dataset. Next, by combining each structured dataset in the entity recognition sequence, the scene label corresponding to each image frame and the behavioral state corresponding to each entity are identified to construct a spatiotemporal knowledge graph. In this way, the spatiotemporal knowledge graph is used to represent the relationship between entities in different image frames, avoiding the problem of false detection or missed detection in the background of distant, occluded, or dynamically blurred image frames caused by single image frame detection. Finally, the entity recognition sequence and the spatiotemporal knowledge graph are combined to generate a video annotation file. In this way, manual annotation is not required, reducing annotation costs and improving video annotation efficiency. Furthermore, the spatiotemporal knowledge graph constructed by combining structured datasets of multiple image frames to represent entity relationships between different image frames effectively avoids false detections or missed detections in the context of distant, occluded, or dynamically blurred image frames, thereby improving the accuracy of video annotation data and ensuring the quality of video annotation data.

[0041] In this embodiment, to improve the accuracy of downstream task models in recognizing video data, the original video to be labeled can be obtained. This original video contains entity content information that needs to be identified or detected. Furthermore, to facilitate subsequent labeling of entity content in the original video, the original video needs to be preprocessed. Specifically, the original video can be segmented into frames to obtain an image frame sequence containing multiple image frames. This allows for subsequent entity recognition of multiple image frames in the image frame sequence, thereby combining the entity recognition results to further construct entity association relationships between different video frames, generating labeled data, and improving the accuracy of downstream task models in recognizing video data.

[0042] The original video can be the original video to be labeled, which can be acquired by a camera component. For example, it can be a recorded video file or real-time video stream data; there is no limitation here. For example, taking a target detection scenario at a border or coastline as an example, in order to enhance target detection in the border or coastline environment, video stream data acquired by a camera component can be obtained to get the original video corresponding to the border or coastline.

[0043] In some implementations, in order to reduce the amount of data processing and improve the efficiency of video annotation, the step 101 of "preprocessing the original video to obtain an image frame sequence" may include: determining the target frame rate, performing adoption processing on the original video according to the target frame rate, and obtaining an image frame sequence containing multiple image frames.

[0044] The image frame can be a corresponding video frame from the original video. The image frame contains relevant entities, such as mountains, beaches, trees, seawater, people, blue sky, white clouds, kites, huts, sentry posts, etc. These are just examples. If the original video is a movie, the image frame can contain entities such as main characters, supporting characters, buildings, vehicles, etc. Each independent object can be understood as an entity.

[0045] For example, after obtaining the original video to be labeled, which can be raw video stream data, such as MP4, AVI, or other video files or real-time video streams, the original video can be decoded. Specifically, the encoded and compressed video data stream (such as MP4 or AVI format) is restored to a continuous, uncompressed sequence of original image frames. Further, from this sequence, an image frame sequence containing multiple image frames is obtained by sampling at a fixed frame rate (e.g., 25 FPS) or using an adaptive sampling strategy (based on scene change detection). These image frames are arranged chronologically. Thus, in preprocessing, using a fixed frame rate or an adaptive sampling strategy means selecting a subset of image frames from all frames of the video stream and arranging them in chronological order. For example, assuming the original video is 60 FPS, sampling at 25 FPS is equivalent to selecting one image frame every two frames. This reduces the amount of data processing in the video labeling process, fundamentally improving the efficiency of video labeling.

[0046] The above methods can be used to obtain the original video that needs to be labeled. This original video contains entity content information that needs to be identified or detected. Further, the original video needs to be preprocessed, specifically by splitting the original video into frames to obtain an image frame sequence containing multiple image frames. This allows for subsequent entity recognition of multiple image frames in the image frame sequence. The entity recognition results are then combined to further construct the entity association relationship between different video frames, generate labeled data, and improve the accuracy of downstream task models in recognizing video data.

[0047] 102. For each image frame in the image frame sequence, extract the entity structured data of each entity contained in the image frame to obtain the entity recognition sequence.

[0048] In this embodiment, after obtaining an image frame sequence containing multiple image frames, entity recognition can be performed on each image frame in the image frame sequence to obtain entity recognition results. Specifically, for each image frame, the entity structured data of each entity is extracted from the image frame. This entity structured data can represent the entity corresponding to the current image frame. Thus, an entity recognition sequence is obtained, which is the entity recognition result corresponding to the image frame sequence. The entity recognition sequence contains a structured dataset corresponding to each image frame, and each structured dataset contains the entity structured data of each entity in the corresponding image frame. In this way, the entity situation corresponding to the image frame sequence is represented by the structured dataset corresponding to each image frame contained in the entity recognition sequence. This allows for the subsequent establishment of a spatiotemporal knowledge graph representing the entity relationships between image frames, based on the structured dataset corresponding to each image frame contained in the entity recognition sequence, thereby completing the annotation of the video data.

[0049] It should be noted that an entity recognition sequence can be a sequence of recognition results for multiple image frames in an image frame sequence. The recognition result of each image frame is represented by a structured data. Thus, the entity recognition sequence contains a structured dataset corresponding to each image frame. This structured dataset is used to represent the entity content information of the corresponding image frame, such as the entity category label, location (boundary box), target mask information, key point coordinates, etc. of all entities in the image frame. The structured dataset corresponding to each image frame contains the entity structured data of each entity in the corresponding image frame. That is, the structured dataset of an image frame is composed of multiple entity structured data corresponding to multiple entities in the image frame.

[0050] Each entity's structured data may include the corresponding entity's bounding box, category label, target mask information, and / or key point coordinates. For example, if the entity is a person, it includes a bounding box indicating the person's position in the image frame. This bounding box is used to select and cover the outline of the person. The shape of the bounding box can be a rectangle, a circle, a triangle, etc. Taking a rectangle as an example, the bounding box of the person can be represented by at least two coordinate points, thus indicating the position of the person in the image frame. It also includes the category label of the person, which can be "person", "human figure", or a number representing the person. For example, "1" can be used to represent the category label of the person entity, and "2", "3", etc. can be used to represent the category labels of other entities. As for the mask information, it is the mask information for the person. Specifically, it can be represented by a binary mask. For example, within the image frame, the outline of the person is used as the mask boundary. The value inside the outline of the person is "1", and the value outside the outline of the person is "0", thus forming the target mask information. The person's limbs, head, and center point can be used as key points, and the coordinates of these parts are the key point coordinates. The above are just examples.

[0051] In some implementations, for each image frame in the image frame sequence, the bounding box, category label, target mask information, and key point coordinates of each entity in the image frame are identified to obtain the entity structured data corresponding to each entity. An entity recognition sequence is then constructed by combining the entity structured data corresponding to each entity in each image frame. For example, the entity structured data includes entity recognition results, target mask information, and key point coordinates. Step 102 may include: (102.1) For each image frame in the image frame sequence, identify the entity bounding box and entity category label corresponding to each entity within the image frame; (102.2) For each image frame in the image frame sequence, perform pixel-level instance segmentation on each entity within the image frame to obtain the target mask information corresponding to each entity in each image frame; (102.3) For each image frame in the image frame sequence, perform pose recognition on the target entity within the image frame to obtain the key point coordinates corresponding to each entity in each image frame; (102.4) Based on the entity bounding box, entity category label, target mask information and key point coordinates corresponding to each entity in each image frame, the entity structured data corresponding to each entity in each image frame is obtained, and the entity recognition sequence is constructed by combining the entity structured data corresponding to each entity in each image frame.

[0052] Specifically, firstly, for each image frame in the image frame sequence, a target detection algorithm (Detection Transformer, DETR) based on an adaptive encoder or decoder (Transformer) architecture is used to extract local feature maps, which are then processed through a compression channel. The adaptive encoder combines the features processed by the compression channel with the positional encoding to generate global context features. Then, the decoder decodes based on the global context features and the received preset query vector to identify the entity category label and entity bounding box of each entity in each image frame. The entity bounding box can be represented in coordinate form.

[0053] Then, for each image frame in the image frame sequence, each entity is segmented at the pixel level to obtain an "entity contour mask," i.e., target mask information. Specifically, for each image frame in the image frame sequence, low-level visual features are extracted and subjected to multiple decoding processes by a pixel decoder. The pixel decoder uses a top-down structure and lateral connections to fuse high-resolution detail features with low-resolution semantic features to generate a multi-scale feature map. Then, an adaptive decoder combines the multi-scale feature map with the received object query vector for decoding. This allows the object query vector to focus on each entity region in the multi-scale feature map through a cross-attention mechanism, obtaining entity category labels and corresponding pixel-level mask embeddings (i.e., vectors) to obtain the high-resolution feature map corresponding to the current image frame. Thus, by combining the pixel-level mask embeddings and the high-resolution feature map, the target mask information corresponding to each entity in the current image frame is determined. For example, taking entity "1" as an example, the target mask information can be represented as follows: the entity identifier is "entity_id":1, and the target mask information corresponding to entity "1" is represented as "mask_info":[0,0,...,1,1,...,0]. The array length of the target mask information is equal to the total number of pixels in the image, such as 1920×1080, depending on the actual situation.

[0054] Furthermore, for "human-like entities" in image frames, the coordinates of skeletal key points (such as head, hands, feet, and joints) are extracted, corresponding to the "key point annotation task" of the basic annotation agent in the solution. For non-human entities (such as drones and vehicles), this step can be skipped, and the key point coordinate field can be left blank or labeled "none". Specifically, for each image frame in the image frame sequence, high-resolution features are extracted, and multiple low-resolution features of different resolutions are extracted step by step. The high-resolution features interact with multiple low-resolution features to provide accurate localization, while the low-resolution features supplement global context information, resulting in fused features. Then, the fused features are combined with predefined key point pairs to generate a heatmap. The heatmap contains the pixel value of each key point, and the key point coordinates corresponding to each entity are determined based on the pixel value of each key point in the heatmap.

[0055] Finally, for each image frame, by combining the entity bounding box, entity category label, target mask information, and keypoint coordinates corresponding to each entity within the image frame, the structured entity data corresponding to each entity within that image frame is obtained. Combined with this structured entity data, a structured dataset corresponding to that image frame is constructed. For example, assuming the image frame is frame 101, the structured dataset would be as follows: { "frame_id": 101, / / 101st image frame "frame_timestamp": "00:04:01.2", / / Timestamp of the 101st image frame "entity_structured_datas": [ / / Structured datasets { "entity_id": 1, / / Entity "1" "class_label": "person", / / The entity category label is "human". "bounding_box": [120, 350, 80, 150], / / Entity marker box "mask_info": [0,0,...,1,1,...,0], / / Target mask information "key_points": [{"name":"head","coord": [160,360],...}] / / Key point coordinates }, { "entity_id": 2, / / Entity "2" "class_label": "drone", the entity category label is "drone". "bounding_box": [480, 220, 25, 20], / / Entity marker box "mask_info": [0,0,...,1,1,...,0], / / Target mask information "key_points": "None" / / Keypoint coordinates are "None" } ]} Furthermore, an entity recognition sequence is constructed by combining multiple structured datasets corresponding to multiple image frames. These structured datasets are ordered within the entity recognition sequence according to the temporal relationships between the image frames. This process completes entity recognition for each image frame, obtaining a structured dataset representing each entity within that frame. This dataset represents the entity information corresponding to the image frame sequence. This information is then used to further establish a spatiotemporal knowledge graph representing the entity relationships between image frames, based on the structured datasets corresponding to each image frame included in the entity recognition sequence. This completes the annotation of the original video data.

[0056] In some implementations, by combining a preset query vector and each image frame, the entity bounding box and entity category label corresponding to each entity within each image frame are identified. For example, step (102.1) may include: performing feature extraction processing on each image frame in the image frame sequence to obtain initial image features corresponding to each image frame; obtaining a preset query vector, and calculating the preset query vector and the initial image features corresponding to each image frame using a cross-attention mechanism to obtain intermediate image features corresponding to each image frame; and identifying the entity bounding box and entity category label corresponding to each entity based on the intermediate image features corresponding to each image frame.

[0057] The preset query vector can be a feature template used to indicate entities in an image frame, indicating key features in the image features to identify the corresponding entity. For example, the extracted initial image features can be considered as a "book," and the preset query vector can be considered as "keywords," such as "feature keywords of a cat (cat ear shape, whiskers, body shape, etc.)." The model can locate the "cat" content in the "book" based on these keywords. This is a illustrative example. In practice, the preset query vector is a feature template used by the model to identify and recognize entities in an image frame. The model learns the ability to recognize corresponding entities after training. Before model training, the query vector can be randomly initialized. During training, the values ​​in the query vector are continuously adjusted through backpropagation to ultimately match the corresponding entity features. For example, there can be preset query vectors specifically used to match entity features such as "person," "car," "cat," "dog," "sea," "tree," "cloud," and "aircraft." During the model inference stage, one or more preset query vectors can be input, each corresponding to a type of entity, to find different categories of entities in the image frame.

[0058] It should be noted that when performing entity recognition on each image frame in the image frame sequence, an object detection algorithm (Detection Transformer, DETR) based on an adaptive encoder or decoder (Transformer) architecture is used. The encoder-decoder structure extracts global features of the image and can receive a set of learnable preset query vectors (object queries). A fixed number of predicted boxes are generated based on the learnable preset query vectors. Finally, a bipartite graph matching algorithm (Hungarian algorithm) is used to align the predicted boxes with the real target boxes one by one, realizing parallel entity detection without redundancy.

[0059] First, for each image frame, a convolutional neural network (CNN) is used as the backbone network to extract local feature maps (e.g., dimensions H×W×d, where H is the height, W is the width, and d is the number of feature channels) based on the image frame. Then, the local feature maps are compressed to d dimensions through 1×1 convolutions and flattened into a sequence form. and add position encoding This location encoding is used to supplement spatial information. Subsequently, it is input into the Transformer encoder, where the location encoding provides spatial information for different locations. The Transformer encoder actively performs self-attention to establish global contextual relationships in Tuxi Town, generating global contextual features. The initial image features are represented as follows;

[0060] Then, a preset query vector is obtained, and the initial image features corresponding to each image frame are calculated using a cross-attention mechanism to obtain the intermediate image features corresponding to each image frame. For example, a set of learnable preset query vectors, called "object queries," is received, denoted as... , This is the preset maximum number of entities to be detected. This refers to the dimension of the preset query vectors. Each preset query vector, after training, will be responsible for predicting a specific target (entity). Through a cross-attention mechanism, the preset query vectors interact with the initial image features corresponding to each image frame. Feature computation is performed, with each preset query vector focusing on a specific region in the initial image features through the cross-attention mechanism, thereby outputting intermediate image features, as shown below:

[0061] Among them, the intermediate image features corresponding to each image frame ,Include Information about the predicted entity.

[0062] Finally, the intermediate image features are processed using two fully connected networks. One fully connected network performs classification based on the intermediate image features to output an entity category label for each entity, such as "person", "drone", and "car". The other fully connected network, belonging to the regression branch, outputs an entity bounding box for each entity based on the intermediate image features, in the format [x, y, w, h], where x and y are the top-left pixel coordinates, and w and h are the width and height. Assume the predicted category label is... The real label is ,in, It is the number of actual entities in the image frame. The missing parts are filled with empty categories of "no object". Furthermore, the Hungarian algorithm (bipartite graph matching) is used to match the predicted results one-to-one with the true labels (or candidate entities in unsupervised scenarios), filtering out "no object" category labels and retaining the entity bounding boxes and entity category labels of valid entities.

[0063] In some implementations, for each image frame in the image frame sequence, an entity mask vector and a high-resolution feature map corresponding to each entity category are determined by combining a preset entity query vector. The target mask information corresponding to each entity in each image frame is then determined based on the entity mask vector and the high-resolution feature map. For example, step (102.2) may include: performing feature extraction processing on each image frame in the image frame sequence using a convolutional neural network to obtain the low-level visual features corresponding to each image frame; performing decoding processing based on the low-level visual features using a pixel-level decoder to obtain a multi-scale feature map; obtaining a preset entity query vector; inputting the preset entity query vector and the multi-scale feature map into an adaptive decoder for multi-layer cross-attention calculation to obtain at least one entity category and an entity mask vector associated with each entity category; generating a high-resolution feature map based on the multi-scale feature map, and performing a dot product operation between each entity mask vector and the high-resolution feature map to obtain the target mask information corresponding to each entity in each image frame.

[0064] The preset entity query vector can be a feature template used to indicate entities in an image frame, indicating key features in the image features to identify the corresponding entity category, so as to determine the entity mask vector corresponding to that entity category. For a description of the preset entity query vector, please refer to the previous section on "Preset Query Vectors". During the model inference stage, one or more preset query vectors can be input, each corresponding to a type of entity, to find entities of different categories in the image frame.

[0065] It should be noted that the Mask2Former model can be used. This model integrates a CNN backbone network, a pixel decoder, and a Transformer decoder as a pixel-level segmentation model architecture. Through this architecture, each entity in each image frame is segmented pixel-level, resulting in an "entity contour mask" for each entity—that is, the target mask information for each entity—to determine the pixel range of each entity in the image frame. Furthermore, this Transformer decoder is used for instance segmentation, semantic segmentation, and panoptic segmentation, unifying these three tasks into a single "mask classification" task.

[0066] Specifically, first, each image frame is input into the backbone of a convolutional neural network (CNN) (such as ResNet or SwinTransformer) to extract the low-level visual features corresponding to each image frame. Then, the low-level visual features output by the backbone network are input into a pixel decoder. This pixel decoder, through a top-down structure and lateral connections, fuses high-resolution detail information and low-resolution semantic information to generate a series of multi-scale feature maps with rich context.

[0067] Furthermore, a set of learnable preset object query vectors (N) is introduced, the number of which is the preset maximum number of detectable entities within a single image frame. Then, this set of preset object query vectors, along with multi-scale feature maps, is input into a Transformer decoder. This Transformer decoder is used to perform instance segmentation, semantic segmentation, and panoptic segmentation on the multi-scale feature maps through a mask classification task. Specifically, through a cross-attention mechanism, the preset object query vectors interact with the multi-scale feature maps corresponding to each image frame for feature computation. Each preset query vector focuses on the entity region in the multi-scale feature map through the cross-attention mechanism. After processing by multiple Transformer decoders, each preset object query vector ultimately outputs an entity category and a pixel-level mask embedding (i.e., an entity mask vector) associated with each entity category through a linear classification layer.

[0068] Finally, the multi-scale feature maps are fused to obtain a high-resolution feature map, or the feature map with the largest size and richest details from the multi-scale feature maps is selected as the high-resolution feature map. Then, the entity mask vector corresponding to each entity category is multiplied by the high-resolution feature map to obtain the feature operation result. An activation function (such as sigmoid) is used to process the feature operation result to generate the final binary instance mask, i.e., the target mask information. In this way, the Mask2Former model can generate a set of prediction pairs containing <category label, target mask information> for each image frame of the original video, thereby accurately segmenting the contour of each entity and providing fine spatial location information for subsequent high-level semantic understanding (such as occlusion relationships between objects, interaction behaviors).

[0069] For example, the Mask2Former model is used. The Mask2Former CNN backbone network extracts low-level visual features from each image frame. These low-level visual features are then input into the pixel decoder. Through a top-down structure and lateral connections, high-resolution detail features and low-resolution semantic features are fused to generate a multi-scale feature map. Furthermore, a set of "learnable preset entity query vectors" corresponding to entity detection (the number of preset entity query vectors is consistent with that of the detection module) is introduced, along with the multi-scale features. Figure 1 The same input Transformer decoder is used. Each preset entity query vector focuses on a single entity region through a cross-attention mechanism, outputting "entity category" and "pixel-level mask embedding". Finally, the pixel-level mask embedding is multiplied by the highest resolution feature map and a binarized instance mask is generated by the Sigmoid activation function (pixels with a mask value of 1 belong to the entity, and 0 indicates that they do not belong to the entity). The target mask information corresponding to each entity is output.

[0070] In some implementations, for each image frame in the image frame sequence, high-resolution features and low-resolution features are extracted, and the high-resolution features and low-resolution features are fused to obtain the high-resolution fused features corresponding to the current image frame. A heatmap is generated by combining predefined key point information and the high-resolution fused features, and the key point coordinates are determined based on the pixel values ​​in the heatmap. For example, step (102.3) may include: inputting each image frame in the image frame sequence into a high-resolution network; performing feature representation on each image frame through the high-resolution backbone network in the high-resolution network to obtain the high-resolution features corresponding to each image frame, and performing feature representation on each image frame through a low-resolution branch network parallel to the high-resolution backbone network to obtain the low-resolution features corresponding to each image frame, and fusing the high-resolution features and low-resolution features corresponding to each image frame to obtain the high-resolution fused features corresponding to each image frame; obtaining predefined key point information, and generating a heatmap corresponding to each preset key point by combining the high-resolution fused features and the predefined key point information through a convolutional network, wherein the heatmap contains the pixel values ​​corresponding to each key pixel; and determining the key point coordinates by combining the pixel values ​​of each key pixel contained in each heatmap.

[0071] It should be noted that for "human-like entities" in the image frame, the coordinates of key points of the skeleton (such as head, hands, feet, joints) are extracted, which corresponds to the "key point annotation task" of the basic annotation agent in the solution; non-human entities (such as drones, vehicles) can skip this step, and the key point coordinate field can be left blank or labeled "none".

[0072] For example, a high-resolution network (HRNet) is employed to maintain high-resolution feature representation and cross-scale fusion throughout the process. Specifically, each image frame is input into the HRNet. Starting with the high-resolution backbone, low-resolution branches are added in parallel step by step, maintaining high-resolution feature channels throughout. Specifically, the high-resolution backbone in the HRNet performs feature representation on each image frame to obtain high-resolution features for each image frame, and multiple low-resolution branch networks running in parallel with the high-resolution backbone perform feature representation on each image frame to obtain multiple low-resolution features of different resolutions for each image frame. Furthermore, at different stages of the network, the high-resolution features of the high-resolution branches and the low-resolution features of the low-resolution branches exchange and fuse information. In this way, the high-resolution features can continuously receive global contextual information from the low-resolution features, while the low-resolution features can also obtain precise localization information of the high-resolution features, resulting in high-resolution fused features for each image frame.

[0073] Furthermore, a convolutional layer is used to predict the heatmap corresponding to each predefined keypoint (such as head, shoulder, wrist, knee, etc.). For example, the predefined keypoint information includes multiple predefined keypoints, such as head, shoulder, wrist, knee, etc. Based on high-resolution fusion features, the predefined keypoint information is combined to generate a heatmap. The value of each pixel on the heatmap represents the confidence that the location is the corresponding keypoint. It should be noted that each keypoint category has its own dedicated heatmap. For example, assuming 17 keypoints are predefined (such as nose, left eye, right eye, left shoulder, right shoulder, etc.), the final output of the model will contain 17 independent heatmap channels. The first image is only responsible for labeling "nose"; the brightest spot on the image is the location of the nose. The second image is only responsible for labeling "left eye," completely ignoring the location of the nose. And so on, the heatmaps do not interfere with each other. The model outputs multiple layers of heatmaps; that is, there are as many heatmaps as there are keypoints, and each layer is only responsible for illuminating the location belonging to its category.

[0074] Finally, the heatmap of each key point is post-processed to find the pixel with the highest confidence. For example, the coordinates of this pixel are regarded as the two-dimensional coordinates of the corresponding key point. Typically, a heatmap with Gaussian smoothing is used to locate the peak point with sub-pixel precision, thereby obtaining the accurate coordinates of the key point.

[0075] Using the above method, for each image frame, the structured entity data of each entity is extracted from that image frame. This structured entity data represents the entity corresponding to the current image frame. This yields an entity recognition sequence, which is the entity recognition result corresponding to the image frame sequence. The entity recognition sequence contains a structured dataset corresponding to each image frame, and each structured dataset contains the structured entity data of each entity in that image frame. Thus, by using the structured dataset corresponding to each image frame in the entity recognition sequence, the entity situation corresponding to the image frame sequence is represented. This allows for the subsequent construction of a spatiotemporal knowledge graph representing the entity relationships between image frames, based on the structured dataset corresponding to each image frame in the entity recognition sequence, thereby completing the annotation of the video data.

[0076] 103. By combining each structured dataset in the entity recognition sequence, the scene label corresponding to each image frame and the behavioral state corresponding to each entity contained therein are identified, thus obtaining a spatiotemporal knowledge graph.

[0077] In this embodiment, after obtaining the entity recognition sequence, the scene label corresponding to each image frame and the behavioral state corresponding to each entity can be identified by combining each structured dataset in the entity recognition sequence. Then, event nodes corresponding to each image frame are constructed by combining the scene label and the behavioral state of each entity. Based on the temporal and causal relationships between multiple event nodes, a spatiotemporal knowledge graph is constructed. This allows for the subsequent combination of the entity recognition sequence and the spatiotemporal knowledge graph to generate video annotation data for the original video, providing a data foundation for target detection in downstream task models and improving the accuracy of video data recognition by downstream task models.

[0078] The scene label can be the background label in the corresponding image frame. For example, if a person is flying a kite at the beach, the scene label is "beach". If a person is on a mountain, the scene label is "mountain". If a person is kayaking at sea, the scene label is "sea". These are just examples and are not limited here.

[0079] In this context, "behavioral state" can refer to the physical behavior of a target entity (such as a person or a pet). For example, taking a person as the target entity, suppose the original video records a person flying a kite on a beach. Suppose the process of flying the kite includes three stages: the person picks up the kite, the person flies the kite, and the person retrieves the kite. Then, "the person picks up the kite" can be understood as one behavioral state, "the person flies the kite" can be understood as another behavioral state, and "the person retrieves the kite" can be understood as yet another behavioral state. This is just an example. It should be noted that each behavioral state can be understood as an event.

[0080] The spatiotemporal knowledge graph can be a graph representing the entity state of the original video. This knowledge graph can be generated by connecting multiple event nodes according to temporal and causal relationships, representing the continuous dynamic changes in the behavioral states of the relevant entities corresponding to the original video. This provides data support for subsequent target detection scenarios such as blurring and occlusion, and avoids target detection interruptions during the subsequent target detection process. It should be noted that a behavioral state and its corresponding scene label can be constructed as an event node. For example, the spatiotemporal knowledge graph can be structured data of "entity-behavior-scene-causal relationship", including event nodes, related edges, scene labels, key event chains, etc. (such as the causal chain of "person(1)→control_drone(2)→drone(2)move"). The above is just an example.

[0081] In some implementations, the behavioral state corresponding to each entity in each image frame and the scene label corresponding to each image frame are identified based on each structured dataset in the entity recognition sequence to construct an event node corresponding to each image frame. Multiple event nodes are then concatenated to generate a spatiotemporal knowledge graph. For example, step 103 may include: identifying the behavioral state corresponding to each entity in each image frame and the scene label corresponding to each image frame based on each structured dataset in the entity recognition sequence; constructing an event node corresponding to each image frame by combining the scene label corresponding to each image frame and the behavioral state corresponding to each entity; determining the temporal relationship between multiple event nodes corresponding to multiple image frames, and calculating the causal weight between any two temporally adjacent event nodes according to the temporal relationship; and concatenating two temporally adjacent event nodes according to the temporal relationship and the causal weight between any two temporally adjacent event nodes to obtain a spatiotemporal knowledge graph.

[0082] The timing relationship of this node can be the chronological order of the occurrence of multiple event nodes.

[0083] The causal weight can be a weight representing the correlation between two corresponding event nodes.

[0084] First, based on each structured dataset in the entity recognition sequence, the behavioral state of each entity corresponding to each image frame is identified. Specifically, this can be the behavioral state identification of the target entity "human" or "pet". VideoMAE (Video Masked Autoencoder) is used as the pre-training foundation, learning spatiotemporal feature representations from a large-scale unlabeled video through self-supervised learning, thereby performing spatiotemporal action classification. Based on each structured dataset in the entity recognition sequence, the behavioral state of at least one target entity corresponding to each image frame is identified. Simultaneously, based on CLIP-driven multimodal semantic matching, the visual encoding and text encoding of the original video are obtained respectively. Specifically, a keyframe sampling strategy is adopted, extracting one frame every 2 seconds, dynamically adjusting through scene change detection, and sending the keyframe data into the CLIP image encoder to obtain the visual encoding. Furthermore, through a pre-built text knowledge base, combined with each structured dataset, a global label is assigned to the scene of each image frame, such as scene labels like "daytime", "street", and "rain".

[0085] Then, the scene label corresponding to each image frame and the behavioral state corresponding to each entity are combined and packaged to construct the event node corresponding to each image frame. Furthermore, the temporal relationship between multiple event nodes corresponding to multiple image frames is determined. Specifically, the order of occurrence of the behavioral states (events) corresponding to each event node can be determined to establish the temporal relationship between multiple event nodes based on the order of occurrence.

[0086] Furthermore, causal weights between any two temporally adjacent event nodes are calculated according to their temporal relationships. Specifically, each event node is processed using a graph neural network (GNN) to obtain a context-rich node embedding (i.e., an event node vector), and nodes that are temporally adjacent (e.g., two event nodes) are selected. Event Node If the time difference between them is less than a preset time difference threshold, such as Two event node vectors that are (seconds) and conform to basic logic are represented as ( Therefore, by concatenating or subtracting the two event node vectors, a relation feature vector is formed, represented as... The relation feature vector is then input into a pre-trained small neural network. For example, the relation feature vector can be input into a multilayer perceptron (MLP), which is trained on a large amount of data with causal labels to learn the probability of judging causal association from the semantic representation of events. It can be understood as a causal scoring function. The output is a scalar value in the range of [0, 1]. This scalar value represents the causal strength between the two event nodes, i.e., the causal weight.

[0087] Finally, combining the temporal relationships between nodes and the causal weights between any two temporally adjacent event nodes, the connection path of the event node with the highest cumulative causal weight is selected to form a chain of event nodes, thus obtaining a spatiotemporal knowledge graph. For example, the causal weights between event pairs calculated in the previous step can be used as the starting point for the event nodes. To the event node The directed edge weights are determined, and then a dynamic programming algorithm (such as Bellman-Ford or DP algorithm for directed acyclic graphs) is used to find the path with the largest sum of cumulative causal weights from the starting event to the ending event. This path is called the "key event chain", which represents a series of most likely causal events inferred by the system. Multiple event nodes are connected along this path to form a spatiotemporal knowledge graph.

[0088] By combining the structured dataset in the entity recognition sequence, we can identify the scene label corresponding to each image frame and the behavioral state corresponding to each entity contained therein. Then, by combining the scene label and the behavioral state of each entity, we can construct event nodes for each image frame. Based on the temporal and causal relationships between these event nodes, we can build a spatiotemporal knowledge graph. This allows us to subsequently combine the entity recognition sequence and the spatiotemporal knowledge graph to generate video annotation data for the original video, providing a data foundation for object detection in downstream task models and improving the accuracy of video data recognition by these models.

[0089] 104. Combine entity recognition sequences and spatiotemporal knowledge graphs to generate video annotation files.

[0090] In this embodiment, after obtaining the entity recognition sequence and spatiotemporal knowledge graph, a video annotation file can be directly generated by combining the entity recognition sequence and spatiotemporal knowledge graph. For example, the entity recognition sequence and spatiotemporal knowledge graph can be integrated into a final video annotation file in a standard format (such as COCO, JSON, or XML). Furthermore, to ensure the quality of the video annotation data, the entity recognition sequence and spatiotemporal knowledge graph can be validated before generating the video annotation file. For example, uncertainty quantization can be used to verify the confidence level of the entity recognition sequence and spatiotemporal knowledge graph, and cross-validation can be used to verify whether there are temporal inconsistencies between the entity recognition sequence and spatiotemporal knowledge graph. If the validation result is a failure, it needs to be re-verified and the relevant data regenerated. If the validation result is a success, the video annotation file is directly generated. This ensures the quality of the video annotation data, provides a data foundation for target detection in downstream task models, and improves the accuracy of downstream task models in recognizing video data.

[0091] The video annotation file can be the annotation information itself, or it can include both the original video frames and the corresponding annotation information.

[0092] In some implementations, the entity recognition sequence and the spatiotemporal knowledge graph are validated using labeled data to obtain a validation result. When the validation result is successful, the entity recognition sequence and the spatiotemporal knowledge graph are encapsulated to obtain a video annotation file. For example, step 104 may include: (104.1) The entity recognition sequence and the spatiotemporal knowledge graph are validated using labeled data to obtain the validation results; (104.2) When the verification result is that the verification is passed, the entity recognition sequence and the spatiotemporal knowledge graph are identified as verified labeled data; (104.3) The verified annotation data is encapsulated to obtain the video annotation file.

[0093] Specifically, the entity recognition sequence and spatiotemporal knowledge graph are verified in multiple dimensions, such as by quantifying uncertainty and verifying logical consistency, to obtain the verification results.

[0094] First, verify the quantification of uncertainty. The entity recognition sequence contains a structured dataset corresponding to each image frame. Each structured dataset contains structured data corresponding to each entity. Each structured data contains the entity category label, entity bounding box, target mask information, and key point coordinates for each entity. Based on this, the first confidence of the annotation results is evaluated using the Monte Carlo Dropout algorithm. Specifically, multiple forward inferences (e.g., 100 times) are performed on key image frames in the entity recognition sequence (e.g., frames with dense targets, severe occlusion, or dynamic blur) to statistically analyze the variance of the entity bounding box (bbox) position (e.g., whether the x / y coordinate fluctuation exceeds the ±3 pixel threshold) and to calculate the classification entropy value corresponding to the entity category label for each entity (e.g., the dispersion of the predicted distribution of the entity category label). If the variance is less than or equal to a preset pixel threshold (e.g., ±3 pixels) and the classification entropy value is ≤ a preset classification entropy threshold (e.g., 0.3), the first confidence of the entity recognition sequence is determined according to a preset confidence threshold. If the first confidence is greater than or equal to the preset confidence threshold, the uncertainty quantification is qualified; otherwise, if the first confidence of the entity recognition sequence is less than the preset confidence threshold, the uncertainty quantification is unqualified.

[0095] Second, logical consistency, specifically comparing the logical correlation between the entity recognition sequence and the spatiotemporal knowledge graph to avoid semantic contradictions. Specifically, (1) verify whether the time range of the event chain in the spatiotemporal knowledge graph matches the state change time of the entity in the entity recognition sequence (e.g., whether the event time of "drone moving" matches the position change time of the entity tag box of the drone in the entity recognition sequence). (2) detect whether the entity category label across image frames in the entity recognition sequence matches the entity category of the target entity of the behavior state of the event node in the spatiotemporal knowledge graph. (3) verify semantic correlation, specifically verify whether the scene label (e.g., "sea_surface") in the spatiotemporal knowledge graph logically matches the entity category label (e.g., "drone", "ship") in the entity recognition sequence to avoid the contradiction of "ships appearing in the desert scene"; and perform behavior and key point matching, specifically verify whether the behavior state (e.g., "control_drone") in the spatiotemporal knowledge graph matches the human body key point state in the entity recognition sequence (e.g., whether the hand key point is in a "control" posture). (4) Causal rationality verification: Specifically check whether the logic of the connection edges between event nodes in the spatiotemporal knowledge graph (such as "person controls drone" and "drone moves") conforms to physical laws and is consistent with the interaction characteristics between entities in the entity recognition sequence (such as whether the relative positions of person and drone support the "control" behavior).

[0096] When the above logical consistency check passes, and the first confidence level is greater than or equal to the preset confidence threshold, the verification result is determined to be "verification passed". That is, if "uncertainty quantification is qualified and logical consistency check is consistent", the verification result is "verification passed". Conversely, if "uncertainty quantification is unqualified" or "logical consistency check is contradictory" (such as target position fluctuation exceeding the threshold, behavior not matching key points), the verification result is "verification failed". Finally, when the verification result is "verification passed", the entity recognition sequence and spatiotemporal knowledge graph are determined as verified labeled data, and the verified labeled data is encapsulated to obtain the video labeled file.

[0097] In some implementations, step (104.1) may include: determining a first confidence level for each structured dataset in the entity recognition sequence and a second confidence level for the spatiotemporal knowledge graph using a Monte Carlo algorithm; performing cross-referencing between the entity recognition sequence and the spatiotemporal knowledge graph to obtain a comparison result; determining the verification result as successful when both the first and second confidence levels are greater than or equal to a preset confidence threshold, and the comparison result indicates that the information between the entity recognition sequence and the spatiotemporal knowledge graph is consistent; and determining the verification result as unsuccessful when both the first and second confidence levels are less than the preset confidence threshold, or the comparison result indicates that the information between the entity recognition sequence and the spatiotemporal knowledge graph is inconsistent.

[0098] The description regarding "determining the first confidence level of each structured dataset in the entity recognition sequence and the second confidence level of the spatiotemporal knowledge graph using the Monte Carlo algorithm" is as follows: First, the Monte Carlo algorithm is used to determine the first confidence level of each structured dataset in the entity recognition sequence. Firstly, structured datasets of key image frames are selected from the entity recognition sequence, prioritizing "high-complexity frames" (such as occluded, motion-blurred, or densely packed entity frames), such as frame_id=105 (occluded person) and frame_id=150 (motion-blurred drone). Each structured dataset contains information such as "entity bounding box (bbox), entity category, target mask information (mask), and keypoint coordinates (pose)." Then, perform 100 forward inference iterations for each structured dataset. The specific process is as follows: enable Dropout inference. In the model of the basic labeled agent (DETR entity detection, Mask2Former segmentation, HRNet keypoint annotation), keep the Dropout layer on during the inference phase. In each inference iteration, some neurons will be randomly turned off to simulate the prediction results of "different sub-models". For the entity bounding box coordinates (x, y, w, h) of each entity, calculate the variance of the 100 inference results (e.g., the range of x-coordinate fluctuation and y-coordinate fluctuation). For the pixel confidence of the entity category (e.g., "person", "drone") and the segmented target mask information mask, calculate the classification entropy value of the 100 inference results (the smaller the entropy value, the more concentrated the prediction distribution and the higher the reliability). For the keypoint coordinates of human pose (e.g., head, hand), calculate the positional variance of the 100 inference results (e.g., whether the x / y coordinate fluctuation of the hand keypoint exceeds ±3 pixels). Finally, the confidence score of a single structured dataset is calculated based on the variance and classification entropy values. Specifically, the variance of the detected entity bounding box position is less than or equal to a preset pixel threshold (±3 pixels), the variance of the detected keypoint coordinate position is less than or equal to a preset pixel threshold (±3 pixels), the classification entropy value of the detected category is less than or equal to 0.3, and the pixel confidence entropy of the detected target mask information mask is less than or equal to 0.2. Based on this, the first confidence score is calculated by combining the number of qualified indicators and the total number of indicators. The first confidence score = (number of qualified indicators / total number of indicators). For example, if all 4 indicators are qualified, the first confidence score = 1.0; if 3 indicators are qualified, the first confidence score = 0.75.

[0099] Second, the second confidence level of the spatiotemporal knowledge graph is determined using the Monte Carlo algorithm. Specifically, first, key semantic units are extracted from the spatiotemporal knowledge graph, such as event nodes (e.g., "person controls drone", "drone moves"), causal connection edges (e.g., the causal weight of "person controls drone → drone moves"), and scene labels (e.g., "sea_surface", "daytime"). These are just examples. Then, Monte Carlo Dropout inference and semantic confidence calculation are performed. Specifically, Monte Carlo Dropout inference is executed based on the high-level semantic agent model (VideoMAE behavior recognition, Graph Neural Network (GNN) event chain inference, and CLIP scene matching). For example, on the one hand, for the behavior label (such as "manipulate drone") in the event node, 100 inferences are performed through the VideoMAE model to calculate the classification entropy value of the behavior state prediction. If the entropy value is ≤0.3, the semantic confidence of the node meets the standard. On the other hand, 100 inferences are performed through the GNN model on the causal weights ("0-1 interval scalar values") of the causal connection edges to calculate the variance of the causal weights. If the variance is ≤0.1, the confidence of the causal connection edges meets the standard. In addition, 100 multimodal matchings (visual features → text labels) are performed through the CLIP model on the scene label (such as "sea_surface") to calculate the mean of the label matching probability. If the mean is ≥0.85, the confidence of the scene label meets the standard. Finally, determine the number of key semantic units that meet the criteria, as well as the total number of key semantic units. The second confidence score is calculated as (number of qualified semantic units / total number of semantic units). Assuming there are 5 semantic units, if all 5 semantic units meet the criteria, the second confidence score is 1.0; if 4 meet the criteria, the second confidence score is 0.8. This second confidence score is then directly used as the final second confidence score of the spatiotemporal knowledge graph.

[0100] The description of "cross-compare the entity recognition sequence with the spatiotemporal knowledge graph to obtain the comparison results" is as follows: The core of cross-referencing information is to "verify the semantic and logical consistency of the two," avoiding situations where "precision meets the standard but semantics contradict each other" (such as "a boat appears in a desert scene" or "a person has no control action but a drone control event"). The specific verification logic is as follows: First, verify the consistency of the timing. Specifically, on the one hand, verify whether the time range of the event node matches the time of the entity state change (e.g., whether the time_range [130-160] of the "drone move" event is consistent with the change time of the entity tag box of "drone" in the entity recognition sequence); on the other hand, verify whether the entity category label is continuous (e.g., whether the ID=1 of "person" in the spatiotemporal knowledge graph is consistent with the object_id=1 of "person" in the entity recognition sequence, and whether there is no ID switching error).

[0101] Second, verify semantic relevance. Specifically, verify whether the scene labels (such as "sea_surface") in the spatiotemporal knowledge graph logically match the entity category labels (such as "drone" and "ship") in the entity recognition sequence to avoid the contradiction of "ships appearing in a desert scene"; and perform behavior and key point matching, specifically verify whether the behavior state (such as "control_drone") in the spatiotemporal knowledge graph matches the human body key point state (such as whether the hand key point is in a "controlling" posture) in the entity recognition sequence.

[0102] Third, verify the causal relationship. Specifically, check whether the event nodes corresponding to the causal understanding edges are supported by entity interaction (such as whether the spatiotemporal knowledge graph "person controls drone → drone moves" is consistent with "the relative distance between person and drone is <50 pixels" (physical interactive range) in the entity recognition sequence.

[0103] Furthermore, after performing verification on each comparison dimension, the verification results are recorded, ultimately forming the overall comparison result. Specifically, if there are no contradictions in all three dimensions, such as "the *sea_surface* scene contains a *drone* entity, the *person*'s hand keypoint pose matches the 'manipulation' behavior, and the *drone*'s movement time matches the change time of the corresponding entity's bounding box," then the comparison result is "verification consistent." Conversely, if there is a contradiction in any one dimension, such as "the spatiotemporal knowledge graph labels 'person manipulates *drone*,' but the *person*'s hand keypoint in the entity recognition sequence is in a 'naturally drooping' pose (behavior-keypoint contradiction)," then the comparison result is "verification inconsistent."

[0104] Finally, the verification result is determined by combining the three conditions: "first confidence level, second confidence level, and comparison result." Specifically, if both the first and second confidence levels are greater than or equal to the preset confidence threshold, and the comparison result determines that the information between the entity recognition sequence and the spatiotemporal knowledge graph is consistent, the verification result is determined to be successful. If both the first and second confidence levels are less than the preset confidence threshold, or the comparison result determines that the information between the entity recognition sequence and the spatiotemporal knowledge graph is inconsistent, the verification result is determined to be unsuccessful.

[0105] In some implementations, when the verification result is "verification failed," a review request is triggered to review the entity recognition sequence and / or the spatiotemporal knowledge graph. For example, it may further include: when the verification result is "verification failed," identifying the abnormal time node corresponding to the abnormal data in the target data, where the target data is at least one of the entity recognition sequence and the spatiotemporal knowledge graph; acquiring at least one target image frame from other time nodes adjacent to the abnormal time node in the original video; performing entity recognition on the target image frame to obtain the target structured dataset corresponding to the target image frame, and identifying the target behavior state corresponding to each entity based on the target structured dataset; verifying the target structured dataset and the target behavior state to obtain a verification result; when the verification result indicates that the verification of the target structured dataset and the target behavior state is passed, determining the verification result as "verification passed."

[0106] Specifically, when the verification result is a failure, the abnormal time point where anomalous data exists in the entity recognition sequence and / or spatiotemporal knowledge graph (e.g., the first confidence level is less than a preset confidence threshold, the second confidence level is less than a preset confidence threshold, and / or the comparison result indicates that the information between the entity recognition sequence and the spatiotemporal knowledge graph is inconsistent) can be determined. That is, the time point when the corresponding image frame with anomalous data appears in the original video, such as the 3rd second, the 5th second, etc., which is not limited here. Then, other image frames near the abnormal time point are re-acquired from the original video as target image frames. One or multiple frames can be acquired. Next, through the basic annotation agent, entity recognition is re-performed on the acquired target image frames. The specific recognition process can be referred to the relevant description in "Step 102" above to obtain the target structured dataset corresponding to the target image frame. And, through a high-level semantic agent, the target behavior state corresponding to each entity in the target image frame is identified based on the target structured dataset. The specific details can be referred to the relevant description in "Step 103" above, which will not be elaborated here.

[0107] Furthermore, the target structured dataset and target behavioral state are validated. From both "accuracy" and "semantic logic" perspectives, the reliability of the newly generated "target structured data" and "behavioral state" is verified to detect abnormal data. This validation process is as follows: First, accuracy verification refers to verifying the accuracy of the target structured data. Specifically, a simplified Monte Carlo Dropout algorithm is used to run multiple inference iterations on the "target structured data." For example, the basic labeled agent model is run 50 more times (each time randomly "turning off" some neurons to simulate the judgments of different sub-models). Then, fluctuations are calculated, specifically the fluctuations in the position of the entity bounding boxes (e.g., whether the x / y coordinate fluctuation of the drone's position is ≤3 pixels), and the "dispersion" of entity categories and keypoint coordinates (e.g., whether the category judgments for "human" are concentrated, and whether the fluctuation of the hand keypoint is ≤3 pixels). If the fluctuations are small and all indicators meet the standards (e.g., fluctuation ≤3 pixels, category judgments concentrated), the verification passes, meaning the "target structured data verification passes"; otherwise, it fails.

[0108] Second, semantic validation specifically verifies the rationality of the behavior state. On one hand, it checks if the behavior and location match. For example, "human controlling drone" must satisfy the condition that "the distance between the person and the drone is close (e.g., <50 pixels), and the drone's position changes." It cannot be labeled as "controlling" when the person is far away and the drone is not moving. On the other hand, it checks if the behavior and key points match. For example, "controlling drone" requires the person's hand to be raised (e.g., the hand's coordinates are higher than the shoulder), and it cannot be labeled as "controlling" when the hand is hanging down. Furthermore, it checks the continuity of behavior; the behavior in adjacent frames must be consistent (e.g., from "no interaction" to "controlling" and then back to "controlling"), and cannot jump around (e.g., "controlling" → "no interaction" → "controlling"). Finally, if the logic is reasonable, all checks pass, meaning "behavior state validation passes"; otherwise, it fails.

[0109] Therefore, the final verification result is updated based on the verification results. If the verification result shows that the target structured dataset and target behavior state have passed, the verification result is determined to be successful. Otherwise, the verification result is determined to be unsuccessful.

[0110] Using the above methods, video annotation files can be directly generated by combining entity recognition sequences and spatiotemporal knowledge graphs. Furthermore, to ensure the quality of the video annotation data, the entity recognition sequences and spatiotemporal knowledge graphs can be validated before generating the annotation files. For example, uncertainty quantization can be used to verify the confidence levels of the entity recognition sequences and spatiotemporal knowledge graphs, and cross-validation can be used to verify whether there are temporal inconsistencies between them. If the validation fails, it needs to be re-verified, and the relevant data needs to be regenerated. If the validation passes, the video annotation files are generated directly. This ensures the quality of the video annotation data, provides a data foundation for target detection in downstream task models, and improves the accuracy of downstream task models in recognizing video data.

[0111] To better understand the embodiments of this application, Figure 3 This is an example diagram of the video annotation process provided in an embodiment of this application. Figure 4 This is an example diagram of a video annotation scene provided in the embodiments of this application, combined with Figure 3 and Figure 4 As shown, the video annotation process is illustrated with an example, as follows: Step 1: Raw Video Input and Preprocessing. Specifically, the raw video data stream (such as MP4, AVI, or other video files, or a real-time video stream) is input. The video stream is decoded and decomposed into a series of independent image frames at a fixed frame rate (e.g., 25 FPS) or using an adaptive sampling strategy (based on scene change detection). This outputs a sequence of image frames arranged in chronological order.

[0112] Step 2: The basic annotation agent extracts atomic facts. Specifically, based on the image frame sequence output in Step 1, the basic annotation agent processes each image frame in parallel. The basic annotation agent includes an entity annotation module, a semantic segmentation module, and a keypoint annotation module (HRNet).

[0113] The Entity Labeling Module (DETR) outputs the bounding box and category label for each object based on each image frame. The semantic annotation module (Mask2Former) outputs a pixel-level instance mask for each object based on each image frame. The keypoint annotation module (HRNet) outputs the coordinates of the skeletal keypoints for each human body instance based on each image frame. Finally, the underlying structured data for each frame is output. This data is called "atomic facts," for example: { frame_id: 101, objects: [{ id: 1, class: 'person', bbox: [x,y,w,h], mask: [...], pose: [...] }, { id: 2, class: 'car', bbox: [...] ... }] }

[0114] Step 3: The advanced semantic agent performs context understanding and reasoning. Specifically, based on the multi-frame atomic fact sequence (i.e., entity recognition sequence) output in Step 2, it performs behavior recognition, scene understanding, and event chain reasoning. The advanced semantic agent includes an action recognition module (VideoMAE), a scene understanding module (CLIP), and an event chain reasoning module (GNN).

[0115] Among them, the Action Recognition Module (VideoMAE) analyzes human posture sequences and object position changes based on atomic fact sequences spanning multiple frames to identify dynamic behaviors, such as "Person-A is running"; Among them, the Scene Understanding (i.e. Street View Labeling) module (CLIP) is based on a sequence of atomic facts spanning multiple frames, combined with the visual features of key frames and a text knowledge base, to give the scene a global label, such as "daytime", "street", "rain" etc. Among them, the Event Chain Reasoning Module (GNN) constructs an event graph based on the atomic fact sequence spanning multiple frames, using all atomic facts (objects, people) and identified behaviors and scenes as nodes, and calculates the causal strength according to the aforementioned method to infer the key event chain.

[0116] Finally, a spatiotemporal knowledge graph containing rich semantic information is output. The spatiotemporal knowledge graph not only contains the state of individual entities, but also the interaction behaviors and causal relationships between entities, for example: {event_chain:["Person-A opens the car door", "Person-A enters the car"]}.

[0117] Step 4: The quality control agent performs verification and feedback. As the system's decision-making center, the quality control agent quantifies annotation uncertainty using Monte Carlo Dropout: it performs multiple forward inferences on the same frame, calculating the positional variance and classification entropy of the predicted bounding boxes. When the variance exceeds a threshold, it is judged as a low-confidence sample, triggering a feedback mechanism to guide the basic annotation agent and the advanced semantic agent to collaboratively re-optimize. This mechanism reduces manual annotation intervention while ensuring that the error rate of the final output annotation remains within a certain range.

[0118] Dynamic coordination and energy efficiency optimization: The system employs deep reinforcement learning to achieve dynamic scheduling of intelligent agents, specifically including: State perception, which analyzes video complexity indicators (occlusion rate, motion blur, target density, etc.) in real time and constructs a 7-dimensional state vector; Policy decision-making, which selects the optimal activation strategy through the Q-Learning algorithm, for example, enabling only entity detection and basic temporal inference in simple scenarios, while enabling full-module coordination in complex multi-entity scenarios; Energy efficiency control, which designs a hardware-level power consumption model and dynamically adjusts the allocation of computing resources based on GPU / CPU utilization.

[0119] Specifically, the quality control agent combines the atomic facts output in step 2 with the spatiotemporal knowledge graph output in step 3 for verification. The verification process includes: Uncertainty quantification: The confidence level of the output results of each module is evaluated using methods such as Monte Carlo Dropout.

[0120] Cross-validation: Compare the outputs of different modules, such as whether the causal inference of GNN is consistent with the timing of the behavior recognition module.

[0121] Trigger feedback: When a low-confidence result or logical contradiction is detected, a reprocessing request with guidance information is sent to the corresponding modules in steps 2 and 3 according to the aforementioned review mechanism.

[0122] Final output: Verified and corrected, high-confidence labeled data. For conflicts that cannot be resolved automatically, a tag awaiting manual review is output.

[0123] Step 5: Generate and output the final annotation results. Specifically, based on the verified annotation data output in Step 4, integrate all atomic facts, behavioral labels, scene descriptions, and event chains into a final annotation file in a standard format (such as COCO, JSON, or XML). Finally, output a complete, multi-layered video annotation file that can be directly used for downstream tasks.

[0124] As can be seen from the above embodiments, this application embodiment can first preprocess the original video to be labeled to obtain an image frame sequence. Then, from each image frame in the image frame sequence, the entity structured data of each entity contained therein is extracted to obtain an entity recognition sequence. The entity recognition sequence contains the structured dataset corresponding to each image frame. In this way, by performing basic annotation on each entity in each image frame, each image frame is represented by each structured dataset. Next, by combining each structured dataset in the entity recognition sequence, the scene label corresponding to each image frame and the behavioral state corresponding to each entity contained therein are identified to construct a spatiotemporal knowledge graph. In this way, the spatiotemporal knowledge graph is used to represent the relationship between entities between different image frames, avoiding the problem of false detection or missed detection in the background of image frames with long distance, occlusion, and dynamic blur caused by single image frame detection. Finally, the entity recognition sequence and the spatiotemporal knowledge graph are combined to generate a video annotation file.

[0125] Therefore, compared to the inefficient and costly manual annotation methods in related technologies, which suffer from inconsistencies due to subjective biases and reliance on single-frame automatic annotation (which reduces accuracy and quality), this application extracts structured entity data from the image frames of the original video and combines it with the structured dataset from the entity recognition sequence to extract scene labels and behavioral states for each image frame, thus constructing a spatiotemporal knowledge graph. This is then combined with the entity recognition sequence and the spatiotemporal knowledge graph to generate a video annotation file. This eliminates the need for manual annotation, reduces annotation costs, and improves video annotation efficiency. Furthermore, the spatiotemporal knowledge graph, constructed by combining structured datasets from multiple image frames, effectively avoids false or missed detections in distant, occluded, or dynamically blurred image frames, improving the accuracy and quality of the video annotation data.

[0126] To facilitate better implementation of the video annotation method provided in this application, this application also provides a video annotation device based on the above description. The meanings of the terms used are the same as in the video annotation method described above, and specific implementation details can be found in the descriptions within the method embodiments.

[0127] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of a video annotation device provided in an embodiment of this application. The video annotation device is integrated into the computer equipment of this application, such as a terminal device or a server. The video annotation device may include an acquisition unit 401, an extraction unit 402, a construction unit 403, and a generation unit 404.

[0128] The acquisition unit 401 is used to acquire the original video to be labeled and preprocess the original video to obtain an image frame sequence; Extraction unit 402 is used to extract the entity structured data of each entity contained in each image frame in the image frame sequence to obtain an entity recognition sequence. The entity recognition sequence contains the structured dataset corresponding to each image frame. The construction unit 403 is used to combine each structured dataset in the entity recognition sequence to identify the scene label corresponding to each image frame and the behavior state corresponding to each entity contained therein, so as to obtain a spatiotemporal knowledge graph. The generation unit 404 is used to combine entity recognition sequences and spatiotemporal knowledge graphs to generate video annotation files.

[0129] In some embodiments, the extraction unit 402 is further configured to: The category is used to obtain the entity recognition result for each entity in each image frame; For each image frame in the image frame sequence, pixel-level instance segmentation is performed on each entity within the image frame to obtain the target mask information corresponding to each entity in each image frame; For each image frame in the image frame sequence, pose recognition is performed on the target entities within the image frame to obtain the key point coordinates corresponding to each entity in each image frame; Based on the entity bounding box, entity category label, target mask information and key point coordinates corresponding to each entity in each image frame, the entity structured data corresponding to each entity in each image frame is obtained, and the entity recognition sequence is constructed by combining the entity structured data corresponding to each entity in each image frame.

[0130] In some embodiments, the extraction unit 402 is further configured to: For each image frame in the image frame sequence, feature extraction processing is performed on the image frame to obtain the initial image features corresponding to each image frame; Obtain a preset query vector, and use a cross-attention mechanism to calculate the preset query vector and the initial image features corresponding to each image frame to obtain the intermediate image features corresponding to each image frame; Based on the intermediate image features corresponding to each image frame, the entity bounding box and entity category label corresponding to each entity are identified.

[0131] In some embodiments, the extraction unit 402 is further configured to: The convolutional neural network is used to extract features from each image frame in the image frame sequence to obtain the underlying visual features corresponding to each image frame. Multi-scale feature maps are obtained by decoding based on low-level visual features using a pixel-level decoder. Obtain a preset entity query vector, input the preset entity query vector and multi-scale feature map into the adaptive decoder for multi-layer cross-attention calculation, and obtain at least one entity category and the entity mask vector associated with each entity category; A high-resolution feature map is generated based on the multi-scale feature map, and a dot product operation is performed between each entity mask vector and the high-resolution feature map to obtain the target mask information corresponding to each entity in each image frame.

[0132] In some embodiments, the extraction unit 402 is further configured to: Each image frame in the image frame sequence is input into the high-resolution network; Each image frame is represented by a high-resolution backbone network in a high-resolution network to obtain high-resolution features for each image frame. Each image frame is also represented by a low-resolution branch network in parallel with the high-resolution backbone network to obtain low-resolution features for each image frame. The high-resolution features and low-resolution features for each image frame are then fused to obtain high-resolution fused features for each image frame. Predefined key point information is obtained, and a heatmap corresponding to each preset key point is generated by combining high-resolution fusion features and predefined key point information through a convolutional network. The heatmap contains the pixel value corresponding to each key pixel. The coordinates of key points are determined by combining the pixel values ​​of each key pixel in each heatmap.

[0133] In some implementations, the building unit 403 is further configured to: Based on each structured dataset in the entity recognition sequence, the behavioral state corresponding to each entity contained in each image frame and the scene label corresponding to each image frame are identified. By combining the scene label corresponding to each image frame and the behavior state corresponding to each entity contained therein, an event node corresponding to each image frame is constructed; Determine the temporal relationship between multiple event nodes corresponding to multiple image frames, and calculate the causal weight between any two temporally adjacent event nodes according to the temporal relationship; Based on the temporal relationship between nodes and the causal weight between any two temporally adjacent event nodes, two temporally adjacent event nodes are connected in series to obtain a spatiotemporal knowledge graph.

[0134] In some embodiments, the generating unit 404 is further configured to: The entity recognition sequence and the spatiotemporal knowledge graph were validated using labeled data to obtain the validation results. When the verification result is successful, the entity recognition sequence and spatiotemporal knowledge graph are identified as verified labeled data. The verified labeled data is encapsulated to obtain video labeled files.

[0135] In some embodiments, the generating unit 404 is further configured to: The Monte Carlo algorithm is used to determine the first confidence level of each structured dataset in the entity recognition sequence, and the second confidence level of the spatiotemporal knowledge graph. The entity recognition sequence and the spatiotemporal knowledge graph are cross-referenced to obtain the comparison results; When both the first confidence level and the second confidence level are greater than or equal to the preset confidence threshold, and the comparison results determine that the information between the entity recognition sequence and the spatiotemporal knowledge graph is consistent, the verification result is determined to be successful. When either the first confidence level or the second confidence level is less than the preset confidence threshold, or when the comparison results indicate that the information between the entity recognition sequence and the spatiotemporal knowledge graph is inconsistent, the verification result is determined to be verification failure.

[0136] In some embodiments, the generating unit 404 is further configured to: When the verification result is that the verification fails, the abnormal time node corresponding to the abnormal data in the target data is determined. The target data is at least one of entity recognition sequence and spatiotemporal knowledge graph. Acquire at least one target image frame from the original video at other time points adjacent to the abnormal time point; Entity recognition is performed on the target image frame to obtain the target structured dataset corresponding to the target image frame, and the target behavior state corresponding to each entity is identified based on the target structured dataset; The target structured dataset and target behavior state are validated to obtain the validation results; When the verification result shows that the target structured dataset and the target behavior state have passed the verification, the verification result is determined to be successful.

[0137] As can be seen from the above, this application extracts the structured entity data of the image frames corresponding to the original video and combines it with each structured dataset in the entity recognition sequence to extract the scene label and behavioral state of each entity corresponding to each image frame, so as to construct a spatiotemporal knowledge graph. Then, the video annotation file is generated by combining the entity recognition sequence and the spatiotemporal knowledge graph. In this way, no manual annotation is required, reducing the annotation cost and improving the efficiency of video annotation. Furthermore, the spatiotemporal knowledge graph constructed by combining the structured datasets of multiple image frames to represent the entity association relationship between different image frames effectively avoids the problems of false detection or missed detection in the background of image frames that are far away, occluded, or dynamically blurred, thus improving the accuracy of video annotation data and ensuring the quality of video annotation data.

[0138] The specific implementation of each of the above units can be found in the previous embodiments, and will not be repeated here.

[0139] Figure 6To implement the structural block diagram of a terminal in this embodiment of the application, the terminal 110 includes: a radio frequency (RF) circuit 510, a memory 515, an input unit 530, a display unit 540, a sensor 550, an audio circuit 560, a wireless fidelity (WiFi) module 570, a processor 580, and a power supply 590, among other components. Those skilled in the art will understand that the terminal 110 structure shown in the figures does not constitute a limitation on a mobile phone or computer, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0140] The RF circuit 510 can be used to receive and transmit signals during information transmission or calls. In particular, it receives downlink information from the base station and processes it with the processor 580; in addition, it transmits uplink data to the base station.

[0141] The memory 515 can be used to store software programs and modules. The processor 580 executes various functional applications and data processing of the terminal by running the software programs and modules stored in the memory 515.

[0142] The input unit 530 can be used to receive input numeric or character information, and to generate key signal inputs related to the terminal's settings and function control. Specifically, the input unit 530 may include a touch panel 531 and other input devices 532.

[0143] The display unit 540 can be used to display input or provided information, as well as various menus of the terminal. The display unit 540 may include a display panel 541.

[0144] Audio circuit 560, speaker 561, and microphone 562 provide an audio interface.

[0145] In this embodiment, the processor 580 included in the terminal 110 can execute the video annotation method of the previous embodiment.

[0146] The terminal 110 in this application embodiment includes, but is not limited to, mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc. This invention embodiment can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0147] Figure 7This is a partial structural block diagram of a server embodying this embodiment of the application. The server 120 can vary significantly due to different configurations or performance characteristics, and may include one or more Central Processing Units (CPUs) 622 (e.g., one or more processors) and a memory 632, and one or more storage media 620 (e.g., one or more mass storage devices) for storing application programs 642 or data 644. The memory 632 and storage media 620 may be temporary or persistent storage. The program stored in the storage media 620 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the server 120. Furthermore, the CPU 622 may be configured to communicate with the storage media 620 and execute the series of instruction operations in the storage media 620 on the server 120.

[0148] Server 120 may also include one or more power supplies 626, one or more wired or wireless network interfaces 650, one or more input / output interfaces 658, and / or one or more operating systems 641, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0149] The central processing unit 622 in server 120 can be used to execute the video annotation method of the embodiments of this application.

[0150] This application also provides a computer-readable storage medium for storing program code for executing the video annotation methods of the foregoing embodiments.

[0151] This application also provides a computer program product, which includes a computer program. The processor of a computer device reads and executes the computer program, causing the computer device to perform the video annotation method described above.

[0152] Furthermore, the terms “comprising” and “including”, and any variations thereof, are intended to cover non-exclusive inclusion, such that a process, method, apparatus, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are expressly listed, but may include other steps or units that are not expressly listed or that are inherent to such process, method, product or device.

[0153] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0154] It should be understood that in the description of the embodiments of this application, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0155] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed between them may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0156] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0157] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0158] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0159] It should also be understood that the various implementation methods provided in this application can be combined arbitrarily to achieve different technical effects.

[0160] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0161] The above is a detailed description of the embodiments of this application. However, this application is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A video annotation method, characterized in that, include: Obtain the original video to be labeled, and preprocess the original video to obtain an image frame sequence; For each image frame in the image frame sequence, the entity structured data of each entity contained in the image frame is extracted to obtain an entity recognition sequence, wherein the entity recognition sequence contains the structured dataset corresponding to each image frame; By combining each structured dataset in the entity recognition sequence, the scene label corresponding to each image frame and the behavioral state corresponding to each entity contained therein are identified, thus obtaining a spatiotemporal knowledge graph; By combining the entity recognition sequence and the spatiotemporal knowledge graph, a video annotation file is generated.

2. The method according to claim 1, characterized in that, The entity structured data includes entity recognition results, target mask information, and key point coordinates. For each image frame in the image frame sequence, the entity structured data of each entity contained within that image frame is extracted to obtain an entity recognition sequence, including: For each image frame in the image frame sequence, identify the entity bounding box and entity category label corresponding to each entity within the image frame; For each image frame in the image frame sequence, pixel-level instance segmentation is performed on each entity within the image frame to obtain the target mask information corresponding to each entity in each image frame; For each image frame in the image frame sequence, pose recognition is performed on the target entity within the image frame to obtain the key point coordinates corresponding to each entity in each image frame. Based on the entity bounding box, entity category label, target mask information and key point coordinates corresponding to each entity in each image frame, the entity structured data corresponding to each entity in each image frame is obtained, and the entity recognition sequence is constructed by combining the entity structured data corresponding to each entity in each image frame.

3. The method according to claim 2, characterized in that, The step of identifying the entity bounding box and entity category label corresponding to each entity within each image frame in the image frame sequence includes: For each image frame in the image frame sequence, feature extraction processing is performed on the image frame to obtain the initial image features corresponding to each image frame; A preset query vector is obtained, and the preset query vector and the initial image features corresponding to each image frame are calculated through a cross-attention mechanism to obtain the intermediate image features corresponding to each image frame. Based on the intermediate image features corresponding to each image frame, the entity bounding box and entity category label corresponding to each entity are identified.

4. The method according to claim 2, characterized in that, For each image frame in the image frame sequence, pixel-level instance segmentation is performed on each entity within the image frame to obtain target mask information corresponding to each entity in each image frame, including: The convolutional neural network is used to perform feature extraction processing on each image frame in the image frame sequence to obtain the underlying visual features corresponding to each image frame; A multi-scale feature map is obtained by decoding the underlying visual features using a pixel-level decoder. Obtain a preset entity query vector, input the preset entity query vector and the multi-scale feature map into the adaptive decoder to perform multi-layer cross-attention calculation, and obtain at least one entity category and an entity mask vector associated with each entity category; A high-resolution feature map is generated based on the multi-scale feature map, and a dot product operation is performed between each entity mask vector and the high-resolution feature map to obtain the target mask information corresponding to each entity in each image frame.

5. The method according to claim 2, characterized in that, For each image frame in the image frame sequence, pose recognition is performed on the target entity within the image frame to obtain the key point coordinates corresponding to each entity in each image frame, including: Each image frame in the image frame sequence is input into a high-resolution network; Each image frame is represented by a high-resolution backbone network in the high-resolution network to obtain high-resolution features corresponding to each image frame. Each image frame is also represented by a low-resolution branch network in parallel with the high-resolution backbone network to obtain low-resolution features corresponding to each image frame. The high-resolution features and low-resolution features corresponding to each image frame are then fused to obtain high-resolution fused features corresponding to each image frame. Obtain predefined key point information, and generate a heatmap corresponding to each preset key point by combining the high-resolution fusion feature and the predefined key point information through a convolutional network. The heatmap contains the pixel value corresponding to each key pixel. The coordinates of key points are determined by combining the pixel values ​​of each key pixel in each heatmap.

6. The method according to claim 1, characterized in that, The process involves combining each structured dataset in the entity recognition sequence to identify the scene label corresponding to each image frame and the behavioral state corresponding to each entity contained therein, thereby obtaining a spatiotemporal knowledge graph, including: Based on each structured dataset in the entity recognition sequence, the behavioral state corresponding to each entity contained in each image frame and the scene label corresponding to each image frame are identified. By combining the scene label corresponding to each image frame and the behavior state corresponding to each entity contained therein, an event node corresponding to each image frame is constructed; Determine the node temporal relationship between multiple event nodes corresponding to multiple image frames, and calculate the causal weight between any two temporally adjacent event nodes according to the node temporal relationship; Based on the temporal relationship between the nodes and the causal weight between any two temporally adjacent event nodes, two temporally adjacent event nodes are connected in series to obtain a spatiotemporal knowledge graph.

7. The method according to claim 1, characterized in that, The step of combining the entity recognition sequence and the spatiotemporal knowledge graph to generate a video annotation file includes: The entity recognition sequence and the spatiotemporal knowledge graph are verified using labeled data to obtain the verification results. When the verification result is successful, the entity recognition sequence and the spatiotemporal knowledge graph are identified as verified labeled data. The verified annotation data is encapsulated to obtain a video annotation file.

8. The method according to claim 7, characterized in that, The step of verifying the entity recognition sequence and the spatiotemporal knowledge graph using labeled data to obtain verification results includes: The Monte Carlo algorithm is used to determine the first confidence level of each structured dataset in the entity recognition sequence and the second confidence level of the spatiotemporal knowledge graph. The entity recognition sequence is cross-referenced with the spatiotemporal knowledge graph to obtain the comparison result. When both the first confidence level and the second confidence level are greater than or equal to the preset confidence threshold, and the information between the entity recognition sequence and the spatiotemporal knowledge graph is determined to be consistent based on the comparison result, the verification result is determined to be successful. When either the first confidence level or the second confidence level is less than the preset confidence threshold, or when the comparison result determines that the information between the entity recognition sequence and the spatiotemporal knowledge graph is inconsistent, the verification result is determined to be verification failure.

9. The method according to claim 7, characterized in that, The method further includes: When the verification result is that the verification fails, the abnormal time node corresponding to the abnormal data in the target data is determined, and the target data is at least one of the entity recognition sequence and the spatiotemporal knowledge graph. At least one target image frame from other time nodes adjacent to the abnormal time node is acquired from the original video; Entity recognition is performed on the target image frame to obtain the target structured dataset corresponding to the target image frame, and the target behavior state corresponding to each entity is identified based on the target structured dataset; The target structured dataset and the target behavior state are validated to obtain the validation results; When the verification result is that the target structured dataset and the target behavior state pass the verification, the verification result is determined to be a successful verification.

10. A video annotation device, characterized in that, include: The acquisition unit is used to acquire the original video to be labeled and preprocess the original video to obtain an image frame sequence; The extraction unit is used to extract the entity structured data of each entity contained in each image frame in the image frame sequence to obtain an entity recognition sequence, wherein the entity recognition sequence contains the structured dataset corresponding to each image frame. The construction unit is used to combine each structured dataset in the entity recognition sequence to identify the scene label corresponding to each image frame and the behavioral state corresponding to each entity contained therein, so as to obtain a spatiotemporal knowledge graph. The generation unit is used to combine the entity recognition sequence and the spatiotemporal knowledge graph to generate a video annotation file.

11. A computer device, characterized in that, The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the video annotation method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to execute the video annotation method according to any one of claims 1 to 9.

Images