Method, system and device for video annotation

Abstract

The application provides a video labeling method, comprising the following steps: extracting a plurality of video frames from an unlabeled video, displaying at least one video frame in the plurality of video frames to a user through a display interface, obtaining a user labeling result of the user labeling the at least one video frame in the display interface, and obtaining a labeling result of other video frames in the plurality of video frames according to the user labeling result. The method makes the user only need to label a key frame once or even only confirm whether the key frame is labeled correctly in the video labeling process, so that the complete labeling result of the video can be obtained, the labeling operation of the user is greatly reduced, and the video labeling efficiency and the use experience of the user are improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI), and in particular to methods, systems and devices for video annotation. Background Technology

[0002] In the current AI field, training and optimizing AI models requires a large number of labeled image and video samples. For example, in image-based model training such as classification, detection, and segmentation, the video is typically first framed, and then each frame is labeled. The labeled video stream, image set, or audio set can then be used to train the AI ​​model.

[0003] Because current video annotation is done manually, and a video consists of tens of thousands of video frames, manual video annotation is a very time-consuming and labor-intensive process. In addition, human energy is limited, resulting in low accuracy and low efficiency of video annotation. Summary of the Invention

[0004] This application provides a video annotation method, system, and device to solve the problems of low efficiency and low accuracy in manual video annotation.

[0005] Firstly, a video annotation method is provided, which includes the following steps: extracting multiple video frames from an unannotated video, presenting at least one of the multiple video frames (which may be referred to as a keyframe) to the user through a display interface, so that the user can annotate the desired target in the keyframe, obtain the user annotation result, and finally automatically annotating the target in the remaining non-keyframes based on the user annotation result, thereby obtaining the annotation result of the entire video.

[0006] Optionally, the annotation results are used for learning by the AI ​​model.

[0007] In the above method, other non-keyframes are automatically labeled based on the annotation results of keyframes, thereby obtaining the annotation results of the entire video. The annotation results of keyframes can be labeled by the user or automatically recommended to the user by the video annotation system and then confirmed by the user. This means that during the entire annotation process, the user only needs to annotate the keyframe once, or even just confirm whether the keyframe is labeled correctly once, to obtain the complete annotation results of the video. This greatly reduces the user's annotation operations, improves video annotation efficiency, and enhances the user experience.

[0008] In one possible implementation of the first aspect, the image similarity between multiple video frames extracted from the unlabeled video is lower than a first threshold, or the amount of object variation between multiple video frames is higher than a second threshold.

[0009] Alternatively, multiple video frames can be extracted from the unlabeled video at a fixed frame rate.

[0010] Optionally, multiple video frames can be extracted from the unlabeled video according to a frame rate manually set by the user.

[0011] Optionally, the frame extraction rate can be dynamically adjusted based on the video content of the unlabeled video to extract multiple video frames from the unlabeled video. In specific implementation, the image similarity between adjacent frames can be determined first, and then the image similarity can be compared with a pre-stored similarity mapping relationship to determine the frame rate corresponding to the image similarity. For example, when the similarity is 0.1, the frame rate is 1, and when the similarity is 0.2, the frame rate is 2. Similarly, the change in objects between frames can be determined first, and then the change in objects can be compared with a pre-stored change mapping relationship to determine the frame rate corresponding to the change in objects between frames. The specific form of the pre-stored similarity mapping relationship and change mapping relationship can be a mathematical formula, a mapping relationship table, etc., and this application does not limit it. It should be understood that when there are moving objects in a video, there will be differences in grayscale between adjacent frames or multiple adjacent frames. Based on this, the rate of change of objects between frames can be obtained. Of course, known video segments and corresponding known changes in objects between frames can also be used as training samples to train a deep neural network. The trained model can infer the corresponding changes in objects between frames based on the input video segments. Alternatively, a sample set can be generated from historical unlabeled videos and their corresponding frame extraction rates, and this sample set can be used to train an AI model. The currently processed unlabeled video can be input into the trained AI model to obtain the frame extraction rate corresponding to each video time segment.

[0012] It should be understood that dynamically adjusting the frame extraction rate based on the video content of unlabeled videos and extracting multiple video frames from unlabeled videos can avoid extracting too many redundant frames or failing to extract video frames containing the target. This allows the finally labeled video frames to be used as high-quality training samples for training AI models, thereby improving the user experience.

[0013] In one possible implementation of the first aspect, keyframes can be manually selected by the user. Specifically, multiple video frames obtained after frame extraction can be presented to the user, who can select one or more video frames as keyframes for keyframe annotation. Keyframes can also be determined based on the video content. Specifically, fixed frame numbers among multiple video frames can be used as keyframes, such as the first or last frame among multiple video frames. Alternatively, keyframes among multiple video frames can be determined based on a keyframe selection model. The keyframe selection model can be obtained by training an AI model using multiple known video frames and their corresponding known keyframes as training samples. The trained keyframe selection model can output the corresponding keyframes based on the input video frames.

[0014] It should be understood that automatically recommending keyframes to users can reduce the number of operations users need to perform in video annotation, improve the user experience, and increase the efficiency of video annotation.

[0015] In one possible implementation of the first aspect, user annotation results can be obtained in the following three ways.

[0016] Method 1: User annotation results on keyframes can be obtained through "automatic annotation". Specifically, after presenting keyframes to users, multiple target boxes (or multiple center points, masks, etc.) can be recommended at the same time, and multiple targets can be automatically annotated. Users do not need to perform any operations to automatically annotate the targets and generate user annotation results, which improves user experience and annotation efficiency.

[0017] In practical implementation, user-annotated results on keyframes can be recommended to users based on video content. For example, highway surveillance videos typically annotate vehicles as targets. When an unannotated video is a highway surveillance video, vehicles can be selected in the keyframes to recommend annotation results to the user. Alternatively, a few keywords can be obtained from the user, such as inputting "vehicle," and the system can annotate the keywords and obtain them from the user. Furthermore, unannotated videos can be input into a recommendation annotation model to obtain recommended annotation results. This recommendation annotation model can be an AI model. Specifically, known unannotated videos and corresponding known annotation results can be used as training samples. After training a deep neural network using computer vision algorithms such as objectness algorithms, a trained recommendation annotation model can be obtained. Alternatively, other mature computer vision algorithms in the industry can be used to implement the above functions. It should be understood that the above examples are for illustration and do not constitute specific limitations.

[0018] Method two involves obtaining user annotation results through human-computer interaction. Specifically, after presenting keyframes to the user, multiple target bounding boxes, center points, or masks can be recommended simultaneously for the user to choose from. The user can simply click on any position within a target bounding box to select the target, without needing to use a dropdown menu or outline selection, thus improving user experience and annotation efficiency. In practice, after recommending annotation results to the user, the user can manually correct the annotation results through the displayed interface to further improve annotation accuracy.

[0019] Method 3: User annotation results can be obtained through manual annotation. Specifically, after presenting the keyframes to the user, the user can also manually draw the target box, center point, or mask. This application does not limit this.

[0020] Understandably, if the user selects "automatic annotation," the video annotation system can automatically generate the user's annotation results on keyframes without requiring any annotation action from the user, greatly improving the user experience and annotation efficiency. If the user selects "human-computer interaction," the video annotation system can recommend annotation results to the user, who only needs to manually confirm the recommended annotation results, such as moving the mouse to any position on the object to automatically generate the center point, without needing to manually find the object's center point, ensuring annotation accuracy while improving annotation efficiency. If the user selects "manual annotation," the user can annotate the keyframes themselves, such as manually drawing target boxes on the object. The keyframe annotation unit can record the user's drawing information and use it as new samples to train the recommended annotation model, thereby making the user annotation results obtained through automatic annotation and human-computer interaction more accurate and improving the user experience.

[0021] In one possible implementation of the first aspect, the annotation result includes one or more of the following: a bounding box, a center point, and a mask. The form of the bounding box may differ depending on the AI ​​algorithm used for video annotation. For example, the bounding box may be a preset shape such as a rectangle, circle, or ellipse. When the video annotation system uses an AI model based on mask RCNN for precise target localization, the shape of the bounding box is close to the shape of the target, such as the outline of the target; that is, the bounding box is the outline of the target. Regardless of the algorithm used for video annotation, the bounding box can be a line bounding box or a scatter bounding box composed of multiple discrete points; this application does not impose specific limitations.

[0022] It should be understood that the video annotation method provided in this application supports multiple annotation formats, especially the annotation methods for masks and center points. If the video is annotated using traditional manual annotation methods, it will consume a lot of time. However, this application only needs to annotate keyframes, or even not annotate them at all. By simply confirming the user annotation results recommended by the video annotation system, the annotation results of the entire video can be obtained, which greatly improves the efficiency of video annotation.

[0023] In one possible implementation of the first aspect, multiple computing units can be invoked to annotate the remaining non-keyframes based on the user annotation results on the keyframes. Several methods for invoking multiple computing units to annotate non-keyframes in parallel are described below.

[0024] Method 1 involves dividing the video frame to be labeled into multiple video segments and randomly assigning these segments to various processing units. Each processing unit processes one video segment, where a video segment may include at least two video frames. For example, processing unit 1 processes video frames 1 through 10 to generate labeling results for those frames, and processing unit 2 processes video frames 11 through 20 to generate labeling results for those frames. The video segments processed by each processing unit can be randomly assigned by the automatic labeling unit; this application does not impose any limitations on this. In this way, the automatic labeling unit can process multiple video segments simultaneously, improving the efficiency of video labeling.

[0025] Method 2 allows for the random allocation of video frames to be labeled to various processing units. Each processing unit can process one video frame, meaning that one processing unit uses the automatic labeling model to infer the labeling result of a video frame and outputs the labeling result of that video frame. In this way, the automatic labeling unit can process multiple video frames simultaneously, improving the processing efficiency of video labeling. Furthermore, since each processing unit processes each video frame independently, both forward and reverse labeling can be achieved, improving the user experience.

[0026] Method 3: Based on user annotation results, the same video frame can be assigned to multiple processing units. Each processing unit can load one algorithm, and multiple processing units can process the same video frame simultaneously. Finally, the outputs of the multiple processing units are superimposed to output the annotation result. For example, suppose processing unit 1 loads an algorithm to annotate the target bounding boxes of the video frame, and processing unit 2 loads an algorithm to annotate the mask of the video frame. In this case, processing units 1 and 2 process frame A1 simultaneously. Processing unit 1 outputs frame A11 with target bounding boxes, and processing unit 2 outputs frame A12 with mask annotations. Finally, the automatic annotation unit can superimpose frames A11 and A12 to obtain frame A1 with both target bounding boxes and masks. It should be understood that the above example is for illustration; the automatic annotation unit can randomly assign video frames to processing units for processing, and this does not constitute a specific limitation.

[0027] In practice, a processing unit can be a program or process, or an independent processor with its own hardware and software system. Processing units can be connected via wired or wireless means, and the physical form of the processing unit can be determined based on its processing power.

[0028] It should be understood that by calling multiple computing units to process the automatic annotation process in parallel, the efficiency of video annotation can be improved, thus enhancing the user experience.

[0029] In one possible implementation of the first aspect, obtaining the annotation results of other video frames among multiple video frames based on the user annotation results includes: annotating video frames after or before the keyframe based on the user annotation results to obtain annotation results. Simply put, the user can choose to perform automatic annotation using either forward annotation or reverse annotation. Forward annotation refers to automatically annotating video frames after the keyframe, while reverse annotation refers to automatically annotating video frames before the keyframe.

[0030] For example, when watching a video, a user selects a frame as a keyframe near the end using manual frame extraction and marks a target manually (either through interactive annotation or automatic annotation). The user can then choose reverse annotation to automatically mark targets in frames preceding the keyframe. Similarly, if the user selects a keyframe at the beginning of the video, they can choose forward annotation to automatically mark targets in frames following the keyframe. This allows for automatic annotation of remaining non-keyframes regardless of where the keyframe is in the video, significantly improving video annotation efficiency and enhancing the user experience.

[0031] In one possible implementation of the first aspect, the method further includes the following steps: receiving modification information of the annotation results, generating new keyframes and new user annotation results based on the modification information, and then automatically annotating new non-keyframes based on the new keyframes and new user annotation results to obtain the modified annotation results.

[0032] Optionally, the modification information can be generated by the user through the display interface. Specifically, the automatic annotation results can be presented to the user through the display interface. If no correction request is received from the user, the next video frame can be processed until the video playback ends. If the user believes that the annotation result of a certain frame is incorrect and needs to be corrected, the user can pause the playback, switch to the annotation correction mode, and then correct the position or size of the target box, center point, or mask by dragging the mouse. The video frame corresponding to the modification information can be used as a new keyframe for automatic annotation again.

[0033] Optionally, the modification information can be obtained by inputting the annotation results into the annotation correction model, wherein the annotation correction model can be obtained by training the AI ​​model in advance using known annotation results and corresponding known modification information as training samples.

[0034] Understandably, automatically detecting the accuracy of annotation results through an annotation correction model can improve annotation accuracy, reduce the number of user operations, and further enhance the user experience. It should be understood that the aforementioned iterative correction and automatic annotation method allows the automatic frame extraction unit, keyframe processing unit, and annotation correction unit to form a closed loop, continuously optimizing the annotation results and improving annotation accuracy. Users can also choose whether to perform annotation correction based on the training objective, and can also choose the number of optimizations. For example, if the required annotation accuracy for the training objective is low, the user can choose not to use annotation correction, thereby quickly obtaining the video annotation results. Conversely, if the required annotation accuracy for the training objective is high, the user can choose to use annotation correction and set the number of corrections here, for example, to 5. It should be understood that the above examples are for illustrative purposes only and do not constitute specific limitations.

[0035] Secondly, a video annotation system is provided, comprising: an automatic frame extraction unit for extracting multiple video frames from an unannotated video; a keyframe processing unit for displaying at least one video frame from the multiple video frames to a user through a display interface; an automatic annotation unit for obtaining user annotation results for the at least one video frame annotated by the user on the display interface, wherein the user annotation results include the image region of the target in the at least one video frame; and an annotation correction unit for obtaining annotation results for other video frames from the multiple video frames based on the user annotation results, wherein the annotation results include the image region of the target in the other video frames.

[0036] In one possible implementation of the second aspect, the automatic frame extraction unit is used to extract multiple video frames from the unlabeled video based on the video content of the unlabeled video, wherein the image similarity between the multiple video frames is lower than a first threshold, or the object variation between the multiple video frames is higher than a second threshold.

[0037] In one possible implementation of the second aspect, at least one video frame is the first or last frame among a plurality of video frames; or, at least one video frame is obtained by inputting a plurality of video frames into a keyframe selection model, wherein the keyframe selection model is obtained by training a neural network model using a plurality of known video frames and corresponding known keyframes as training samples.

[0038] In one possible implementation of the second aspect, the keyframe processing unit is used to input at least one video frame into the recommendation annotation model to obtain recommendation annotation results, wherein the recommendation annotation results include image regions of at least one recommended target in at least one video frame; the keyframe annotation unit is used to display the recommendation annotation results to the user through a display interface and obtain the user annotation results selected by the user in the recommendation annotation results.

[0039] In one possible implementation of the second aspect, the annotation result includes one or more of the following: target bounding box, center point, and mask.

[0040] In one possible implementation of the second aspect, the automatic annotation unit is used to call multiple computing units to process other video frames in parallel based on the user annotation results, and obtain annotation results for other video frames, wherein one computing unit processes one video frame, or at least one computing unit processes one video frame, and each computing unit in at least one computing unit generates an annotation result.

[0041] In one possible implementation of the second aspect, the automatic annotation unit is used to annotate video frames after or before the keyframes based on user annotation results, and obtain annotation results.

[0042] In one possible implementation of the second aspect, the system further includes an annotation correction unit. The keyframe processing unit is used to receive modification information of the annotation results from the user through a display interface. The modification information comes from the user's modification of the image region of the target in other video frames. Alternatively, the annotation correction unit is used to obtain the modification information of the annotation results through an annotation correction model. The annotation correction model is obtained by training a neural network model using multiple known annotation results and corresponding known modification information as training samples. The automatic annotation unit is used to modify the annotation results of other video frames in multiple video frames according to the modification information.

[0043] In one possible implementation of the second aspect, the annotation results are used for learning by an artificial intelligence (AI) model.

[0044] It should be understood that the video annotation system in this embodiment can be a physical machine, such as an x86 server, or a virtual machine, or a computer cluster composed of multiple physical machines or virtual machines. Furthermore, the internal unit modules of the video annotation system can be divided in various ways. Each module can be a software module, a hardware module, or a combination of software and hardware modules. This application does not limit these aspects.

[0045] In summary, this application provides a video annotation system that can automatically annotate other non-keyframes based on the annotation results of keyframes, thereby obtaining the annotation results of the entire video. The annotation results of keyframes can be user-annotated or automatically recommended to the user by the video annotation system and then confirmed by the user. This means that during the entire annotation process, the user only needs to annotate a keyframe once, or even just confirm whether the keyframe is annotated correctly once, to obtain the complete video annotation results, which greatly reduces the user's annotation operations, improves video annotation efficiency, and enhances the user experience.

[0046] Thirdly, a computer program product is provided, including a computer program that, when read and executed by a computing device, implements the method described in the first aspect.

[0047] Fourthly, a computer-readable storage medium is provided, including instructions that, when executed on a computing device, cause the computing device to perform the method as described in the first aspect.

[0048] Fifthly, a computing device is provided, including a processor and a memory, wherein the processor executes code in the memory to implement the method as described in the first aspect.

[0049] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0051] Figure 1 This is a system architecture diagram provided in this application;

[0052] Figure 2 This is a deployment diagram of a video annotation system provided in this application;

[0053] Figure 3 This is a flowchart illustrating the steps of a video annotation method provided in this application;

[0054] Figure 4 This is a schematic diagram of the interface for manual frame extraction in a video annotation method provided in this application;

[0055] Figure 5 This is a schematic diagram of the interface for keyframe annotation in a video annotation method provided in this application;

[0056] Figure 6 This is a flowchart illustrating the automatic annotation process in a video annotation method provided in this application;

[0057] Figure 7 This is a flowchart illustrating the annotation correction process in a video annotation method provided in this application;

[0058] Figure 8 This is a schematic diagram of an exemplary display interface provided in this application;

[0059] Figure 9 This is a schematic diagram of the structure of a video annotation system provided in this application;

[0060] Figure 10This is a schematic diagram of the structure of a computing device provided in this application. Detailed Implementation

[0061] To facilitate understanding of the technical solutions of the present invention, some of the terms involved in the present invention will be explained first. It should be understood that the terms used in the embodiments section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0062] AI: AI is the theory, methods, technology, and application system that uses digital computers or computing devices controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. The applications of artificial intelligence are very broad, including facial recognition, vehicle recognition, pedestrian re-identification, data processing applications, and so on.

[0063] Labeling: In the field of AI, labeling refers to the process of adding labels to unlabeled data in a relevant context. For example, unlabeled data consists of unlabeled images. In image classification, unlabeled images are assigned their category. In object detection, objects in unlabeled images are assigned their location information and category.

[0064] Mask: A mask is obtained by occluding a selected image, graphic, or object in the image to be processed. It is used to control the image processing area or process. In the field of computer vision, the image area selected by the mask will be occluded during image processing. Masks can be used to segment images, such as extracting regions of interest, and only processing the regions of interest selected by the mask. Masks can also be used to shield, and the areas selected by the mask will not participate in the processing.

[0065] Cloud computing: The core attribute of cloud computing is shared resource service, referring to cloud infrastructure and services provided by third-party providers to users that can be used through public networks (such as the Internet). Users obtain access to cloud infrastructure and services by paying a fee. The video annotation method provided in this application can be provided to users as a cloud computing service.

[0066] Next, the application scenarios of "video annotation" involved in this application will be explained.

[0067] AI is the theory, methods, technology, and application system that uses digital computers or computing devices controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. The applications of artificial intelligence are very broad, including facial recognition, vehicle recognition, pedestrian re-identification, data processing applications, and so on.

[0068] An AI model is a set of mathematical methods for implementing AI. A large number of training samples can be used to train an AI model, enabling it to acquire predictive capabilities. For example, to train a model for classifying spam emails, the training phase can begin by training a neural network with a set of training samples labeled with multiple spam tags and multiple non-spam tags. The neural network continuously captures the relationships between these emails and tags, adjusting and refining its network structure parameters. Then, in the prediction phase, the neural network can classify new emails without tags as spam. It should be understood that the above examples are for illustrative purposes and do not constitute specific limitations. It is worth noting that the AI ​​model mentioned in this article is a general term, and AI models can include deep learning models, machine learning models, and so on.

[0069] Therefore, the collection and processing of training samples is a crucial aspect of AI technology, and efficiently collecting training samples has become a focus of effort for many researchers. In the field of computer vision, training samples for AI models are often obtained by processing acquired videos. Specifically, the video can be frame-sampling at a certain rate to obtain multiple images. Each image is then manually labeled. For example, a labeling tool presents the images to be labeled to the user through a graphical user interface (GUI). The user labels the targets in the images by inputting content or dragging the mouse, and the labeling tool generates labels for the images in the background. It should be understood that labeled video streams, image sets, or audio sets can be used as training samples for training AI models such as object detection models, object recognition models, or classification models.

[0070] However, the process of obtaining training samples through video annotation described above has several problems. First, when extracting frames from a video, a fixed extraction rate is often used. A high extraction rate leads to an excessive number of images, resulting in many similar redundant frames. Training the AI ​​model with a large number of similar images fails to improve its prediction accuracy. Conversely, a low extraction rate may prevent the extraction of video frames containing the target, which are precisely the samples needed for AI model training. For example, when training an AI model for vehicle recognition, extracting frames from surveillance videos on both sides of the road at a low extraction rate may result in a large number of frames containing only the road and no vehicles. Using these vehicle-free frames as samples for training the AI ​​model will not improve its prediction accuracy. Furthermore, the information in a video is not uniformly distributed. For instance, vehicles during a red light are mostly stationary images, while those during a green light are mostly moving images. Therefore, a fixed frame rate extraction may not only fail to extract video frames containing the target but also result in a large number of redundant frames. Secondly, after extracting frames from the video, a large number of images to be labeled will be obtained. For example, if a video with a duration of 1 hour is extracted using a frame extraction rate of 10fps, 36,000 images will be generated. For each image, users still need to perform tedious operations to ensure the labeling accuracy. This results in a large amount of time and manpower being spent labeling images, increasing the labeling cost, and the labeling efficiency cannot meet business needs.

[0071] To address the issues of high manual costs, low efficiency, and poor accuracy in video annotation, this application provides a video annotation system. This system can dynamically adjust the frame extraction rate based on the video content to extract suitable video frames for annotation. Furthermore, during the annotation process, it can not only recommend annotable targets to the user when annotating keyframes, but also automatically mark the target in other non-keyframes after the user has annotated the target object in a keyframe. This reduces the workload and time required for manual video annotation, and improves annotation accuracy and efficiency.

[0072] Figure 1 This is an architecture diagram of a video annotation system provided in this application, such as... Figure 1 As shown, the video annotation system architecture includes a data acquisition node 110, a video annotation system 120, and a storage node 130. The data acquisition node 110, the video annotation system 120, and the storage node 130 are connected, which can be wireless or wired, and can be external or internal network connections; this application does not impose any specific limitations.

[0073] Data acquisition node 110 is used to collect various unlabeled videos, including unlabeled video files, video streams, image sets, or point cloud sets, which are not specifically limited in this application. Specifically, data acquisition node 110 can be a surveillance camera, electronic police system, depth camera, drone, etc., or it can be radar or satellite. Of course, the data acquisition node can also be a cloud server used to store unlabeled videos. This cloud server can be deployed with specific services, such as Kafka and / or Flume. Kafka is used to provide high-throughput, highly scalable distributed message queue services, while Flume is a highly reliable, highly available, distributed system for massive log collection, aggregation, and transmission.

[0074] Storage node 130 is used to store the annotated video streams, image sets, or audio sets output by the video annotation system 120. Specifically, it can be a physical server, such as an x86 server; it can also be a virtual machine (VM) implemented based on a general-purpose physical server combined with network functions virtualization (NFV) technology. A VM refers to a complete computer system simulated by software, possessing full hardware system functionality and running in a completely isolated environment, such as a VM within cloud data; or it can be a computer cluster composed of the aforementioned physical servers and / or VMs, such as a computer cluster deployed with a distributed file system (Hadoop Distributed File System, HDFS). This application does not impose specific limitations. It should be understood that the annotated video streams, image sets, or audio sets stored in storage node 130 can be used for training AI models.

[0075] The video annotation system 120 is used to annotate the unannotated video sent by the data acquisition device 110, and then send the annotated video stream, image set and / or audio set to the storage node 130.

[0076] The video annotation system 120 is flexible in deployment and can be deployed in an edge environment. Specifically, it can be one or more edge computing devices or a software system running on one or more edge computing devices in an edge environment. An edge environment refers to a cluster of edge computing devices that are geographically close to the data acquisition node 110 and provide computing, storage, and communication resources, such as edge computing all-in-one machines or computing boxes located on both sides of a road. For example, the video annotation system 120 can be deployed on one or more edge computing devices near an intersection, or it can be a software system running on an edge computing device near an intersection. Two cameras, camera 1 and camera 2, are installed at the intersection to monitor it. Camera 1 and camera 2 act as data acquisition nodes 110, acquiring and sending the monitoring video (unannotated video) to the edge computing device. The edge computing device can then annotate the monitoring video and send the annotated video to the storage node 130 for storage.

[0077] The video annotation system 120 can also be deployed on end devices. End devices include, but are not limited to, user terminals such as desktop computers, laptops, and smartphones. Video annotation can be achieved by running the video annotation system on these user terminals. The end device can also serve as an image providing device for video annotation. In some possible implementations, the video annotation system 120 can also be the same device as the data acquisition node, such as the data acquisition node and the video annotation system 120 being deployed on the same camera. This application does not impose specific limitations.

[0078] The video annotation system 120 can also be deployed in a cloud environment, which is an entity that provides cloud services to users using basic resources under the cloud computing model. The cloud environment includes a cloud data center and a cloud service platform. The cloud data center includes a large amount of basic resources (including computing resources, storage resources, and network resources) owned by the cloud service provider. The video annotation system 120 can be a server in the cloud data center, a virtual machine created in the cloud data center, or a software system deployed on a server or virtual machine in the cloud data center. This software system can be distributed across multiple servers, multiple virtual machines, or a combination of virtual machines and servers; this application does not impose specific limitations. For example, the video annotation system 120 can be deployed in a cloud data center located far from an intersection. Two cameras, camera 1 and camera 2, are installed at this intersection to monitor it. Camera 1 and camera 2 act as data acquisition nodes 110, collecting and sending monitoring video (unannotated video) to the cloud data center. The cloud data center can annotate the monitoring video and send the annotated video to storage node 130 for storage. It should be understood that when the video annotation system 120 is deployed in a cloud environment, the video annotation method can be provided to users as a cloud service.

[0079] It should be understood that, such as Figure 1 As shown, the video annotation system 120 includes multiple unit modules. Therefore, the various unit modules of the video annotation system 120 can be deployed in a distributed manner in different environments. For example, some unit modules of the video annotation system 120 can be deployed in any two or three environments: cloud environment, edge environment, and terminal device environment. For example, as... Figure 2 As shown, some units of the video annotation system 120 (e.g.) Figure 1 The automatic frame extraction unit 121 in the middle is deployed in the edge computing device, and some units (such as Figure 1 The keyframe processing unit 122, automatic annotation unit 123, and annotation correction unit 124 are deployed in the cloud data center, and some units (such as...) Figure 1 The display unit 125 is deployed on the end device, and it should be understood that... Figure 2 This is for illustrative purposes only and does not constitute a specific limitation.

[0080] It should be noted that the internal unit modules of the video annotation system 120 can also be divided in various ways. Each module can be a software module, a hardware module, or a combination of both. This application does not impose any restrictions on this. Figure 1 As an example of a partitioning method, such as Figure 1 As shown, the video annotation system 120 includes an automatic frame extraction unit 121, a key frame processing unit 122, an automatic annotation unit 123, an annotation correction unit 124, and a display unit 125. The automatic frame extraction unit 121, the key frame processing unit 122, the automatic annotation unit 123, the annotation correction unit 124, and the display unit 125 can be connected through a communication link 126. The communication link 126 can be an internal bus or other communication links such as the Internet. This application does not make specific limitations.

[0081] The automatic frame extraction unit 121 includes a frame extraction unit 1211 and a frame rate adjustment unit 1212. The frame rate adjustment unit is used to dynamically adjust the frame extraction rate of each video segment according to the video content of the unlabeled video. The frame extraction unit 1211 is used to extract frames from the unlabeled video according to the frame extraction rate to obtain multiple video frames. The frame extraction rate used by the frame extraction unit 1211 can be the frame extraction rate output by the frame rate adjustment unit 1212 or the frame extraction rate manually set by the user.

[0082] The keyframe processing unit 122 includes a keyframe selection unit 1221 and a keyframe annotation unit 1222. The keyframe selection unit 1221 is used to determine at least one video frame as a keyframe among multiple video frames and present the keyframe to the user through the display unit 1223. The user can manually annotate the targets in the keyframe to obtain the user annotation result, or the keyframe can be automatically annotated by the keyframe annotation unit 1222. The user can manually determine whether the keyframe annotation unit 1222 has annotated correctly to obtain the user annotation result. The user annotation result shows the targets annotated by the user in the keyframe. For example, if the user needs to train a network model for pedestrian recognition, the keyframe processing unit 122 will annotate pedestrians in the video frame. If the user needs to train a network model for vehicle recognition, the keyframe processing unit 122 will annotate vehicles in the video frame.

[0083] The automatic annotation unit 123 includes an annotation unit 1231 and multiple calculation units 1232. Figure 1 The example uses three operation units (operation units 1232a~1232c). This application does not limit the number of operation units 1232. The annotation unit 1231 is used to call multiple operation units 1232 to annotate the same target in the remaining non-key frames based on the key frame and the user annotation result. If the user annotates all pedestrians in the key frame using the target box, then the automatic annotation unit 123 also annotates all pedestrians in other video frames using the target box. If the user annotates pedestrian A using the center point, then the automatic annotation unit 123 also annotates pedestrian A using the center point in other video frames. It should be noted that a computing unit can be understood as a program or a processing unit with independent hardware and software systems and computing functions, such as a processor, virtual machine, or physical machine. The specific form of the computing unit can be determined according to business needs. If the automatic labeling unit 123 is deployed in a cloud data center, then the computing unit 1232 can be any physical machine or virtual machine in the cloud data center. If the automatic labeling unit 123 is deployed on a multi-chip, multi-core computing device, the computing unit 1232 can be a processor of the computing device. It should be understood that the above examples are for illustration, and this application does not limit the form of the computing unit 1232.

[0084] The annotation correction unit 124 is used to generate modification information corresponding to the annotation results through the annotation correction model, correct the annotation results, and obtain new annotation results. The new annotation results generated by the annotation correction unit 124 can be used as new keyframes and new user annotation results. These are then input into the keyframe processing unit 122 to modify the recommended user annotation results, and into the automatic annotation unit 123 to modify the automatic annotation results, and so on, continuously improving the annotation accuracy. The annotation correction model can be obtained by training an AI model using multiple known annotation results and corresponding known modification information as training samples.

[0085] The display unit 125 is used to present a portion of the interface to the user to obtain user requirements, such as the frame extraction method, annotation format, keyframes, and the targets to be annotated in the keyframes. It should be understood that the automatic frame extraction unit 121, keyframe processing unit 122, and automatic annotation unit 123 are all connected to the display unit 125. Each unit can receive user requirements from the display unit 125 via the network and perform corresponding operations in response. For example, the display unit 125 can display keyframes to the user, receive user-generated annotations, and feed them back to the automatic annotation unit 123, enabling it to annotate non-keyframes based on the keyframes and user annotations. Alternatively, the display unit 125 can display annotation results to the user, receive manually modified annotations, and feed them back to the automatic annotation unit 123, enabling it to re-annotate non-keyframes based on the modified annotations. Through the display unit 125, the video annotation system can continuously obtain user modification information, thereby continuously improving annotation accuracy.

[0086] In summary, the video annotation system provided in this application can automatically annotate other non-keyframes based on the annotation results of keyframes, thereby obtaining the annotation results of the entire video. The annotation results of keyframes can be annotated by the user or automatically recommended to the user by the video annotation system and then confirmed by the user. This means that during the entire annotation process, the user only needs to annotate the keyframe once, or even just confirm whether the keyframe is annotated correctly once, to obtain the complete annotation results of the video, which greatly reduces the user's annotation operations, improves video annotation efficiency, and enhances the user experience.

[0087] The following, with reference to the accompanying drawings, details the specific steps and processes by which the video annotation system provided in this application identifies overloaded vehicles.

[0088] like Figure 3 As shown, this application provides a video annotation method, which includes the following steps:

[0089] S310: Extract multiple video frames from an unlabeled video. For a description of the unlabeled video, please refer to [link / reference]. Figure 1 The specific implementation details are not repeated here, and this step can be performed by... Figure 1 The automatic frame extraction unit 121 in the embodiment can extract multiple video frames from the unlabeled video in a variety of ways.

[0090] Optionally, frame extraction can be performed using a "fixed frame rate," meaning multiple video frames are extracted from the unlabeled video at a fixed extraction rate. This fixed frame rate can be a user-defined rate, such as setting a frame extraction rate of 10fps to extract 36,000 video frames from an unlabeled video with a total duration of one hour. Alternatively, the fixed frame rate can be determined based on historical frame extraction rates. For example, if the unlabeled video content is surveillance footage of an intersection, and historical records show that the corresponding historical frame extraction rate for that intersection's surveillance footage is 20fps, then multiple video frames can be extracted from the unlabeled video at a frame extraction rate of 20fps. It should be understood that the above examples are for illustrative purposes only and this application does not impose specific limitations.

[0091] Optionally, frame extraction can be performed manually, that is, frame extraction is performed according to the frame extraction rate input by the user. Specifically, the frame extraction interface can be presented to the user through the display unit 125, such as displaying a video playback window. The user can set the frame rate at any video segment position and perform segmented frame extraction by inputting the start time, end time and frame extraction rate of the video segment.

[0092] For example, Figure 4 This is an exemplary interface that the display unit 125 can display after the user selects the "manual frame extraction" method. This interface may include a video area 410, a playback control 420, and a frame extraction input box 430. The video area 410 displays the video frame of the unlabeled video, the playback control 420 controls the playback progress, and the frame extraction input box 430 obtains the start time, end time, and corresponding frame extraction rate of the video segment input by the user. In one application scenario, if the user needs to label non-motorized vehicles, they can play the video through the playback control 420. If the non-motorized vehicle first appears in the video at 00:00:01 and last appears at 00:00:06, the user can enter 00:00:01 for the start time and 00:00:06 for the end time, with a frame extraction rate of 10fps. However, the frame extraction rate for a start time of 00:00:00 and an end time of 00:00:01 is 0fps, meaning frame extraction is not required. It should be understood that... Figure 4 The user interface shown is for illustrative purposes only and is not intended to limit the scope of this application.

[0093] Optionally, frame extraction can be performed using an "automatic frame extraction" method, which dynamically adjusts the frame extraction rate based on the video content and performs dynamic frame extraction on unlabeled videos. Specifically, the image similarity between adjacent frames can be determined first, and then the image similarity can be compared with a pre-stored similarity mapping relationship to determine the frame rate corresponding to that image similarity. For example, when the similarity is 0.1, the frame rate is 1, and when the similarity is 0.2, the frame rate is 2. Similarly, the change in objects between frames can be determined first, and then the change in objects can be compared with a pre-stored change mapping relationship to determine the frame rate corresponding to that change in objects between frames. The specific form of the pre-stored similarity mapping relationship and change mapping relationship can be a mathematical formula, a mapping table, etc., and this application does not limit this. It should be understood that when moving objects exist in a video, there will be differences in grayscale between adjacent frames or multiple adjacent frames. Based on this, the rate of change of objects between frames can be obtained. Of course, known video segments and corresponding known inter-frame object changes can also be used as training samples to train a deep neural network. The trained model can infer the corresponding inter-frame object changes based on the input video segments. Alternatively, a sample set can be generated from known unlabeled videos and their corresponding frame extraction rates, and this sample set can be used to train the AI ​​model. By inputting the currently processed unlabeled video into the trained AI model, the frame extraction rate corresponding to each video segment can be obtained. It should be understood that using automatic frame extraction to extract frames from unlabeled videos can dynamically adjust the frame extraction rate according to the video content, avoiding the extraction of too many redundant frames or the inability to extract video frames containing the target. This ensures that the final labeled video frames can be used as high-quality training samples for training the AI ​​model, improving the user experience.

[0094] S320: Display at least one video frame from a plurality of video frames to the user through a display interface. For ease of understanding, this at least one video frame will be referred to as a keyframe below. It should be understood that the keyframe can be... Figure 1 The key frame selection unit 1221 in the key frame processing unit 122 of the embodiment is determined from multiple video frames extracted by the automatic frame extraction unit 121.

[0095] In one embodiment, keyframes can be manually selected by the user. Specifically, the display unit 125 can present multiple video frames obtained after frame extraction to the user, from which the user selects one or more video frames as keyframes in preparation for keyframe annotation. Keyframes can also be automatically selected by the keyframe selection unit 1221. Specifically, a fixed frame number among the multiple video frames can be used as a keyframe, such as the first or last frame among the multiple video frames. Alternatively, keyframes among the multiple video frames can be determined according to a keyframe selection model. The keyframe selection model can be obtained by using multiple known video frames and corresponding known keyframes as training samples to train an AI model. The trained keyframe selection model can output the corresponding keyframes based on the input video frames.

[0096] S330: Obtain user annotation results for at least one video frame displayed on the screen, wherein the user annotation results include the image region of the target in at least one video frame.

[0097] Optionally, the annotation result may include a bounding box. Specifically, the form of the bounding box may differ depending on the AI ​​algorithm used for video annotation. For example, the shape of the bounding box may be a rectangle, circle, or ellipse, or other preset shapes. When the video annotation system uses an AI model based on mask RCNN to precisely locate the target, the shape of the bounding box is close to the shape of the target, such as the shape of the target's outline; that is, the bounding box is the outline of the target. Regardless of the algorithm used for video annotation, the bounding box can be a line bounding box or a scatter bounding box composed of multiple discrete points; this application does not impose specific limitations. It should be noted that the bounding box can be a line bounding box, specifically including line bounding boxes formed by solid lines or dashed lines, for example... Figure 5 The solid-line rectangle 510 is shown. In some implementations, the target box can also be a scatter plot formed by multiple discrete points. This application does not limit this.

[0098] Optionally, the annotation results may include center points, for example Figure 5The center point 520 shown can be determined in several ways. First, a rectangular bounding box can be used to select the target. Then, other information can be combined to obtain the target's center point. For example, the center point can be determined using object centroid detection. Based on the video frame containing the target selected by the rectangular bounding box and information from other wireless sensors, a unique point (mass) whose position does not change due to rigid motion is detected using methods such as maximum likelihood estimation and weighting. The mass point's position represents its location in the video frame. Alternatively, the center point can be determined through 3D detection. Using point cloud maps, adding object height or depth, etc., the original 2D object detection is converted to 3D object detection, resulting in a 3D model of the target object. A specific location within this 3D model is then determined as the center point, representing the target's position. Finally, the center point can be determined directly from the rectangular bounding box on the 2D pixel screen, combined with the video content. For example, when the target is a motor vehicle, vehicles traveling straight are generally aligned horizontally or vertically, so the midpoint of the lower edge of the rectangle is often chosen as the center point of the target. For vehicles in the foreground, due to their larger size and perspective distortion, the lower right corner of the rectangle is often chosen as the center point. For vehicles in the background, due to their smaller size and smaller rectangle, the center point of the rectangle is often chosen as the center point. It should be understood that the methods for obtaining the center point listed above are merely illustrative; other methods can also be used to obtain the center point, and this application does not impose any specific limitations on them.

[0099] Optionally, the annotation results may include a mask, such as Figure 5 The mask 530 is shown. For a detailed description of the mask, please refer to the aforementioned terminology explanation, which will not be repeated here.

[0100] Optionally, the annotation results may also include one or more of the following: the target bounding box, the center point, and the mask, for example... Figure 5 The annotation result shown is 540, which includes the rectangle, center point, and mask.

[0101] It should be understood that, due to the variety of annotation formats, in order to make this application more understandable, the annotation results described below will mostly use annotation boxes as examples for explanation, but this application does not limit the annotation format.

[0102] In one embodiment, user annotation results can be obtained through automatic annotation. Specifically, after presenting a keyframe to the user, recommended annotation results for that keyframe can be generated simultaneously, automatically annotating multiple targets that the user may need to annotate. The user can automatically annotate the targets without performing any operation, and the recommended annotation results can be directly used as the user's annotation results, thereby improving the user experience and increasing annotation efficiency.

[0103] In practical implementation, recommended annotation results for the keyframe can be generated based on the video content. For example, surveillance videos on highways typically annotate vehicles as targets. Therefore, when an unannotated video is a highway surveillance video, vehicles can be selected in the keyframe to recommend annotation results to the user. Alternatively, a few keywords can be obtained from the user, such as inputting "vehicles." The system can then annotate the keywords and select all vehicles in the keyframe to obtain the user's annotation results. Furthermore, keyframes from unannotated videos can be input into a recommendation annotation model to obtain user annotation results. These user annotation results can be from an AI model. Specifically, known keyframes from unannotated videos and corresponding known user annotation results can be used as training samples. A deep neural network can be trained using computer vision algorithms such as objectness algorithms to obtain a trained recommendation annotation model. Alternatively, other mature computer vision algorithms can be used to implement the above functions. It should be understood that the above examples are for illustrative purposes only and do not constitute specific limitations.

[0104] In one embodiment, user annotation results can be obtained through human-computer interaction. Specifically, the recommended annotation results can be presented to the user through a display unit for the user to select, and the user annotation result selected by the user from the recommended annotation results can be obtained. For example... Figure 5 As shown, the keyframe processing unit 122 can generate multiple recommended targets on the keyframe, and transmit the recommendation annotation results containing multiple recommended targets through... Figure 5 The displayed interface allows users to easily select targets simply by clicking anywhere within a target box, eliminating the need for mouse or touch selection via dropdown menus or outlines. This enhances user experience and improves annotation efficiency. In practice, after recommending annotation results to the user, the user can manually correct these results through the interface, further improving annotation accuracy.

[0105] In one embodiment, user annotation results can be obtained through manual annotation. Specifically, after presenting keyframes to the user, the user can also manually draw the target box, center point, or mask. This application does not limit this.

[0106] For example, before displaying a keyframe to the user, the display unit 125 can first obtain the annotation method for this keyframe annotation from the user. If the user selects "automatic annotation," the video annotation system can automatically generate the user's annotation results on the keyframe without requiring any annotation action from the user, greatly improving the user experience and annotation efficiency. If the user selects "human-computer interaction," the video annotation system can recommend annotation results to the user. The user only needs to manually confirm the recommended annotation results, such as moving the mouse to any position on the object to automatically generate the center point, without having to manually find the object's center point, ensuring annotation accuracy while improving annotation efficiency. If the user selects "manual annotation," the user can annotate the keyframe themselves, such as manually drawing a target box on the object. The keyframe annotation unit can record the user's drawing information and use it as a new sample to train the recommended annotation model.

[0107] S340: Based on the user's annotation results, obtain the annotation results for other video frames in the multiple video frames. That is, annotate the remaining non-keyframes. The annotation results include the image regions of the target desired by the user in other video frames. It should be understood that this step can be performed by... Figure 1 The automatic annotation unit 123 in the embodiment is implemented. In specific implementation, the user can choose to export the annotation results of other video frames in a certain format, such as exporting multiple video frames in JPEG format and the annotation file in XML format, and then use them as training samples for training the AI ​​model. It should be understood that the above format is for illustrative purposes only and should not constitute a specific limitation.

[0108] In one embodiment, the user can choose between forward annotation or reverse annotation for automatic annotation. Forward annotation refers to automatically annotating video frames after the keyframe, while reverse annotation refers to automatically annotating video frames before the keyframe. For example, if a user manually extracts frames and selects a frame as a keyframe near the end of the video, and then manually annotates a target (either through human-computer interaction or automatic annotation), the user can then choose reverse annotation to automatically annotate targets in other frames before that keyframe. Similarly, if the user selects a frame as a keyframe at the beginning of the video, they can choose forward annotation to automatically annotate targets in other frames after that keyframe. It is understandable that by annotating one or more keyframes and then having all frames automatically annotated, the user can significantly improve video annotation efficiency and enhance the user experience.

[0109] For example, such as Figure 6As shown, assuming the user uses frame 1 as a keyframe and annotates the truck with a rectangle and the bus with a center point, and assuming the user selects the forward annotation method for automatic annotation, then the automatic annotation unit 123 can automatically annotate the following frames 2 and 3, similarly annotating the truck in frame 2 with a rectangle and the bus in frame 2 with a center point, and similarly annotating the truck in frame 3 with a rectangle and the bus in frame 3 with a center point. It should be understood that... Figure 6 For illustrative purposes only, this application does not limit the automatic annotation interface.

[0110] In one embodiment, the automatic annotation unit can input keyframes and user annotation results into the automatic annotation model. The automatic annotation model can infer from non-keyframes based on the input keywords and user annotation results to obtain image regions of targets that meet preset conditions, thereby determining the position and size of the target bounding box, center point, or mask, and obtaining annotation results for other video frames. This automatic annotation model can be trained using mature computer vision algorithms in the industry, such as target tracking, target recognition, target detection, and segmentation. This application does not limit the training method of the automatic annotation model.

[0111] In specific implementation, such as Figure 1 As shown, the automatic annotation unit 123 can simultaneously call multiple computing units 1232 to execute inference algorithms on multiple video frames at the same time, and then superimpose the computing results to generate annotation results. It can be understood that using multiple computing units to process the inference algorithm in parallel can improve the processing efficiency of the automatic annotation unit 123, improve the video annotation efficiency, and enhance the user experience.

[0112] Optionally, the automatic annotation unit 123 can divide the video frame to be annotated into multiple video segments and randomly assign these video segments to various processing units. Each processing unit processes one video segment, where a video segment may include at least two video frames. For example, processing unit 1 processes video frames 1 to 10 to generate annotation results for video frames 1 to 10, and processing unit 2 processes video frames 11 to 20 to generate annotation results for video frames 11 to 20. The video segments processed by each processing unit can be randomly assigned by the automatic annotation unit 123, and this application does not limit this. In this way, the automatic annotation unit 123 can process multiple video segments simultaneously, improving the processing efficiency of video annotation.

[0113] Optionally, the automatic annotation unit 123 can randomly assign the video frames to be annotated to each computing unit. Each computing unit can process one video frame, that is, one computing unit uses the automatic annotation model to infer on one video frame and outputs the annotation result of the video frame. In this way, the automatic annotation unit can process multiple video frames at the same time, improving the processing efficiency of video annotation. Furthermore, since each computing unit processes each video frame independently, forward annotation and reverse annotation can be realized, improving the user experience.

[0114] Optionally, the automatic annotation unit 123 can assign the same video frame to multiple processing units based on the user's annotation results. Each processing unit can load one algorithm, and multiple processing units can process the same video frame simultaneously. Finally, the outputs of the multiple processing units are superimposed to output the annotation result. For example, suppose processing unit 1 loads an algorithm for annotating target boxes on the video frame, and processing unit 2 loads an algorithm for annotating masks on the video frame. Then, processing units 1 and 2 process frame A1 simultaneously. Processing unit 1 outputs frame A11 with target boxes annotated, and processing unit 2 outputs frame A12 with masks annotated. Finally, the automatic annotation unit 123 can superimpose frames A11 and A12 to obtain frame A1 with both target boxes and masks annotated. It should be understood that the above example is for illustration; the automatic annotation unit 123 can randomly assign video frames to processing units and is not inherently limited.

[0115] In practice, a processing unit can be a program or process, or an independent processor with its own hardware and software system. Processing units can be connected via wired or wireless means, and the physical form of the processing unit can be determined based on its processing power.

[0116] It should be noted that the computer vision algorithms such as object detection, object tracking, and segmentation algorithms in the embodiments of this application can be any of the existing neural network models with superior performance in the industry, such as: the You Only Look Once: Unified, Real-Time Object Detection (YOLO) model, the Single Shot multi box Detector (SSD) model, the Region Convolutional Neural Network (RCNN) model, or the Fast Region Convolutional Neural Network (Fast-RCNN) model, etc., and this application does not make specific limitations.

[0117] In one embodiment, after step S340, the keyframe processing unit 122 can receive modification information of the annotation results, generate new keyframes and new user annotation results based on the modification information, and the automatic annotation unit 123 can modify the annotation results of non-keyframes based on the new keyframes and new user annotation results, and so on, continuously improving the annotation accuracy.

[0118] Optionally, the modification information can be generated by the user through the display unit 125. Specifically, after the automatic annotation unit 123 processes a video frame, it can present the automatic annotation result to the user through the display unit 125. If no correction request is received from the user, the automatic annotation unit 123 can continue to process the next video frame until the video playback ends. If the user believes that the annotation result of a certain frame is incorrect and needs to be corrected when seeing the annotation result, the user can pause the playback, switch to the annotation correction mode, and then correct the position or size of the target box, center point, or mask by dragging the mouse. After the display unit receives the user's modification information, the automatic annotation unit can take the video frame corresponding to the modification information as a new keyframe and continue to perform automatic annotation based on the modified user annotation result.

[0119] Still with Figure 6 Taking the application scenario shown as an example, suppose the automatically labeled frame 3 is as follows: Figure 7 As shown, the center point of the rectangle used to label the truck and the center point of the rectangle used to label the bus in frame 3 have both shifted. After the user corrects frame 3 and obtains a new frame 3, frame 3 can be used as a new keyframe. The automatic labeling unit 123 labels frame 4 according to the new user labeling results, obtaining the following... Figure 7 The annotation results for frame 4 shown should be understood as follows: Figure 7 For illustrative purposes only, this application does not limit the interface form.

[0120] Optionally, the information can also be modified. Figure 1The annotation correction unit 124 shown can obtain the modification information of the annotation result through the annotation correction model. The annotation correction model can be obtained by training the AI ​​model in advance using known annotation results and corresponding known modification information as training samples. In simple terms, if the user selects the "automatic optimization" function, the annotation results generated by the automatic annotation unit 123 after automatically annotating non-keyframes will be input into the annotation correction unit 124 to automatically detect whether the annotations are incorrect. If an error is found, corresponding modification information will be generated and fed back to the keyframe processing unit 122, thereby generating new keyframes and new user annotation results. The automatic annotation unit 123 can generate new annotation results based on the new keyframes and new user annotation results, so that the automatic frame extraction unit, keyframe processing unit, and annotation correction unit form a closed loop, continuously optimizing the annotation results and improving annotation accuracy. In specific implementation, the user can choose whether to use the annotation correction unit 124 for annotation correction according to the training objective, and can also choose the number of optimizations. For example, if the required annotation accuracy of the training objective is low, the user can choose not to use the annotation correction unit 124 for annotation correction, thereby achieving the goal of quickly obtaining video annotation results. Or, if the required annotation accuracy of the training objective is high, the user can choose to use the annotation correction unit 124 for annotation correction and increase the number of corrections, such as setting it to 5 times. It should be understood that the above examples are for illustration and do not constitute specific limitations.

[0121] To facilitate understanding of the beneficial effects of the video annotation method proposed in the embodiments of this application, exemplarily, Figure 8 This is an exemplary display interface of a video annotation system provided in this application. Before step S310, the display unit can present to the user... Figure 8 The interface shown, exemplarily, may include a title bar 810, a file loading bar 820, a frame extraction settings bar 830, a keyframe selection settings bar 840, a keyframe annotation settings bar 850, an annotation format settings bar 860, an automatic optimization settings bar 870, and a save control 880.

[0122] The title bar 810 includes the interface name and window adjustment controls, for example, Figure 8 The title bar 810 shown includes a close control for closing the window, a shrink control for shrinking the window, and a full-screen control for maximizing the window. In specific implementations, the title bar 810 may also include more elements. The title bar 810 may be displayed as text information, icons, or other forms, and this application does not impose any specific limitations.

[0123] The file loading bar 820 is used to search for video files that match the characters entered by the user, i.e., the unlabeled videos mentioned above.

[0124] The frame extraction setting bar 830 is used to extract frames from the aforementioned video file according to the frame extraction method selected by the user, obtaining multiple video frames. The display unit 125 can send the frame extraction method selected by the user to the automatic frame extraction unit 121, so that the automatic frame extraction unit 121 can extract frames according to the frame extraction method selected by the user. Specifically, if the user selects "automatic frame extraction", the automatic frame extraction unit 121 can dynamically adjust the frame extraction rate according to the video content and dynamically extract frames from unlabeled videos; if the user selects "fixed frame rate", the automatic frame extraction unit 121 can extract frames according to a fixed frame rate; if the user selects "manual frame extraction", the display unit 125 can first display a video playback window to the user, for example... Figure 4 The video playback interface shown then sends the start time, end time, and frame rate of the video segment input by the user to the automatic frame extraction unit 121, so that the automatic frame extraction unit can perform frame extraction. It should be understood that the specific implementation of the above frame extraction method can refer to the aforementioned step S310 and its optional steps, and will not be elaborated here.

[0125] The keyframe selection setting field 840 is used to select keyframes from multiple video frames obtained after frame extraction according to the keyframe selection method selected by the user. The display unit 125 can send the keyframe selection method selected by the user to the keyframe processing unit 122, so that the keyframe processing unit 122 can select one or more keyframes from the multiple video frames for annotation. Specifically, if the user selects "automatic selection", the keyframe processing unit 122 can use video frames with fixed frame numbers as keyframes, such as the first frame or the last frame, or input multiple video frames obtained after frame extraction into the keyframe selection model, and determine the keyframes according to the output of the model. For details, please refer to the aforementioned step S320 and its optional steps, which will not be elaborated here.

[0126] The keyframe annotation setting bar 850 is used to annotate keyframes according to the keyframe annotation method selected by the user and obtain the user's annotation results. Specifically, the display unit 125 can send the keyframe annotation method selected by the user to the keyframe processing unit 122, so that the keyframe processing unit 122 can annotate the keyframes based on it. If the user selects "automatic annotation", the video annotation system can automatically generate the user's annotation results on the keyframes without requiring any annotation action from the user, which greatly improves the user experience and annotation efficiency. If the user selects "human-computer interaction", the video annotation system can recommend annotation results to the user. The user only needs to manually confirm the recommended annotation results, such as moving the mouse to any position of the object to automatically generate the center point, without having to manually find the center point of the object, ensuring annotation accuracy while improving annotation efficiency. If the user selects "manual annotation", the user can annotate the keyframes by himself, such as manually drawing the target box on the object, obtaining the center point drawn manually at the center of the object, etc. This application does not limit the specific form of manual annotation.

[0127] The annotation format setting bar 860 is used to automatically annotate non-key frames in multiple video frames obtained after frame extraction according to the annotation format selected by the user. For example, the annotation format may include rectangles, center points, masks, polygons, etc. Of course, other annotation formats may also be included, which are not specifically limited in this application.

[0128] The automatic optimization setting bar 870 is used to correct the annotation results based on the user's selection. If the user selects "No", the annotation results of the automatic annotation unit 123 will be directly output. If the user selects "Yes", the video annotation system will call the annotation correction unit 124 to perform annotation correction. Furthermore, the automatic optimization setting bar 870 may also include annotation correction levels. The user can select the number of annotation corrections according to the training objective. If the training objective requires samples with higher annotation accuracy, the number of annotation corrections can be increased. This application does not limit the specific form of the automatic optimization setting bar 870.

[0129] The save control 880 is used to receive user operations (such as mouse clicks or touch operations). In response to the detected user operation, the display unit 125 can send the options entered by the user in the file loading bar 820, frame extraction setting bar 830, keyframe selection setting bar 840, keyframe annotation setting bar 850, annotation format setting bar 860, and automatic optimization setting bar 870 to the corresponding unit module for processing.

[0130] It should be understood that after the user clicks the save control 880, the display unit 125 can continue to display the next interface according to the user's selection. For example, if the user selects "manual selection" in the keyframe selection settings bar 840, the display unit 125 can continue to display the following: Figure 4 As shown in the interface, if the user selects "Human-Computer Interaction" in the keyframe annotation settings 850, the display unit 125 can continue to display, as shown in the image. Figure 5 As shown in the interface, after obtaining the user annotation results, the display unit 125 can display the following: Figure 6 The automatic annotation interface shown presents the annotation results to the user. If the user selects "Yes" in the automatic optimization settings section 870, the display unit can also display... Figure 7 The annotation modification interface shown allows users to modify the annotation results, generate new keyframes and new user annotation results. For details, please refer to the previous content, which will not be repeated here.

[0131] Understandable Figure 8 The illustration merely shows the interface presented to the user by the display unit 150 before step S310 and should not be construed as limiting the embodiments of this application.

[0132] In summary, the video annotation method provided in this application can automatically annotate other non-keyframes based on the annotation results of keyframes, thereby obtaining the annotation results of the entire video. The annotation results of keyframes can be user-annotated or automatically recommended to the user by the video annotation system and then confirmed by the user. This means that during the entire annotation process, the user only needs to annotate the keyframe once, or even just confirm whether the keyframe is annotated correctly once, to obtain the complete video annotation results, which greatly reduces the user's annotation operations, improves video annotation efficiency, and enhances the user experience.

[0133] The methods of the embodiments of this application have been described in detail above. In order to facilitate better implementation of the above-described solutions of the embodiments of this application, relevant equipment for cooperating in implementing the above solutions is also provided below.

[0134] Figure 9 This application provides a video annotation system, such as... Figure 9 As shown, the system includes:

[0135] Automatic frame extraction unit 121 is used to extract multiple video frames from unlabeled video;

[0136] The keyframe processing unit 122 is used to display at least one video frame from a plurality of video frames to a user through a display interface.

[0137] Automatic annotation unit 123 is used to obtain user annotation results of at least one video frame in the display interface, wherein the user annotation results include the image region of the target in at least one video frame;

[0138] The annotation correction unit 124 is used to obtain the annotation results of other video frames in multiple video frames based on the user annotation results, wherein the annotation results include the image regions of the target in other video frames.

[0139] In one embodiment, the automatic frame extraction unit 121 is used to extract multiple video frames from the unlabeled video based on the video content of the unlabeled video, wherein the image similarity between the multiple video frames is lower than a first threshold, or the object change between the multiple video frames is higher than a second threshold.

[0140] In one embodiment, at least one video frame is the first or last frame among a plurality of video frames; or, at least one video frame is obtained by inputting a plurality of video frames into a keyframe selection model, wherein the keyframe selection model is obtained by training a neural network model using a plurality of known video frames and corresponding known keyframes as training samples.

[0141] In one embodiment, the keyframe processing unit 122 is used to input at least one video frame into the recommendation annotation model to obtain recommendation annotation results, wherein the recommendation annotation results include image regions of at least one recommended target in at least one video frame; the keyframe annotation unit is used to display the recommendation annotation results to the user through a display interface and obtain the user annotation results selected by the user in the recommendation annotation results.

[0142] In one embodiment, the annotation result includes one or more of the following: target bounding box, center point, and mask.

[0143] In one embodiment, the automatic annotation unit 123 is used to call multiple computing units to process other video frames in parallel based on the user annotation results, and obtain annotation results for other video frames. In this case, one computing unit processes one video frame, or at least one computing unit processes one video frame, and each computing unit in the at least one computing unit generates an annotation result.

[0144] In one embodiment, the automatic annotation unit 123 is used to annotate video frames after or before keyframes based on user annotation results to obtain annotation results.

[0145] In one embodiment, the system further includes a labeling correction unit 124. The keyframe processing unit 122 is used to receive modification information of the labeling results from the user through the display interface. The modification information comes from the user's modification of the image region of the target in other video frames. Alternatively, the labeling correction unit 124 is used to obtain the modification information of the labeling results through a labeling correction model. The labeling correction model is obtained by training a neural network model using multiple known labeling results and corresponding known modification information as training samples. The automatic labeling unit 123 is used to modify the labeling results of other video frames in multiple video frames according to the modification information.

[0146] In one embodiment, the annotation results are used for learning by the AI ​​model.

[0147] It should be understood that the video annotation system 120 in this embodiment can be a physical machine, such as an x86 server, or a virtual machine, or a computer cluster composed of multiple physical machines or virtual machines. Furthermore, the internal unit modules of the video annotation system 120 can be divided in various ways. Each module can be a software module, a hardware module, or a combination of software modules and hardware modules. This application does not limit these. Figure 9 This is an exemplary division method, and this application does not impose specific limitations. The video annotation system 120 according to an embodiment of this application can correspondingly execute the method described in this embodiment, and the above and other operations and / or functions of each unit in the video annotation system 120 are respectively for implementing... Figures 3 to 8For the sake of brevity, the corresponding processes of each method in the code will not be elaborated here.

[0148] In summary, this application provides a video annotation system that can automatically annotate other non-keyframes based on the annotation results of keyframes, thereby obtaining the annotation results of the entire video. The annotation results of keyframes can be user-annotated or automatically recommended to the user by the video annotation system and then confirmed by the user. This means that during the entire annotation process, the user only needs to annotate a keyframe once, or even just confirm whether the keyframe is annotated correctly once, to obtain the complete video annotation results, which greatly reduces the user's annotation operations, improves video annotation efficiency, and enhances the user experience.

[0149] Figure 10 This is a schematic diagram of the structure of a computing device 1000 provided in this application. The computing device 1000 can be the video annotation system 120 mentioned above. For example... Figure 10 As shown, the computing device 1000 includes a processor 1010, a communication interface 1020, and a memory 1030. The processor 1010, communication interface 1020, and memory 1030 can be interconnected via an internal bus 1040, or they can communicate via wireless transmission or other means. This embodiment uses a connection via bus 1040 as an example. Bus 1040 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Bus 1040 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, Figure 10 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0150] Processor 1010 may consist of at least one general-purpose processor, such as a central processing unit (CPU), or a combination of a CPU and hardware chips. The hardware chips may be application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or combinations thereof. The PLDs may be complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), generic array logic (GALs), or any combination thereof. Processor 1010 executes various types of digital storage instructions, such as software or firmware programs stored in memory 1030, enabling computing device 1000 to provide a variety of services.

[0151] The memory 1030 stores program code, which is executed by the processor 1010 to perform the processing steps of the video annotation system described in the above embodiment. The program code may include one or more software modules, which can be... Figure 9 The software modules provided in the embodiment, such as the automatic frame extraction unit, the keyframe processing unit, and the automatic annotation unit, are as follows: the automatic frame extraction unit is used to extract multiple video frames from the unannotated video; the keyframe processing unit is used to display at least one video frame from the multiple video frames to the user through a display interface and obtain the user annotation results for the at least one video frame annotated by the user in the display interface; and the automatic annotation unit is used to obtain the annotation results for the other video frames from the multiple video frames based on the annotation results. Specifically, these modules can be used to execute... Figure 3 Steps S310-S340 and their optional steps in the embodiment can also be used to perform Figures 3-8 Other steps performed by the video annotation system as described in the embodiments will not be repeated here.

[0152] The memory 1030 may include volatile memory, such as random access memory (RAM); the memory 1030 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); the memory 1030 may also include combinations of the above types. The memory 1030 may store program code, specifically including code for execution. Figures 3-8 The program code for the other steps described in the embodiments will not be repeated here.

[0153] The communication interface 1020 can be a wired interface (such as an Ethernet interface), an internal interface (such as a high-speed serial computer expansion bus (Peripheral Component Interconnect express, PCIe) bus interface), a wired interface (such as an Ethernet interface), or a wireless interface (such as a cellular network interface or a wireless LAN interface) for communicating with other devices or modules.

[0154] It needs to be explained that, Figure 10 This is merely one possible implementation of an embodiment of this application. In practical applications, the computing device 1000 may include more or fewer components, and this is not a limitation. For content not shown or described in the embodiments of this application, please refer to the foregoing. Figures 3-8 The relevant descriptions in the embodiments will not be repeated here.

[0155] It should be understood that Figure 10 The computing device shown can be implemented as a general-purpose physical server, such as an ARM server or an x86 server, or it can be implemented as a virtual machine based on a general-purpose physical server combined with NFV technology. A virtual machine refers to a complete computer system with complete hardware system functions simulated by software and running in a completely isolated environment. It can also be a computer cluster consisting of at least one physical server or virtual machine. This application does not make any specific limitations.

[0156] This application also provides a computer-readable storage medium storing instructions that, when executed on a processor. Figures 3-8 The method flow shown is thus implemented.

[0157] This application also provides a computer program product that, when run on a processor, provides a solution for... Figures 3-8 The demonstrated method and process are thus implemented.

[0158] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs, DVDs), or semiconductor media. Semiconductor media can be SSDs.

[0159] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A video annotation method, characterized by, The method comprises: determining the image similarity between adjacent frames in the unlabeled video and the object change amount, comparing the image similarity with a pre-stored similarity mapping relationship, comparing the object change amount with a pre-stored change amount mapping relationship, determining the frame rate corresponding to the image similarity and the object change amount, and dynamically adjusting the frame extraction rate of each video segment in the unlabeled video; extracting a plurality of video frames from each video segment in the unlabeled video using the frame extraction rate, wherein the image similarity between the plurality of video frames extracted from the unlabeled video is lower than a first threshold, or the object change amount between the plurality of video frames is higher than a second threshold; displaying at least one video frame in the plurality of video frames to the user through a display interface; obtaining a user annotation result of the user annotating the at least one video frame in the display interface, wherein the user annotation result includes an image region of a target in the at least one video frame; obtaining an annotation result of other video frames in the plurality of video frames according to the user annotation result, wherein the annotation result includes an image region of the target in the other video frames.

2. The method of claim 1, wherein, The at least one video frame is the first frame or the last frame in the plurality of video frames; or The at least one video frame is obtained by inputting the plurality of video frames into a key frame selection model, and the key frame selection model is obtained by training a neural network model using a plurality of known video frames and corresponding known key frames as training samples.

3. The method of claim 2, wherein, The obtaining of the user annotation result of the user annotating the at least one video frame in the display interface comprises: inputting the at least one video frame into a recommended annotation model to obtain a recommended annotation result, wherein the recommended annotation result includes an image region of at least one recommended target in the at least one video frame; displaying the recommended annotation result to the user through the display interface, and obtaining a user annotation result selected by the user in the recommended annotation result.

4. The method of claim 3, wherein, The annotation result includes one or more of a target box, a center point, and a mask.

5. The method of claim 4, wherein, The obtaining of the annotation result of the other video frames in the plurality of video frames according to the user annotation result comprises: According to the user annotation result, a plurality of computing units are called and the other video frames are processed in parallel to obtain the annotation result of the other video frames, wherein one computing unit processes one video frame, or at least one computing unit processes one video frame, and each computing unit in the at least one computing unit generates one annotation result.

6. The method of claim 5, wherein, The obtaining of the annotation result of the other video frames in the plurality of video frames according to the user annotation result comprises: According to the user annotation result, the video frames after or before the at least one video frame are annotated to obtain an annotation result.

7. The method according to any one of claims 1 to 6, characterized in that, The method further comprises: receiving modification information of the user on the annotation result through the display interface, and the modification information is from the user's modification of the image region of the target in the other video frames; or The modification information of the labeling result is obtained by using a labeling correction model, wherein the labeling correction model is obtained by using a plurality of known labeling results and corresponding known modification information as training samples to train a neural network model; According to the modification information, the labeling results of other video frames in the plurality of video frames are modified.

8. The method according to any one of claims 1 to 6, characterized in that, The labeling result is used for learning by an artificial intelligence (AI) model.

9. A video annotation system characterized by, The system comprises: An automatic frame extraction unit is configured to determine image similarity and object change amount between adjacent frames in an unlabeled video, compare the image similarity with a pre-stored similarity mapping relationship, compare the object change amount with a pre-stored change amount mapping relationship, determine a frame rate corresponding to the image similarity and the object change amount, and dynamically adjust a frame extraction rate of each video segment in the unlabeled video; The automatic frame extraction unit is configured to extract a plurality of video frames from each video segment in the unlabeled video by using the frame extraction rate, wherein the image similarity between the plurality of video frames extracted from the unlabeled video is lower than a first threshold value, or the object change amount between the plurality of video frames is higher than a second threshold value; A key frame processing unit is configured to display at least one video frame in the plurality of video frames to a user through a display interface; An automatic labeling unit is configured to obtain a user labeling result of labeling the at least one video frame by the user in the display interface, wherein the user labeling result comprises an image region of a target in the at least one video frame; A labeling correction unit is configured to obtain labeling results of other video frames in the plurality of video frames according to the user labeling result, wherein the labeling result comprises an image region of the target in the other video frames.

10. The system of claim 9, wherein, The at least one video frame is a first frame or a last frame in the plurality of video frames; or The at least one video frame is obtained by inputting the plurality of video frames into a key frame selection model, wherein the key frame selection model is obtained by using a plurality of known video frames and corresponding known key frames as training samples to train a neural network model.

11. The system of claim 10, wherein, The key frame processing unit is configured to input the at least one video frame into a recommended labeling model to obtain a recommended labeling result, wherein the recommended labeling result comprises an image region of at least one recommended target in the at least one video frame; The key frame processing unit is configured to display the recommended labeling result to the user through the display interface and obtain a user labeling result selected by the user in the recommended labeling result.

12. The system of claim 11, wherein, The labeling result comprises one or more of a target box, a center point, and a mask.

13. The system of claim 12, wherein, The automatic labeling unit is configured to call a plurality of computing units according to the user labeling result, process the other video frames in parallel to obtain the labeling results of the other video frames, wherein one computing unit processes one video frame, or at least one computing unit processes one video frame, and each computing unit in the at least one computing unit generates one labeling result.

14. The system of claim 13, wherein, The automatic labeling unit is configured to label video frames before or after the at least one video frame according to the user labeling result to obtain a labeling result.

15. The system of any of claims 9 to 14, wherein, The system further comprises a labeling correction unit, The key frame processing unit is configured to receive, through the display interface, modification information of the user on the labeling result, the modification information being from modification of the user on an image region of the target in the other video frames; or, The labeling correction unit is configured to obtain the modification information of the labeling result through a labeling correction model, the labeling correction model being obtained by training a neural network model using a plurality of known labeling results and corresponding known modification information as training samples; The automatic labeling unit is configured to modify the labeling result of the other video frames in the plurality of video frames according to the modification information.

16. The system of any of claims 9 to 14, wherein, The labeling result is configured to be learned by an AI model.

17. A computer readable storage medium characterized by: comprising instructions that, when executed on a computing device, cause the computing device to perform the method of any one of claims 1 to 8.

18. A computing device, comprising: comprising a processor and a memory, the processor executing code in the memory to perform the method of any one of claims 1 to 8.

19. A computer program product, characterised in that, comprising a computer program that, when read and executed by a computing device, causes the computing device to perform the method of any one of claims 1 to 8.

Images