Method, device and storage medium for annotating video data
By using centroid tracking algorithms and deep neural network models to automatically annotate video data, the problem of time-consuming and labor-intensive video data annotation has been solved, achieving efficient and accurate annotation results.
Patent Information
- Application Number
- CN202411774508.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-04
- Publication Date
- 2026-03-03
- Estimated Expiration
- 2044-12-04
AI Technical Summary
Existing video data annotation methods consume a lot of manpower and time, making it difficult to meet the needs of practical applications. Furthermore, traditional manual annotation methods are arduous and difficult.
By acquiring video data to be labeled, masking and category labeling of moving objects are performed, and identity association labeling is performed using centroid tracking algorithm and deep neural network model to form mask, category and identity labeling information.
It enables automated annotation of video data, saving manpower and time costs, reducing the intensity and difficulty of annotation work, improving annotation efficiency, and enhancing annotation accuracy.
Smart Images
Figure CN119863620B_ABST
Abstract
Description
Technical Field
[0001] This disclosure generally relates to the field of video image processing technology. More specifically, this disclosure relates to a method, apparatus, and computer-readable storage medium for annotating video data. Background Technology
[0002] Video instance segmentation (VIS) is an extension of image segmentation tasks, aiming to simultaneously segment, detect, and track every moving object in a video. It is a more challenging fundamental task than image segmentation. VIS plays a crucial role in downstream tasks such as video editing, short video tools, autonomous driving, surveillance and security, and life sciences. Deep learning-based video segmentation methods can provide automatic, temporally stable, and accurate segmentation results, and can track each individual moving object, enabling instance-level understanding and editing of videos.
[0003] Deep learning video instance segmentation methods can be categorized into online and offline methods. Regardless of the method, a large, high-quality video dataset is essential to accurately provide key information such as classification, mask, bounding box, and tracking markers for each moving object within each video segment. This is crucial for training a superior video segmentation model. Currently, datasets such as YouTube-VIS, OVIS, VIPSeg, and LSVOS are widely used for training and evaluating video segmentation models. To apply video segmentation models to more specific scenarios and applications, and to fully leverage the gains from pre-trained transfer learning, there is an urgent need to release richer, more diverse, and more complex video segmentation datasets. However, traditional manual annotation methods consume significant manpower and time, and are labor-intensive and difficult, making it difficult to meet the needs of practical applications.
[0004] In view of this, there is an urgent need to provide a solution for automated and efficient video data annotation, so as to save manpower and time costs, reduce the intensity and difficulty of video data annotation work, and improve the efficiency of video data annotation. Summary of the Invention
[0005] In order to address at least one or more of the technical issues mentioned above, this disclosure proposes a scheme for annotating video data in the following aspects.
[0006] In a first aspect, this disclosure provides a method for annotating video data, comprising: acquiring video data to be annotated; performing annotation operations on moving objects in the video data to be annotated to form annotation information of the video data to be annotated, wherein the annotation information includes mask annotation information of the moving objects; determining the centroid of each moving object in the video data to be annotated based on the mask annotation information; and performing identity association annotation on the same moving object in different frames of the video data to be annotated based on the centroid of the moving objects to form identity annotation information.
[0007] In some embodiments, the step of performing identity association annotation on the same moving object in different frames of the video data to be annotated based on the centroid of the moving object includes: for each pair of adjacent first and second frames in the video data to be annotated, performing a matching operation on the first moving object in the first frame and the second moving object in the second frame using a centroid tracking algorithm based on the centroid of the moving object to obtain a matching result between the first moving object and the second moving object; and performing identity association annotation on the same moving object in the first frame and the second frame based on the matching result and the first velocity of the first moving object and the second velocity of the second moving object.
[0008] In some embodiments, the first velocity and the second velocity are determined using the centroid of the moving object. Determining the first velocity and the second velocity using the centroid of the moving object includes: determining the first velocity based on the centroid of the first moving object in the first frame image and the centroid of the first preceding image, wherein the first preceding image is a predetermined number of images in the video data to be labeled that are located before the first frame image; and determining the second velocity based on the centroid of the first moving object in a second preceding image and the centroid of the second moving object in the second frame image, wherein the second preceding image is a predetermined number of images in the video data to be labeled that are located before the second frame image.
[0009] In some embodiments, the step of performing identity association annotation on the same moving object in the first frame image and the second frame image based on the matching result and the first speed of the first moving object and the second speed of the second moving object includes: determining the number of second moving objects that match the first moving object based on the matching result; when the number is equal to 1, determining the first moving object and the second moving object that matches it as the same moving object; when the number is greater than 1, selecting a second moving object whose speed is closest to that of the first moving object from a plurality of second moving objects based on the first speed and the second speed, and determining it as the same moving object as the first moving object; and performing identity association annotation on the first moving object and the second moving object determined to be the same moving object using the same identity identifier.
[0010] In some embodiments, selecting a second moving object from a plurality of second moving objects whose speed is closest to that of the first moving object based on the first speed and the second speed, and determining it as the same moving object as the first moving object, includes: calculating the difference between the second speed of the plurality of second moving objects and the first speed to obtain a plurality of speed differences; comparing the absolute values of the plurality of speed differences to obtain the speed difference with the smallest absolute value; and determining the second moving object corresponding to the speed difference with the smallest absolute value as the same moving object.
[0011] In some embodiments, the annotation information further includes category annotation information of moving objects. Performing annotation operations on moving objects in the video data to be annotated includes: inputting the video data to be annotated into a predetermined annotation model to perform annotation operations to obtain mask annotation information and category annotation information of the video data to be annotated; and performing correction operations on the mask annotation information and category annotation information using a predetermined annotation method to obtain corrected mask annotation information and category annotation information.
[0012] In some embodiments, the predetermined annotation model is a model obtained by training a deep neural network model. The process of obtaining the predetermined annotation model by training a deep neural network model includes: acquiring training video data; performing annotation operations on moving objects in the training video data using the predetermined annotation method to form training annotation information for the training video data, wherein the training annotation information includes training mask information and training category information; and inputting the training video data and the training annotation information as training data into the deep neural network model to train the deep neural network model and obtain the predetermined annotation model.
[0013] In some embodiments, training the deep neural network model further includes: inputting the corrected mask annotation information and category annotation information as new training data into the deep neural network model to continuously train and optimize the deep neural network model.
[0014] In a second aspect, this disclosure provides an apparatus for annotating video data, comprising: a processor; and a memory storing program instructions for annotating video data, wherein when the program instructions are executed by the processor, the method described in the first aspect and several embodiments thereof are implemented.
[0015] In a third aspect, this disclosure provides a computer-readable storage medium having stored program instructions thereon for annotating video data, which, when executed by a processor, implement the method described in the first aspect and several embodiments thereof.
[0016] Using the above-described scheme for annotating video data, this embodiment can generate mask annotation information for moving objects in the video data to be annotated, and can perform identity association annotation on the same moving object in different frames of the video data to be annotated, forming identity annotation information. Thus, automated annotation of moving objects in video data is achieved. Compared with existing manual annotation methods, the scheme disclosed herein for annotating video data can significantly save manpower and time costs, reduce the intensity and difficulty of video data annotation work, and help improve the efficiency of video data annotation.
[0017] Furthermore, in some embodiments, when performing identity association annotation on the same moving object within different frames of the video data to be annotated, the embodiments disclosed herein not only utilize the centroid tracking algorithm to determine the same moving object within different frames, but also consider the motion characteristics of the moving object, using the object's velocity as well to determine the same moving object within different frames. This improves the accuracy of the determination results and identity annotation information for the same moving object within different frames, thereby enhancing the precision of video data annotation work. Attached Figure Description
[0018] The above and other objects, features, and advantages of exemplary embodiments of this disclosure will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of this disclosure are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein:
[0019] Figure 1 This is a schematic diagram illustrating an exemplary structure of a video data annotation system according to an embodiment of this disclosure;
[0020] Figure 2 This is an exemplary flowchart illustrating a method for annotating video data according to an embodiment of this disclosure;
[0021] Figure 3 This is an exemplary schematic diagram illustrating mask labeling information according to embodiments of this disclosure;
[0022] Figure 4 This is an exemplary schematic diagram illustrating the querying of information about a moving object according to an embodiment of this disclosure;
[0023] Figure 5 An exemplary structural block diagram of the device according to an embodiment of this disclosure is shown. Detailed Implementation
[0024] The technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, not all of them. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.
[0025] Exemplary System
[0026] Figure 1 This is an exemplary structural diagram illustrating a video data annotation system 100 according to an embodiment of this disclosure. Figure 1 As shown, the system 100 may include, but is not limited to, an acquisition device 101, an annotation device 102, a storage device 103, and a display device 104. Here, the acquisition device 101 can be any general-purpose or dedicated shooting device depending on the actual shooting needs. The annotation device 102 can be any suitable electronic device with data processing capabilities, such as, but not limited to, terminals and servers. The storage device 103 can be any suitable electronic device with storage capabilities, such as, but not limited to, local storage devices or remote storage devices. The display device 104 can be any suitable electronic device with video data display capabilities. In actual operation, the acquisition device 101, annotation device 102, storage device 103, and display device 104 can work together to achieve video data annotation.
[0027] During the annotation process of video data, the acquisition device 101 can acquire the video data to be annotated and can also send the video data to be annotated to the annotation device 102. The annotation device 102 can perform annotation operations on the video data to be annotated to obtain the annotation information of the video data. Afterwards, the annotation device 102 can output the video data to be annotated and its annotation information, for example, it can output it to the storage device 103 for storage, or it can output it to the display device 104 for display. Thus, the annotators can view the video data to be annotated and its annotation information through the display device 104 to verify and correct the annotation information.
[0028] The video data annotation system disclosed herein can automate and efficiently annotate video data, effectively saving manpower and time costs, helping to reduce the intensity and difficulty of video data annotation work, and improving the efficiency of video data annotation.
[0029] Exemplary methods
[0030] Figure 2 This is an exemplary flowchart illustrating a method 200 for annotating video data according to an embodiment of this disclosure. It will be understood that method 200 can be performed by any suitable device with data processing capabilities, such as, but not limited to, terminal devices and servers.
[0031] like Figure 2 As shown, in step S201, method 200 can acquire video data to be labeled, which includes multiple consecutive frames of images. The moving object in the video data to be labeled disclosed herein can be any suitable moving object, such as a person, animal, or vehicle, or even a cell. In practical applications, those skilled in the art can select an appropriate acquisition device 101 to obtain the video data to be labeled based on the actual size of the moving object.
[0032] In one implementation scenario, the moving object disclosed herein can be a reproductive cell, such as bovine sperm. In this scenario, a bright-field photograph of the bovine sperm specimen on its stage can be performed using an inverted microscope of a sperm selector to obtain video data to be labeled. In the scheme disclosed herein, method 200 obtains a total of 51 segments of video data to be labeled for X sperm and 52 segments of video data to be labeled for Y sperm, with each segment containing approximately 60 frames. It should be understood that the number of video data to be labeled and the number of frames in each segment are merely exemplary, and the implementation method disclosed herein is not limited thereto. Those skilled in the art can obtain video data to be labeled according to actual needs.
[0033] Next, in step S202, method 200 can perform annotation operations on moving objects in the video data to be annotated to form annotation information for the video data to be annotated. Here, the annotation information for the video data to be annotated may include mask annotation information and category annotation information of the moving objects. In one embodiment, method 200 can save the annotation information as a JSON file in COCO format for use in downstream model training tasks.
[0034] In the disclosed scheme, when performing annotation operations on moving objects in the video data to be annotated, method 200 can first input the video data to be annotated into a predetermined annotation model to perform annotation operations, thereby obtaining mask annotation information and category annotation information of the video data to be annotated. Furthermore, in order to obtain higher quality annotation information, a predetermined annotation method can also be used to perform correction operations on the mask annotation information and category annotation information to obtain corrected mask annotation information and category annotation information.
[0035] The pre-defined annotation method disclosed herein can be a method of performing annotation operations on moving objects using an annotation tool. In practice, the annotation tool can be, for example, the free and open-source X-AnyLabeling. This annotation tool sets up an open-source API interface for Meta AI's SAM (Segment Anything Model). In one embodiment, the Vit-Large version of the SAM model can be selected to perform instance segmentation on each frame of the video data to be annotated, so as to annotate the mask and category of each moving object in the image, forming mask annotation information and category annotation information.
[0036] In one implementation scenario, the video data to be labeled may contain severely occluded or intersecting moving objects, or moving objects that are poorly segmented using annotation tools. In such cases, to obtain higher-quality annotation information, manual annotation can be used to label the moving objects.
[0037] Next, in step S203, method 200 can determine the centroid of each moving object in the video data to be annotated based on the mask annotation information. The mask annotation information can record the scatter coordinates of the edges of the moving objects. Thus, method 200 can determine the centroid of each moving object in the video data to be annotated based on the scatter coordinates of the edges of the moving objects.
[0038] Finally, in step S204, method 200 can perform identity association annotation on the same moving object in different frames of the video data to be annotated based on the centroid of the moving object, so as to form identity annotation information.
[0039] In the disclosed scheme, the identity labeling information can record the identity identifier of a moving object. It is understood that the same moving object within different frames of the video data to be labeled should have the same identity identifier, and different moving objects should have different identity identifiers. Here, identity association labeling for the same moving object means assigning the same identity identifier to the same object and using that same identity identifier to label the same moving object.
[0040] In implementing this disclosure, the inventors discovered that the same moving object in adjacent first and second frame images typically has a small centroid distance and a small velocity difference. Here, adjacent first and second frame images can be any pair of images in the video data to be labeled. Based on this, when performing identity association labeling on the same moving object in the first and second frame images, method 200 can first perform a matching operation on the first moving object in the first frame image and the second moving object in the second frame image based on the centroid of the moving object, using a centroid tracking algorithm, to obtain the matching result between the first and second moving objects. Then, based on the matching result and the first velocity of the first moving object and the second velocity of the second moving object, identity association labeling can be performed on the same moving object in the first and second frame images to form identity labeling information.
[0041] The above combination Figure 2 A method 200 for annotating video data is described. This method 200 can generate mask annotation information and category annotation information for moving objects in the video data to be annotated. It can also perform identity association annotation for the same moving object in different frames of the video data to be annotated, forming identity annotation information. Thus, automated annotation of moving objects in video data is achieved. Compared with existing manual annotation methods, the scheme disclosed here for annotating video data can greatly save manpower and time costs, reduce the intensity and difficulty of video data annotation work, and help improve the efficiency of video data annotation. The improved efficiency of video data annotation can promote the rapid progress of downstream video instance segmentation tasks, thereby improving the efficiency of video instance segmentation tasks.
[0042] In this disclosed solution, the mask annotation information can record the scatter coordinates of the edges of the moving object, and the category annotation information can record the category of the moving object. In one embodiment, when the video data to be annotated is bovine sperm video data and the moving object is bovine sperm, the mask annotation information can record the scatter coordinates of the edges of the head of the bovine sperm, and the category annotation information can record the category of the bovine sperm. Additionally or optionally, the category of bovine sperm may include X sperm and Y sperm, and the category of sperm can be labeled using "x_sperm" or "y_sperm".
[0043] Regarding the masking information disclosed herein, please refer to... Figure 3 To understand it. For example... Figure 3 As shown, the left image is the video frame to be labeled, and the right image is the labeled video frame. The video frame to be labeled contains multiple moving objects (sperm bovine sperm), and the labeled video frame adds mask annotation information for the moving objects (sperm bovine sperm). For each sperm bovine sperm, the mask annotation information records the scattered coordinates of the edge of its head. Figure 3 In the image, the green curve surrounding the head of the bovine sperm reveals the masking information of the sperm, and the dots inside the green curve indicate the centroid of the sperm.
[0044] As previously mentioned, the video instance segmentation methods described in this disclosure can be broadly categorized into online and offline methods. To facilitate understanding of the disclosed solutions, online and offline methods are described below. Online methods use both the current frame and historical frames as input when predicting the current frame's result, primarily applicable to scenarios requiring real-time processing, such as real-time perception in autonomous driving. Existing state-of-the-art (SOTA) online methods follow a technical approach of first performing image segmentation and then associating instances frame by frame. This approach does not optimize the segmentation result of the current frame based on information from other frames, thus lacking effective utilization of video information. Offline methods can use any frame in the video as input when predicting the current frame's result, primarily applicable to scenarios requiring offline processing, such as video editing. Existing state-of-the-art (SOTA) offline methods employ a tightly coupled network to process video segmentation tasks end-to-end. While this approach theoretically utilizes video information more effectively, its performance is unsatisfactory in long videos and complex scenes. Especially when multiple similar targets in the video occlude or swap positions, target tracking results become erratic, and segmentation accuracy is affected.
[0045] Next, continue to refer to... Figure 2In step S202, the predetermined annotation model performs video instance segmentation on the video data to be annotated, thereby annotating moving objects in the video data and forming annotation information for the video data to be annotated. This predetermined annotation model can be a model trained using a deep neural network model, such as the YOLOv8 model. Additionally or preferably, the deep neural network model can be the highest-precision version of the YOLOv8 model, such as the YOLOv8x-seg version. In practical applications, the training parameters of the YOLOv8 model can be set as follows: number of loops set to 300, early stopping strategy set to 40, optimizer set to the Adam optimizer, batch size set to 8, and dropout probability set to 0.2. It is understood that those skilled in the art can select the training parameters of the YOLOv8 model according to actual needs, and this disclosure does not impose specific limitations on this.
[0046] In one embodiment, when training a deep neural network to obtain a predetermined labeled model, training video data can be acquired first. Then, a predetermined annotation method can be used to annotate moving objects in the training video data to form training annotation information for the training video data. This training annotation information may include training mask information and training category information. Next, the training video data and training annotation information can be input as training data into the deep neural network model to train the deep neural network model and obtain the predetermined labeled model. Furthermore, the corrected mask annotation information and category annotation information obtained in step S202 can also be input as new training data into the deep neural network model for continuous training and optimization.
[0047] In practical applications, for video datasets containing multiple videos to be labeled, a batch-based, cyclical labeling approach can be used to complete the labeling task. The cyclical labeling process may include the following steps:
[0048] Step A: Select a subset of the unannotated video data from the dataset to be labeled as the first subset. Here, the subset of unannotated video data can be a predetermined number of unannotated video data from the dataset. Those skilled in the art can select a specific value for the predetermined number according to actual needs, and this disclosure does not limit this selection.
[0049] Step B: Perform annotation operations on the video data to be annotated in the first subset using a predefined annotation method to obtain the annotation information of the first subset.
[0050] Step C: Use the first subset and its annotation information as training data to input into the predetermined annotation model in order to train the predetermined annotation model.
[0051] Step D: Select a new subset of video data from the set of videos to be labeled as the second subset, and use the pre-trained labeling model to perform labeling operations on the video data in the second subset to obtain the labeling information of the second subset.
[0052] Step E: Perform a correction operation on the annotation information of the second subset using a predetermined annotation method to obtain the corrected annotation information of the second subset.
[0053] Step F: Input the second subset and its corrected annotation information as new training data into the predetermined annotation model to continuously train and optimize the predetermined annotation model.
[0054] Step G: Repeat steps D to F until all video data in the video dataset to be labeled has been labeled.
[0055] When the dataset contains 100 sperm videos, the cyclic annotation process disclosed herein can complete the annotation of all sperm videos within 3 minutes with an accuracy of over 85%, whereas manual annotation would require 150 hours of continuous work. Therefore, the video data annotation scheme disclosed herein can significantly save manpower and time costs while ensuring annotation accuracy, reducing the intensity and difficulty of video data annotation work and improving its efficiency.
[0056] In step S204, method 200 uses a centroid tracking algorithm to perform a matching operation on a first moving object in the first frame image and a second moving object in the second frame image, obtaining a matching result between the first moving object and the second moving object. Since the motion trajectory of the moving object is random, when multiple moving objects in the second frame image are close to each other, the number of second moving objects that match one of the first moving objects in the first frame image is not necessarily 1.
[0057] Based on this, when determining a second moving object that is identical to the first moving object, method 200 can first determine the number of second moving objects that match the first moving object based on the matching results. Then, it can determine whether this number is equal to 1. When the number is equal to 1, the first moving object and its matching second moving object can be identified as the same moving object. When the number is not equal to 1 but greater than 1, a second moving object whose speed is closest to the first moving object can be selected from multiple second moving objects based on the first and second speeds, and this object can be identified as the same moving object as the first moving object. Subsequently, method 200 can use the same identity identifier to perform identity association labeling on the first and second moving objects identified as the same moving object.
[0058] The second moving object whose speed is closest to that of the first moving object refers to the second moving object whose speed difference between its second speed and its first speed is the smallest. In one embodiment, selecting a second moving object whose speed is closest to that of the first moving object from among a plurality of second moving objects based on the first speed and the second speed, and determining it as the same moving object as the first moving object, may specifically include: calculating the difference between the second speed and the first speed of the plurality of second moving objects to obtain a plurality of speed differences; comparing the absolute values of the plurality of speed differences to obtain the speed difference with the smallest absolute value; and determining the second moving object corresponding to the speed difference with the smallest absolute value as the same moving object.
[0059] In the disclosed scheme, the first velocity and the second velocity are determined using the centroid of the moving object. In practice, the first velocity can be determined based on the centroid of the first moving object in the first frame image and the centroid of the first preceding image, wherein the first preceding image is a predetermined number of images in the video data to be labeled that are located before the first frame image. Furthermore, the second velocity can be determined based on the centroid of the first moving object in the second preceding image and the centroid of the second moving object in the second frame image, wherein the second preceding image is a predetermined number of images in the video data to be labeled that are located before the second frame image. It is understood that those skilled in the art can select the predetermined number according to actual needs; for example, the predetermined number can be 2, 3, 4, or 5, etc., and this disclosure does not limit the specific value of the predetermined number.
[0060] To facilitate understanding of this disclosure, the process of determining the first and second velocities is illustrated herein with an example. As an example, assuming a predetermined quantity of four, the first velocity can be determined based on the centroid of the first moving object in the first frame and the centroid of the first preceding image, where the first preceding image consists of four frames in the video data to be labeled that precede the first frame. Furthermore, the second velocity can also be determined based on the centroid of the first moving object in a second preceding image and the centroid of the second moving object in the second frame, where the second preceding image consists of four frames in the video data to be labeled that precede the second frame.
[0061] In one embodiment, a first velocity is determined based on the centroid of the first moving object in the first frame image and the centroid of the first preceding image. Specifically, a first motion distance of the first moving object can first be determined based on the centroids of the first moving object in the first frame image and the first preceding image. Then, the ratio of the first motion distance to the first motion time can be calculated to determine the first velocity. Here, the first motion time is equal to a predetermined number plus 1.
[0062] In another embodiment, the second velocity is determined based on the centroid of the first moving object in the second front image and the centroid of the second moving object in the second frame image. Specifically, the second motion distance of the second moving object can first be determined based on the centroids of the first moving object in the second front image and the second moving object in the second frame image. Then, the ratio of the second motion distance to the second motion time can be calculated to determine the second velocity. Here, the second motion time is equal to a predetermined number plus 1.
[0063] After completing the video data annotation operations described above, method 200 can also output the video data to be annotated and its annotation information externally, for example, to storage device 103. Users can use storage device 103 to query information about each moving object in each video data using query conditions, which has important applications in personalized dynamic analysis in the life sciences field. Here, query conditions may include, but are not limited to, the identity identifier of the moving object in its respective video data to be annotated and the identity identifier of the moving object in the set of video data to be annotated; information may include, but is not limited to, the moving object's motion trajectory and motion pattern.
[0064] This can be used as a reference. Figure 4 This illustrates an exemplary schematic diagram of querying information about a moving object according to an embodiment of this disclosure. Figure 4 As shown, information on 14 bovine sperm cells from 354_group_1 to 367_group_14 was retrieved. Since the aforementioned masking information records the scatter coordinates of the edges of the bovine sperm heads, Figure 3 Each frame of the image shows the head of each bovine sperm, but not the tail. Here, 354 to 367 are the identifiers of the bovine sperm in the set of video data to be labeled, and 1 to 14 are the identifiers of the bovine sperm in their respective sets of video data to be labeled.
[0065] It is understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or corresponding parts can be referred to each other. For the purpose of brevity, this disclosure will not go into detail.
[0066] Next, combine Figure 5 This disclosure provides an exemplary description of a device 500 for annotating video data, as provided in this embodiment. For example... Figure 5 As shown, the device 500 in this embodiment may include a processor 501, a memory 502, and a communication bus 503.
[0067] In specific embodiments, the processor 501 described above can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), CPU, controller, microcontroller, and microprocessor. It is understood that for different devices, the electronic device used to implement the above processor function can also be other types, and this embodiment does not specifically limit it.
[0068] In this disclosed embodiment, the communication bus 503 is used to realize the connection communication between the processor 501 and the memory 502; the memory 502 stores program instructions for training a sperm sex classification model or for classifying sperm sex; when the processor 501 executes the program instructions stored in the memory 502, it implements the combination of this disclosure. Figure 2 and Figure 4 The method described is for annotating video data.
[0069] The above combination Figure 5 This application describes a device that can be used to train a sperm sex classification model or to classify sperm sex according to the disclosure. It should be understood that the device structure or architecture described herein is merely exemplary, and the implementation methods and entities described herein are not limited thereto, but can be modified without departing from the spirit of this application.
[0070] It should be understood that when the terms "first," "second," "third," and "fourth," etc., are used in the claims, specification, and drawings of this application, they are used only to distinguish different objects and not to describe a specific order. The terms "comprising" and "including" as used in the specification and claims of this application indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.
[0071] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure. As used in this disclosure and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this disclosure and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.
[0072] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."
[0073] Based on the foregoing description in conjunction with the accompanying drawings, those skilled in the art will understand that the embodiments of this application can also be implemented by software programs. Therefore, this application also provides a computer-readable storage medium. This computer-readable storage medium can be used to implement the present disclosure. Figures 2 to 4 The method described is for annotating video data.
[0074] It should be noted that although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. On the contrary, the steps depicted in the flowchart can be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.
[0075] The embodiments of this disclosure have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this disclosure. Furthermore, any changes or modifications made by those skilled in the art based on the ideas of this disclosure, and on the specific implementation methods and application scope of this disclosure, are all within the scope of protection of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.
Claims
1. A method for annotating video data, comprising: obtaining video data to be annotated; performing an annotation operation on moving objects in the video data to be annotated to form annotation information of the video data to be annotated, wherein the annotation information comprises mask annotation information of the moving objects; determining a center of mass of each moving object in the video data to be annotated based on the mask annotation information; and performing identity association annotation on the same moving object within different frame images in the video data to be annotated based on the center of mass of the moving object to form identity annotation information. The performing identity association annotation on the same moving object within different frame images in the video data to be annotated based on the center of mass of the moving object comprises: for each pair of adjacent first frame image and second frame image in the video data to be annotated, performing a matching operation on a first moving object in the first frame image and a second moving object in the second frame image based on the center of mass of the moving object using a center of mass tracking algorithm to obtain a matching result between the first moving object and the second moving object; and performing identity association annotation on the same moving object in the first frame image and the second frame image based on the matching result and a first speed of the first moving object and a second speed of the second moving object.
2. The method of claim 1, wherein the first speed and the second speed are determined using the center of mass of the moving object, and the determining the first speed and the second speed using the center of mass of the moving object comprises: determining the first speed based on the center of mass of the first moving object in the first frame image and a center of mass in a first preceding image, wherein the first preceding image is a predetermined number of images in the video data to be annotated before the first frame image; and determining the second speed based on a center of mass in a second preceding image and the center of mass of the second moving object in the second frame image, wherein the second preceding image is a predetermined number of images in the video data to be annotated before the second frame image.
3. The method of claim 1 or 2, wherein the performing identity association annotation on the same moving object in the first frame image and the second frame image based on the matching result and a first speed of the first moving object and a second speed of the second moving object comprises: determining a number of second moving objects matched with the first moving object based on the matching result; when the number is equal to 1, determining the first moving object and the second moving object matched therewith as the same moving object; when the number is greater than 1, selecting one second moving object from the plurality of second moving objects that is closest to the first moving object in speed based on the first speed and the second speed, and determining the selected second moving object as the same moving object as the first moving object; and performing identity association annotation on the first moving object and the second moving object determined as the same moving object using the same identity.
4. A device for annotating video data, comprising: an obtaining unit configured to obtain video data to be annotated; an annotation unit configured to perform an annotation operation on moving objects in the video data to be annotated to form annotation information of the video data to be annotated, wherein the annotation information comprises mask annotation information of the moving objects; a center of mass determining unit configured to determine a center of mass of each moving object in the video data to be annotated based on the mask annotation information; and an identity association unit configured to perform identity association annotation on the same moving object within different frame images in the video data to be annotated based on the center of mass of the moving object to form identity annotation information. The identity association unit is configured to: for each pair of adjacent first frame image and second frame image in the video data to be annotated, perform a matching operation on a first moving object in the first frame image and a second moving object in the second frame image based on the center of mass of the moving object using a center of mass tracking algorithm to obtain a matching result between the first moving object and the second moving object; and perform identity association annotation on the same moving object in the first frame image and the second frame image based on the matching result and a first speed of the first moving object and a second speed of the second moving object.
5. The device of claim 4, wherein the first speed and the second speed are determined using the center of mass of the moving object, and the center of mass determining unit is configured to: determine the first speed based on the center of mass of the first moving object in the first frame image and a center of mass in a first preceding image, wherein the first preceding image is a predetermined number of images in the video data to be annotated before the first frame image; and determine the second speed based on a center of mass in a second preceding image and the center of mass of the second moving object in the second frame image, wherein the second preceding image is a predetermined number of images in the video data to be annotated before the second frame image.
6. The device of claim 4 or 5, wherein the identity association unit is configured to: determine a number of second moving objects matched with the first moving object based on the matching result; when the number is equal to 1, determine the first moving object and the second moving object matched therewith as the same moving object; when the number is greater than 1, select one second moving object from the plurality of second moving objects that is closest to the first moving object in speed based on the first speed and the second speed, and determine the selected second moving object as the same moving object as the first moving object; and perform identity association annotation on the first moving object and the second moving object determined as the same moving object using the same identity.
4. The method of claim 3, wherein selecting, based on the first speed and the second speeds, one of the second moving objects that is closest to the first moving object as the same moving object as the first moving object comprises: calculating a difference between the first speed and the second speeds of the second moving objects respectively to obtain a plurality of speed differences; comparing absolute values of the plurality of speed differences to obtain a smallest absolute value of the speed differences; and determining the second moving object corresponding to the smallest absolute value of the speed differences as the same moving object.
5. The method of claim 1, wherein the annotation information further comprises class annotation information of the moving object, and performing the annotation operation on the moving object in the to-be-annotated video data comprises: inputting the to-be-annotated video data into a predetermined annotation model to perform the annotation operation to obtain mask annotation information and class annotation information of the to-be-annotated video data; and performing correction operation on the mask annotation information and the class annotation information by using a predetermined annotation manner to obtain corrected mask annotation information and class annotation information.
6. The method of claim 5, wherein the predetermined annotation model is a model obtained by training a deep neural network model, and the training the deep neural network to obtain the predetermined annotation model comprises: obtaining training video data; performing annotation operation on the moving object in the training video data by using the predetermined annotation manner to form training annotation information of the training video data, wherein the training annotation information comprises training mask information and training class information; and inputting the training video data and the training annotation information as training data into the deep neural network model to train the deep neural network model to obtain the predetermined annotation model.
7. The method of claim 6, wherein the training the deep neural network model further comprises: inputting the corrected mask annotation information and class annotation information as new training data into the deep neural network model to continuously train and optimize the deep neural network model.
8. An apparatus for annotating video data, comprising: a processor; and a memory storing program instructions for annotating video data, which when executed by the processor, cause the implementation of the method according to any one of claims 1-7.
9. A computer readable storage medium storing program instructions for annotating video data, which when executed by a processor, cause the implementation of the method according to any one of claims 1-7.
Citation Information
Patent Citations
Method and system for automatically tracking and identifying ships on sea
CN106910204A