Method and device for labeling video data and storage medium

Abstract

The invention discloses a method and device for labeling video data and a storage medium. The method comprises the steps of obtaining to-be-labeled video data; annotation operation is executed on a moving object in the to-be-annotated video data to form annotation information of the to-be-annotated video data, and the annotation information comprises mask annotation information of the moving object; determining the centroid of each moving object in the to-be-labeled video data based on the mask labeling information; and based on the centroid of the moving object, performing identity association labeling on the same moving object in different frames of images in the to-be-labeled video data to form identity labeling information. According to the scheme disclosed by the invention, automatic labeling of the moving object in the video data is realized, manpower and time cost can be greatly saved, the intensity and difficulty of video data labeling work can be reduced, and improvement of video data labeling efficiency is facilitated.

Description

Technical Field

[0001] The present disclosure generally relates to the field of video image processing technology. More specifically, the present disclosure relates to a method, device and computer-readable storage medium for annotating video data. Background Art

[0002] Video Instance Segmentation (VIS) is an extension of the image segmentation task. It aims to simultaneously segment, detect, and track each moving object in the video. It is a more challenging basic task than image segmentation. Video instance segmentation plays an important role in downstream tasks such as video editing, short video tools, autonomous driving, surveillance and security, and life sciences. Deep learning video segmentation methods can provide automatic time-stable and accurate segmentation results, and can track each individual moving object, realizing instance-level understanding and editing functions in the video.

[0003] Deep learning video instance segmentation methods can be divided into online methods and offline methods. Regardless of the method, a large, high-quality video dataset is needed as support to accurately provide key information such as the classification, mask, bounding box, tracking mark, etc. of each moving object in each video data segment, so as to train a video segmentation model with better performance. Currently, datasets such as YouTube-VIS, OVIS, VIPSeg, and LSVOS are widely used for the training and evaluation of video segmentation models. In order to apply video segmentation models to more specific scenarios and applications, and to make full use of the gains brought by pre-training transfer learning, it is urgent to release more diverse and complex video segmentation datasets. However, the traditional manual annotation method consumes a lot of manpower and time costs, and the work intensity and difficulty are high, so it is difficult to meet the needs of actual applications.

[0004] In view of this, there is an urgent need to provide a solution for automatically and efficiently labeling video data in order to save manpower and time costs, reduce the intensity and difficulty of video data labeling work, and improve the efficiency of video data labeling. Summary of the invention

[0005] In order to at least solve one or more of the technical problems mentioned above, the present disclosure proposes a solution for labeling video data in the following aspects.

[0006] In a first aspect, the present disclosure provides a method for annotating video data, comprising acquiring video data to be annotated; performing an annotation operation on moving objects in the video data to be annotated to form annotation information of the video data to be annotated, wherein the annotation information includes mask annotation information of the moving objects; determining the center of mass of each moving object in the video data to be annotated based on the mask annotation information; and performing identity association annotation on the same moving object in different frame images in the video data to be annotated based on the center of mass of the moving object to form identity annotation information.

[0007] In some embodiments, the identity association annotation of the same moving object in different frame images in the video data to be annotated based on the centroid of the moving object includes: for each pair of adjacent first frame images and second frame images in the video data to be annotated, based on the centroid of the moving object, performing a matching operation on the first moving object in the first frame image and the second moving object in the second frame image using a centroid tracking algorithm to obtain a matching result between the first moving object and the second moving object; and based on the matching result and the first speed of the first moving object and the second speed of the second moving object, identity association annotation of the same moving object in the first frame image and the second frame image.

[0008] In some embodiments, the first speed and the second speed are determined using the center of mass of the moving object, and the determining of the first speed and the second speed using the center of mass of the moving object includes: determining the first speed based on the center of mass of the first moving object in the first frame image and the center of mass in a first preceding image, wherein the first preceding image is a predetermined number of images in the video data to be labeled that are located before the first frame image; and determining the second speed based on the center of mass of the first moving object in a second preceding image and the center of mass of the second moving object in the second frame image, wherein the second preceding image is a predetermined number of images in the video data to be labeled that are located before the second frame image.

[0009] In some embodiments, the identity association labeling of the same moving object in the first frame image and the second frame image based on the matching result and the first speed of the first moving object and the second speed of the second moving object includes: determining the number of second moving objects matching the first moving object based on the matching result; when the number is equal to 1, determining the first moving object and the second moving object matching it as the same moving object; when the number is greater than 1, selecting a second moving object closest to the speed of the first moving object from multiple second moving objects based on the first speed and the second speed, and determining it as the same moving object as the first moving object; and using the same identity identifier to perform identity association labeling on the first moving object and the second moving object determined as the same moving object.

[0010] In some embodiments, based on the first speed and the second speed, a second moving object whose speed is closest to the first moving object is selected from multiple second moving objects, and it is determined as the same moving object as the first moving object, including: calculating the difference between the second speeds of the multiple second moving objects and the first speed respectively to obtain multiple speed differences; comparing the absolute values ​​of the multiple speed differences to obtain the speed difference with the smallest absolute value; and determining the second moving object corresponding to the speed difference with the smallest absolute value as the same moving object.

[0011] In some embodiments, wherein the labeling information also includes category labeling information of moving objects, performing a labeling operation on the moving objects in the video data to be labeled includes: inputting the video data to be labeled into a predetermined labeling model to perform a labeling operation to obtain mask labeling information and category labeling information of the video data to be labeled; and performing a correction operation on the mask labeling information and the category labeling information using a predetermined labeling method to obtain corrected mask labeling information and category labeling information.

[0012] In some embodiments, the predetermined labeling model is a model obtained by training using a deep neural network model, and the method of obtaining the predetermined labeling model by training using a deep neural network includes: acquiring training video data; performing labeling operations on moving objects in the training video data using the predetermined labeling method to form training labeling information for the training video data, wherein the training labeling information includes training mask information and training category information; and inputting the training video data and the training labeling information into the deep neural network model as training data to train the deep neural network model to obtain the predetermined labeling model.

[0013] In some embodiments, the training of the deep neural network model further includes: inputting the corrected mask annotation information and category annotation information into the deep neural network model as new training data to continuously train and optimize the deep neural network model.

[0014] In a second aspect, the present disclosure provides a device for labeling video data, comprising: a processor; and a memory storing program instructions for labeling video data, wherein when the program instructions are executed by the processor, the method described in the first aspect and its multiple embodiments are implemented.

[0015] In a third aspect, the present disclosure provides a computer-readable storage medium having stored thereon program instructions for labeling video data, wherein when the program instructions are executed by a processor, the method described in the first aspect and its multiple embodiments are implemented.

[0016] Through the scheme for annotating video data provided above, the disclosed embodiment can form mask annotation information of moving objects in the video data to be annotated, and can perform identity association annotation on the same moving object in different frame images in the video data to be annotated to form identity annotation information. Thus, automatic annotation of moving objects in video data is achieved. Compared with the existing manual annotation method, the disclosed scheme for annotating video data can greatly save manpower and time costs, can reduce the intensity and difficulty of video data annotation work, and help improve the efficiency of video data annotation.

[0017] Furthermore, in some embodiments, when the same moving object in different frames of the video data to be annotated is annotated with identity association, the disclosed embodiments not only use the centroid tracking algorithm to determine the same moving object in different frames, but also consider the motion characteristics of the moving object and use the speed of the moving object to determine the same moving object in different frames. As a result, the accuracy of the determination results and identity annotation information of the same moving object in different frames can be improved, thereby improving the accuracy of video data annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] By reading the detailed description below with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0019] Figure 1 is a schematic diagram showing an exemplary structure of a video data annotation system according to an embodiment of the present disclosure;

[0020] Figure 2 is an exemplary flow chart illustrating a method for annotating video data according to an embodiment of the present disclosure;

[0021] Figure 3 is an exemplary schematic diagram showing mask annotation information according to an embodiment of the present disclosure;

[0022] Figure 4 is an exemplary schematic diagram showing querying information of a moving object according to an embodiment of the present disclosure;

[0023] Figure 5 An exemplary structural block diagram of a device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0025] Exemplary Systems

[0026] Figure 1 FIG. 1 is a schematic diagram showing an exemplary structure of a video data annotation system 100 according to an embodiment of the present disclosure. Figure 1 As shown in , the system 100 may include but is not limited to an acquisition device 101, an annotation device 102, a storage device 103 and a display device 104. Here, the acquisition device 101 may be any general or special shooting device according to actual shooting needs. The annotation device 102 may be any appropriate electronic device with data processing capabilities, and the electronic device with data processing capabilities may include but is not limited to a terminal and a server, etc. The storage device 103 may be any appropriate electronic device with storage capabilities, and the electronic device with storage capabilities may include but is not limited to a local storage device or a remote storage device, etc. The display device 104 may be any appropriate electronic device with video data display capabilities. In actual operation, the acquisition device 101, the annotation device 102, the storage device 103 and the display device 104 may work together to achieve video data annotation.

[0027] In the process of annotating video data, the video data to be annotated can be obtained through the acquisition device 101, and the video data to be annotated can also be sent to the annotation device 102. The annotation device 102 can perform an annotation operation on the video data to be annotated to obtain the annotation information of the video data to be annotated. Thereafter, the annotation device 102 can output the video data to be annotated and its annotation information to the outside, for example, it can be output to the storage device 103 for storage, and it can also be output to the display device 104 for display. Thus, the annotator can view the video data to be annotated and its annotation information through the display device 104, so as to verify and correct the annotation information.

[0028] The video data labeling system disclosed in the present invention can realize automatic and efficient labeling of video data, which can effectively save manpower and time costs, help reduce the intensity and difficulty of video data labeling work, and improve the efficiency of video data labeling.

[0029] Exemplary Methods

[0030] Figure 2 2 is an exemplary flow chart showing a method 200 for annotating video data according to an embodiment of the present disclosure. It is understood that the method 200 may be executed by any appropriate device having data processing capabilities, such as but not limited to a terminal device and a server.

[0031] like Figure 2 As shown in , at step S201, method 200 can obtain video data to be labeled including multiple consecutive frames of images. The moving object in the video data to be labeled disclosed in the present disclosure can be any appropriate moving object, such as a person, an animal, a vehicle, etc., or a cell. In practical applications, those skilled in the art can select an appropriate acquisition device 101 to obtain the video data to be labeled according to the actual size of the moving object.

[0032] In one implementation scenario, the moving object of the present disclosure may be a reproductive cell, such as a bovine sperm. In this implementation scenario, the inverted microscope of the sperm optimization instrument may be used to perform bright field photography of the bovine sperm specimen on its stage to obtain video data to be labeled. In the scheme of the present disclosure, method 200 obtains a total of 51 segments of video data to be labeled for X sperm and 52 segments of video data to be labeled for Y sperm, and each segment of video data to be labeled contains about 60 frames of images. It should be understood that the number of video data to be labeled and the number of image frames in each segment of video data to be labeled are merely exemplary, and the implementation of the present disclosure is not limited thereto. Those skilled in the art may obtain video data to be labeled according to actual needs.

[0033] Next, at step S202, the method 200 may perform a labeling operation on the moving objects in the video data to be labeled to form labeling information of the video data to be labeled. Here, the labeling information of the video data to be labeled may include mask labeling information and category labeling information of the moving objects. In one embodiment, the method 200 may save the labeling information into a JSON file in COCO format for use in downstream model training tasks.

[0034] In the solution disclosed herein, when the method 200 performs a labeling operation on the moving object in the video data to be labeled, the video data to be labeled may be first input into a predetermined labeling model to perform a labeling operation to obtain mask labeling information and category labeling information of the video data to be labeled. Furthermore, in order to obtain higher quality labeling information, the mask labeling information and the category labeling information may be corrected using a predetermined labeling method to obtain corrected mask labeling information and category labeling information.

[0035] The predetermined labeling method disclosed herein may be a method of performing labeling operations on moving objects using a labeling tool. In actual operation, the labeling tool may be, for example, the free and open source X-AnyLabeling. The labeling tool sets up an open source API interface for Meta AI's SAM model (Segment Anything Model). In one embodiment, the Vit-Large version of the SAM model may be selected to perform instance segmentation on each frame of the video data to be labeled, so as to annotate the mask and category of each moving object in the image, and form mask labeling information and category labeling information.

[0036] In one implementation scenario, there may be moving objects with severe occlusion and intersection in the video data to be annotated, or moving objects that are poorly segmented using annotation tools. In such cases, in order to obtain higher quality annotation information, manual annotation can be used to perform annotation operations on the moving objects.

[0037] Next, at step S203, method 200 can determine the centroid of each moving object in the video data to be labeled based on the mask annotation information. The mask annotation information can record the scattered coordinates of the edge of the moving object. Thus, method 200 can determine the centroid of each moving object in the video data to be labeled based on the scattered coordinates of the edge of the moving object.

[0038] Finally, at step S204, method 200 may perform identity association annotation on the same moving object in different frame images in the video data to be annotated based on the centroid of the moving object to form identity annotation information.

[0039] In the scheme disclosed herein, the identity tagging information can record the identity of the moving object. It is understandable that the same moving object in different frames of the video data to be tagged should have the same identity, and different moving objects should have different identities. Here, identity-associated tagging of the same moving object means assigning the same identity to the same object, and using the same identity to tag the same moving object.

[0040] In the process of implementing the present disclosure, the inventors discovered that the same moving object in the adjacent first frame image and the second frame image usually has the characteristics of a small center of mass distance and a small speed difference. Here, the adjacent first frame image and the second frame image can be any pair of images in the video data to be annotated. Based on this, when method 200 performs identity association annotation on the same moving object in the first frame image and the second frame image, it can first perform a matching operation on the first moving object in the first frame image and the second moving object in the second frame image based on the center of mass of the moving object using a center of mass tracking algorithm to obtain a matching result between the first moving object and the second moving object. Then, based on the matching result and the first speed of the first moving object and the second speed of the second moving object, the same moving object in the first frame image and the second frame image can be annotated with identity association to form identity annotation information.

[0041] Combination of the above Figure 2 A method 200 for annotating video data is described. Through this method 200, mask annotation information and category annotation information of moving objects in the video data to be annotated can be formed, and identity association annotation can be performed on the same moving object in different frame images in the video data to be annotated to form identity annotation information. Thus, automatic annotation of moving objects in video data is achieved. Compared with the existing manual annotation method, the scheme for annotating video data disclosed in the present invention can greatly save manpower and time costs, can reduce the intensity and difficulty of video data annotation work, and help improve the efficiency of video data annotation. The improvement of the efficiency of video data annotation can promote the rapid progress of downstream video instance segmentation tasks, thereby improving the efficiency of video instance segmentation tasks.

[0042] In the solution disclosed herein, the mask annotation information may record the scattered coordinates of the edge of the moving object, and the category annotation information may record the category of the moving object. In one embodiment, when the video data to be annotated is bovine sperm video data and the moving object is bovine sperm, the mask annotation information may record the scattered coordinates of the edge of the head of the bovine sperm, and the category annotation information may record the category of the bovine sperm. Additionally or alternatively, the category of the bovine sperm may include X sperm and Y sperm, and the category of the sperm may be annotated using "x_sperm" or "y_sperm".

[0043] Regarding the mask annotation information disclosed in this disclosure, you can combine Figure 3 Understand it. Figure 3 As shown in the figure, the image on the left is the video frame image to be annotated, and the image on the right is the annotated video frame image. The video frame image to be annotated contains multiple moving objects (bull sperm), and the annotated video frame image adds mask annotation information of the moving objects (bull sperm). For each bull sperm, the mask annotation information records the scattered coordinates of the edge of its head. Figure 3 In the figure, the green curve around the head of the bull sperm shows the mask annotation information of the bull sperm, and the dot inside the green curve shows the center of mass of the bull sperm.

[0044] As mentioned above, the video instance segmentation method described in the present disclosure can be roughly divided into an online method and an offline method. In order to facilitate the understanding of the scheme of the present disclosure, the online method and the offline method are described below. The online method uses the current frame and the historical frame as input when predicting the result of the current frame, and is mainly used in demand scenarios that require real-time processing, such as real-time perception in autonomous driving. The existing SOTA online method follows the technical route of first performing image segmentation and then associating instances frame by frame. This technical route does not optimize the segmentation result of the current frame based on the information of other frames, so it lacks effective use of video information. The offline method can use any frame in the video as input when predicting the result of the current frame, and is mainly used in demand scenarios for offline processing, such as video editing. The existing SOTA offline method uses a tightly coupled network to process the video segmentation task end-to-end. Although this technical route can theoretically make more effective use of video information, the performance is unsatisfactory in long videos and complex scenes. Especially when there are many similar targets in the video that are mutually occluded and transposed, the target tracking results are confused and the segmentation accuracy is also affected.

[0045] Next, continue to refer to Figure 2At step S202, the predetermined annotation model implements the annotation of the moving objects in the video data to be annotated by performing video instance segmentation on the video data to be annotated, thereby forming the annotation information of the video data to be annotated. The predetermined annotation model may be a model obtained by training using a deep neural network model, and the deep neural network model may be, for example, a YOLOv8 model. Additionally or preferably, the deep neural network model may be a YOLOv8 model with the highest accuracy, such as a YOLOv8x-seg version. In practical applications, the training parameters of the YOLOv8 model may be set as follows: the number of cycles is set to 300, the early stopping strategy is set to 40, the optimizer is set to the Adam optimizer, the batch size is set to 8, and the dropout probability is set to 0.2. It is understandable that those skilled in the art may select the training parameters of the YOLOv8 model according to actual needs, and the present disclosure does not specifically limit this.

[0046] In one embodiment, when a deep neural network is used for training to obtain a predetermined annotation model, training video data may be obtained first. Then, a predetermined annotation method may be used to perform annotation operations on moving objects in the training video data to form training annotation information for the training video data, where the training annotation information may include training mask information and training category information. Next, the training video data and the training annotation information may be input as training data into the deep neural network model to train the deep neural network model to obtain the predetermined annotation model. Furthermore, the corrected mask annotation information and category annotation information obtained in the above step S202 may also be input as new training data into the deep neural network model to continuously train and optimize the deep neural network model.

[0047] In practical applications, for a video data set to be annotated that contains multiple video data to be annotated, the annotation task can be completed in batches and in a cyclical manner. The cyclical annotation process can include the following steps:

[0048] Step A: Selecting part of the video data to be labeled from the video data set to be labeled as the first subset. Here, the part of the video data to be labeled can be a preset number of video data to be labeled in the video data set to be labeled. Those skilled in the art can select a specific value of the preset number according to actual needs, and this disclosure does not limit this.

[0049] Step B: performing a labeling operation on the to-be-labeled video data in the first subset using a predetermined labeling method to obtain labeling information of the first subset.

[0050] Step C: Use the first subset and its annotation information as training data and input them into a predetermined annotation model to train the predetermined annotation model.

[0051] Step D: Select a new part of the video data to be labeled from the set of video data to be labeled as the second subset, and use the trained predetermined labeling model to perform a labeling operation on the video data to be labeled in the second subset to obtain labeling information of the second subset.

[0052] Step E: performing a correction operation on the labeling information of the second subset using a predetermined labeling method to obtain the corrected labeling information of the second subset.

[0053] Step F: Input the second subset and the revised annotation information thereof as new training data into the predetermined annotation model to continuously train and optimize the predetermined annotation model.

[0054] Step G: loop through steps D to F until all the video data to be labeled in the video data set to be labeled are labeled.

[0055] When the video data set to be labeled contains 100 sperm video data, the circular labeling process disclosed herein can complete the labeling of all sperm video data within 3 minutes, and the accuracy can reach more than 85%, while the manual labeling method requires 150 hours of uninterrupted work. It can be seen that the video data labeling scheme disclosed herein can greatly save manpower and time costs while ensuring the accuracy of labeling, reduce the intensity and difficulty of video data labeling work, and help improve the efficiency of video data labeling.

[0056] At step S204, the method 200 uses a centroid tracking algorithm to perform a matching operation on the first moving object in the first frame image and the second moving object in the second frame image, and obtains a matching result between the first moving object and the second moving object. Since the motion trajectory of the moving object is random, when the multiple moving objects in the second frame image are close to each other, the number of second moving objects matching one first moving object in the first frame image is not necessarily 1.

[0057] Based on this, when determining the second moving object that is the same as the first moving object, method 200 can first determine the number of second moving objects that match the first moving object based on the matching result. Then, it can be determined whether the number is equal to 1. When the number is equal to 1, the first moving object and the second moving object that matches it can be determined as the same moving object. When the number is not equal to 1 and is greater than 1, a second moving object that is closest to the speed of the first moving object can be selected from multiple second moving objects based on the first speed and the second speed, and it can be determined as the same moving object as the first moving object. Thereafter, method 200 can use the same identity identifier to perform identity association labeling on the first moving object and the second moving object determined to be the same moving object.

[0058] The second moving object closest to the speed of the first moving object is the second moving object whose speed difference with the first speed is the smallest. In one embodiment, selecting a second moving object closest to the speed of the first moving object from a plurality of second moving objects based on the first speed and the second speed, and determining it as the same moving object as the first moving object may specifically include: calculating the difference between the second speeds of the plurality of second moving objects and the first speed respectively to obtain a plurality of speed differences; comparing the absolute values ​​of the plurality of speed differences to obtain the speed difference with the smallest absolute value; and determining the second moving object corresponding to the speed difference with the smallest absolute value as the same moving object.

[0059] In the scheme of the present disclosure, the first speed and the second speed are determined using the center of mass of the moving object. In actual operation, the first speed can be determined based on the center of mass of the first moving object in the first frame image and the center of mass in the first preceding image, wherein the first preceding image is a predetermined number of images located before the first frame image in the video data to be annotated. Further, the second speed can be determined based on the center of mass of the first moving object in the second preceding image and the center of mass of the second moving object in the second frame image, wherein the second preceding image is a predetermined number of images located before the second frame image in the video data to be annotated. It is understandable that those skilled in the art can select the predetermined number according to actual needs, for example, the predetermined number can be 2, 3, 4, 5, etc., and the present disclosure does not limit the specific value of the predetermined number.

[0060] In order to understand the disclosed solution, the process of determining the first speed and the second speed is described here with reference to an example. As an example, assuming that the predetermined number is 4, the first speed can be determined based on the center of mass of the first moving object in the first frame image and the center of mass in the first preceding image, wherein the first preceding image is the 4 frames of images before the first frame image in the video data to be annotated. Furthermore, the second speed can also be determined based on the center of mass of the first moving object in the second preceding image and the center of mass of the second moving object in the second frame image, wherein the second preceding image is the 4 frames of images before the second frame image in the video data to be annotated.

[0061] In one embodiment, the first speed is determined based on the center of mass of the first moving object in the first frame image and the center of mass of the first preceding image. Specifically, the first moving distance of the first moving object can be determined based on the center of mass of the first moving object in the first frame image and the center of mass of the first preceding image. Then, the ratio of the first moving distance to the first moving time can be calculated to determine the first speed. Here, the first moving time is equal to the predetermined number plus 1.

[0062] In another embodiment, the second speed is determined based on the center of mass of the first moving object in the second preceding image and the center of mass of the second moving object in the second frame image. Specifically, the second moving distance of the second moving object can be determined based on the center of mass of the first moving object in the second preceding image and the center of mass of the second moving object in the second frame image. Then, the ratio of the second moving distance to the second moving time can be calculated to determine the second speed. Here, the second moving time is equal to the predetermined number plus 1.

[0063] After completing the above-mentioned video data annotation operation, the method 200 can also output the video data to be annotated and its annotation information to the outside, for example, it can be output to the storage device 103 for storage. The user can use the query conditions through the storage device 103 to query the information of each moving object in each video data, which has important applications for personalized dynamic analysis in the field of life sciences. Here, the query conditions may include but are not limited to the identity of the moving object in the video data to be annotated and the identity of the moving object in the video data set to be annotated, and the information may include but is not limited to the motion trajectory and motion mode of the moving object.

[0064] You can refer to here Figure 4 , which shows an exemplary schematic diagram of querying information of a moving object according to an embodiment of the present disclosure. Figure 4 As shown in , a total of 14 bovine sperm information from 354_group_1 to 367_group_14 were queried. Since the aforementioned mask annotation information records the scattered coordinates of the edge of the bovine sperm head, Figure 3 Each frame of the image shows the head of each bull sperm, but does not show the tail of the bull sperm. Here, 354 to 367 are the identities of the bull sperm in the video data set to be labeled, and 1 to 14 are the identities of the bull sperm in the video data to be labeled.

[0065] It is understandable that the description of each embodiment in this disclosure emphasizes the differences between the embodiments, and the same or corresponding parts can be referenced to each other. For the purpose of brevity, this disclosure will not repeat them one by one.

[0066] Next, combine Figure 5 An exemplary introduction is given to a device 500 for labeling video data provided by the embodiment of the present disclosure. Figure 5 As shown, the device 500 of the embodiment of the present disclosure may include a processor 501 , a memory 502 , and a communication bus 503 .

[0067] In the process of a specific embodiment, the processor 501 may be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing image processing device (DSPD), a programmable logic image processing device (PLD), a field programmable gate array (FPGA), a CPU, a controller, a microcontroller, and a microprocessor. It is understandable that for different devices, the electronic device used to implement the function of the processor may also be other, which is not specifically limited in this embodiment.

[0068] In the embodiment of the present disclosure, the communication bus 503 is used to realize the connection and communication between the processor 501 and the memory 502; the memory 502 stores program instructions for training the sperm sex classification model or for classifying the sperm sex; when the processor 501 executes the program instructions stored in the memory 502, the present disclosure is realized. Figure 2 and Figure 4 A method for annotating video data is described.

[0069] Combination of the above Figure 5 Devices for training sperm sex classification models or for classifying sperm sex that can be used to perform the present disclosure are described. It should be understood that the device structure or architecture here is only exemplary, and the implementation method and implementation entity of the present application are not limited thereto, but can be changed without departing from the spirit of the present application.

[0070] It should be understood that when the terms "first", "second", "third" and "fourth" are used in the claims, the specification and the drawings of the present application, they are only used to distinguish different objects, rather than to describe a specific order. The terms "include" and "comprise" used in the specification and claims of the present application indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their collections.

[0071] It should also be understood that the terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. As used in this disclosure and claims, the singular forms of "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" used in this disclosure and claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations.

[0072] As used in this specification and claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0073] According to the above description in combination with the accompanying drawings, those skilled in the art can also understand that the embodiments of the present application can also be implemented by a software program. Therefore, the present application also provides a computer-readable storage medium. The computer-readable storage medium can be used to implement the present disclosure in combination with the accompanying drawings. Figures 2 to 4 A method for annotating video data is described.

[0074] It should be noted that although the operations of the method of the present application are described in a specific order in the accompanying drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. On the contrary, the steps depicted in the flow chart can be performed in a different order. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps.

[0075] The embodiments of the present disclosure are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present disclosure. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present disclosure. At the same time, changes or deformations made by those skilled in the art based on the ideas of the present disclosure, the specific implementation methods and the scope of application of the present disclosure, all belong to the scope of protection of the present disclosure. In summary, the content of this specification should not be understood as a limitation on the present disclosure.

Claims

1. A method for labeling video data, comprising: Obtain the video data to be labeled; Performing a labeling operation on the moving object in the video data to be labeled to form labeling information of the video data to be labeled, wherein the labeling information includes mask labeling information of the moving object; Determine the center of mass of each moving object in the to-be-annotated video data based on the mask annotation information; as well as Based on the centroid of the moving object, identity association annotation is performed on the same moving object in different frame images in the video data to be annotated to form identity annotation information.

2. The method according to claim 1, wherein the step of performing identity association annotation on the same moving object in different frame images in the video data to be annotated based on the centroid of the moving object comprises: For each pair of adjacent first frame images and second frame images in the to-be-annotated video data, based on the centroid of the moving object, a centroid tracking algorithm is used to perform a matching operation on a first moving object in the first frame image and a second moving object in the second frame image, so as to obtain a matching result between the first moving object and the second moving object; as well as Based on the matching result and the first speed of the first moving object and the second speed of the second moving object, identity association annotation is performed on the same moving object in the first frame image and the second frame image.

3. The method according to claim 2, wherein the first speed and the second speed are determined by using the center of mass of the moving object, and the determining the first speed and the second speed by using the center of mass of the moving object comprises: Determining the first speed based on the center of mass of the first moving object in the first frame of image and the center of mass of a first preceding image, wherein the first preceding image is a predetermined number of images in the video data to be labeled that are located before the first frame of image; as well as The second speed is determined based on the center of mass of the first moving object in a second preceding image and the center of mass of the second moving object in the second frame image, wherein the second preceding image is a predetermined number of images in the video data to be labeled that are located before the second frame image.

4. The method according to claim 2 or 3, wherein the performing identity association annotation on the same moving object in the first frame image and the second frame image based on the matching result and the first speed of the first moving object and the second speed of the second moving object comprises: determining the number of second moving objects that match the first moving object based on the matching result; When the number is equal to 1, determining the first moving object and the second moving object matched therewith as the same moving object; When the number is greater than 1, selecting a second moving object whose speed is closest to the first moving object from a plurality of second moving objects based on the first speed and the second speed, and determining the second moving object as the same moving object as the first moving object; as well as The first moving object and the second moving object determined to be the same moving object are labeled with identity association using the same identity identifier.

5. The method according to claim 4, wherein selecting a second moving object whose speed is closest to the first moving object from a plurality of second moving objects based on the first speed and the second speed and determining the second moving object as the same moving object as the first moving object comprises: respectively calculating the differences between the second speeds of the plurality of second moving objects and the first speed to obtain a plurality of speed differences; Comparing the absolute values ​​of the plurality of speed differences to obtain a speed difference with the smallest absolute value; as well as The second moving object corresponding to the speed difference with the smallest absolute value is determined as the same moving object.

6. The method according to claim 1, wherein the labeling information further includes category labeling information of moving objects, and performing labeling operations on the moving objects in the video data to be labeled comprises: Inputting the video data to be annotated into a predetermined annotation model to perform an annotation operation to obtain mask annotation information and category annotation information of the video data to be annotated; as well as A correction operation is performed on the mask annotation information and the category annotation information using a predetermined annotation method to obtain corrected mask annotation information and category annotation information.

7. The method according to claim 6, wherein the predetermined annotation model is a model obtained by training using a deep neural network model, and the method of obtaining the predetermined annotation model by training using a deep neural network comprises: Get training video data; Performing a labeling operation on the moving objects in the training video data using the predetermined labeling method to form training labeling information of the training video data, wherein the training labeling information includes training mask information and training category information; as well as The training video data and the training annotation information are input into the deep neural network model as training data to train the deep neural network model and obtain the predetermined annotation model.

8. The method according to claim 7, wherein the training of the deep neural network model further comprises: The corrected mask annotation information and category annotation information are input into the deep neural network model as new training data to continuously train and optimize the deep neural network model.

9. A device for labeling video data, comprising: processor; as well as A memory storing program instructions for marking video data, wherein when the program instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium storing program instructions for labeling video data, wherein when the program instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images