Image and video annotation system for small sample scene

Abstract

The embodiment of the invention provides a small sample scene-oriented image and video marking system, which comprises a client and a server, and is characterized in that the client is used for requesting an image file, marking a target marking detection frame and a key point in a picture or marking an event in a video, and marking the target marking detection frame and the key point in the image or marking an event in the video after the marking is completed; uploading the labeled image file to a server side in a set labeling format; the server side is used for managing an image data set, a video data set and annotation data, monitoring and controlling annotation quality of the client side, calculating user scores and conducting data distribution according to rules, and annotation quality detection, a data distribution strategy, an annotation sampling inspection strategy and a data augmentation algorithm are included. The method is advantaged in that improvement of data labeling quality is facilitated, a neural network training overfitting problem caused by too little image data is solved, an effect of obtaining large-batch diversified image data from small-batch image data is realized, and related technical research of subsequent neural network target detection and event detection is supported.

Description

Technical Field

[0001] The present invention relates to the field of video image detection technology based on deep learning in computer technology, and in particular to an image and video annotation system, which is an image and video annotation system for small sample scenes. Background Art

[0002] With the rapid iterative development of neural networks and deep learning technologies, in addition to traditional image processing tasks such as target detection, key point detection, and target tracking, neural networks also have the ability to achieve higher-order tasks, such as behavioral action recognition and job status recognition. On the other hand, the performance upper limit of neural networks often depends on the quantity and quality of training data. In some special cases, due to the restrictions of regulations and systems, it is difficult to obtain a large amount of raw data, such as surveillance videos along the railway. Therefore, considering these factors, higher requirements are also placed on the functions of the annotation system. Not only should the efficiency of annotation be solved, and the annotation errors caused by human negligence should be avoided as much as possible, but more importantly, the high quality, diversity, and low error rate of the annotation data should be guaranteed, so as to better guide the efficient convergence of the neural network, have stronger generalization ability, and achieve tasks such as target detection and event detection.

[0003] In the patented image annotation information processing method, device, server and system (inventors are Hu Yue, etc.), a multi-node image annotation system is proposed. The specific implementation mainly includes three nodes. The second node reviews the annotation result of the first node. If the task image is incorrectly annotated, it is returned to the first node. The third node is used to accept the processing result of the second node, and is also used to perform annotation sampling according to predetermined rules, and return the results to the first and second nodes. However, the patent is only limited to the detection box and detection category of the target in the image, and still relies on people to perform error checking, with a low level of intelligence.

[0004] In the patented method and device for monitoring image annotation quality (inventors: Han Fenghui, etc.), a method is proposed to improve the annotation quality of monitoring images. First, the annotation accuracy of a single task of annotators is obtained. Then, the variance between the accuracy of annotation data of multiple tasks and the preset average annotation accuracy is calculated. Finally, the number of sample images or image types to be sent to the annotation account is determined based on the variance. Through these processes, the annotation efficiency of annotators is improved and their error rate is reduced.

[0005] In the patented image annotation method, device and electronic device (inventors: Lian Shiguo, etc.), it is proposed to annotate images according to the acquired voice information, so as to improve the convenience of annotation and shorten the time period of annotation. In the patented interactive method for processing image annotation tasks based on a crowdsourcing platform (inventors: Ding Xianghua, etc.), an online crowdsourcing work mode is proposed, a classification system for image annotation tasks is constructed, and the design criteria and design process of the interactive method are established, so as to realize the interaction of image annotation tasks. The realized image interaction tasks mainly include image semantic annotation, image point annotation, image area annotation, and image sequence screening annotation. In the patented image annotation method and device (inventors: Liu Siyuan, etc.), it arranges image annotation and image annotation evaluation tasks for multiple annotators, so as to determine the annotation quality of each annotator respectively, and determine the annotator to distribute the image annotation task next time.

[0006] Existing methods have the following problems: (1) They have single functions. Most of them are only used for low-level image processing tasks such as image detection, semantic segmentation, and region classification, and are not suitable for high-level tasks such as video understanding and event detection. (2) The verification of annotations is relatively simple, often one node verifies another node. The judgment of data annotation quality is mostly determined by the annotation end, which involves subjective uncertainties. (3) They are unable to improve the diversity of small sample data and ignore data augmentation. Existing data augmentation methods contain few image transformation categories and the augmentation process is relatively fixed. When the image differences of each target object are small and the number distribution is uneven, it will make it difficult for the deep neural network to learn relevant features, thereby affecting the target detection and video tracking effects. Summary of the invention

[0007] In view of the above problems existing in the prior art, the embodiments of the present invention meet the needs of higher-order visual detection tasks of neural networks and propose an image and video annotation system for small sample scenes.

[0008] The embodiment of the present invention provides an image and video annotation system for small sample scenarios, including: a client and a server, wherein:

[0009] The client is used to request an image file, mark a detection frame and key points of an object in the image, or mark an event in a video, and after the marking is completed, upload the marked image file to the server in a set marking format;

[0010] The server is used to manage image data sets, video data sets and annotation data, monitor and control the annotation quality of the client, calculate user scores and distribute data according to rules, including annotation quality detection, data distribution strategy, annotation sampling strategy and data augmentation algorithm.

[0011] In some embodiments of the present invention, the annotation quality detection is used to describe the correctness of the annotation results of the client on the image file, including the use of an annotation quality monitoring algorithm based on a voting strategy for determination, in which the annotation results of different clients for the same image are voted on, and based on the principle of minority obeys majority, the annotation with more votes is considered correct.

[0012] In some embodiments of the present invention, the labeling quality detection, for a certain client, is specifically:

[0013] The server randomly extracts a certain proportion r of labeled data, and randomly selects another client to re-label it, and calculates the difference D between the two labeled results;

[0014] If the difference D ≥ β, where β is the threshold, it is considered suspicious labeled data and the server administrator is asked to make a judgment;

[0015] The correct annotations are adopted and the incorrect annotations are discarded. At the same time, the scores of the two clients are updated and the inspection ratio is dynamically adjusted;

[0016] When calculating the difference of the annotation results, three structures are included: the number of annotation boxes of the same annotation results is the same, the spatial position of the annotation boxes of the same annotation results is the same, and the category of the annotation boxes of the same annotation results is the same;

[0017] set up They are two annotation results for the same image, each row represents a annotation box, where p 0 is the coordinate of the upper left corner of the annotation box, l 0 is the length and width of the annotation box, t 0 For the annotation box type, the annotation result difference D is defined as follows:

[0018] D(L i ,L j )=αNum(L i ,L j )+βPos(L i ,L j )+(1-α-β)Type(L i ,L j )

[0019] Among them, α=0.4, β=0.1,

[0020] Num(L i ,L j )=||N i -N j || / max(N i ,N j ) is used to measure the difference in the number of annotation boxes of the annotation results, Ni and Nj represent the number of annotation boxes;

[0021] Pos(L i ,L j ) is used to measure the difference in the spatial position of the annotation boxes of the two annotation results. The intersection-over-union (IOU) ratio between the annotation boxes is used as the weight. The IOU value of the two annotation boxes is the ratio of the intersection area of ​​the two annotation boxes to the area of ​​the two annotation boxes. The Kuhn-Munkres algorithm is used to calculate the correspondence between the annotation boxes in the two annotation results.

[0022] Then, add the IOU values ​​between the corresponding annotation boxes to get Pos(L i ,L j ),

[0023] Type(L i ,L j ) is used to measure the difference between the annotation box types of the two annotation results, which is the number of corresponding annotation boxes with different types;

[0024] If D≥β, β=0.5, it is suspicious labeled data, please ask the server administrator to make a judgment, the client judged as wrong will score S=S-1; otherwise, the client score will not be affected;

[0025] If D<β, it is considered to be correctly labeled, and the client score S=S+1.

[0026] In some embodiments of the present invention, the labeling sampling strategy includes randomly selecting a certain proportion of the labeling data for quality inspection, dynamically adjusting the inspection ratio based on the real-time calculation of the labeling score of each client, lowering the inspection ratio for clients with high labeling quality, and increasing the inspection ratio for clients with low labeling quality, so as to maximize the labeling quality and labeling efficiency.

[0027] In some embodiments of the present invention, the labeling sampling strategy is specifically:

[0028] The sampling ratio r of the client is defined as follows:

[0029]

[0030] Among them, r b =30%, r lower =10%;

[0031] In some embodiments of the present invention, the data distribution strategy includes dynamically adjusting the data distribution ratio based on the annotation efficiency and calculating the annotation score of each client in real time, with a high data distribution ratio for clients with high annotation quality and a low data distribution ratio for clients with low annotation quality, so as to maximize the annotation quality and annotation efficiency;

[0032] The data distribution ratio p of the client is defined as p=1 / r.

[0033] In some embodiments of the present invention, the data augmentation algorithm is a multi-order data augmentation algorithm, which obtains a new image by applying a series of image transformation operations to the original image to achieve the expansion of the data set. The image transformation operators in the operator library include: fuzzy transformation, noise perturbation, convolution transformation, Euclidean transformation, contrast transformation, and other transformations including color channel transformation, shear transformation, flip transformation, pooling operation, atomization transformation and cloud transformation;

[0034] During the calculation, firstly, 3 types of operators from the above 6 major types are randomly selected, and the calculation order is randomly determined. Then, 2 sub-types of operators from the major types are randomly selected and the calculation order is randomly determined to obtain the first-order image augmentation calculation process.

[0035] Compared with the prior art, the beneficial effects of the image and video annotation system for small sample scenarios provided by the embodiment of the present invention are: it is conducive to improving the quality of data annotation, and specifically designs unified data distribution and data quality detection by the administrator server, so as to achieve consistency in the data annotation quality judgment standard, which is suitable for small data volume annotation scenarios. At the same time, it is also conducive to solving the problem of overfitting of neural network training caused by too little image data, and specifically proposes to embed image data augmentation technology based on image transformation in the annotation system, so as to achieve the effect of obtaining large batches of diversified image data from small batches of image data, and supports the subsequent research on related technologies of neural network target detection and event detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of the overall structure of an image and video annotation system for small sample scenarios provided by an embodiment of the present invention;

[0037] Figure 2 A schematic diagram of the flow of a multi-order data augmentation algorithm in an image and video annotation system for small sample scenarios provided by an embodiment of the present invention;

[0038] Figure 3 A flowchart for implementing target detection and annotation in an image and video annotation system for small sample scenarios provided by an embodiment of the present invention;

[0039] Figure 4 A flowchart for implementing video event annotation in an image and video annotation system for small sample scenarios provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0041] Various aspects and features of the present application are described herein with reference to the accompanying drawings.

[0042] These and other characteristics of the present application will become apparent from the following description of a preferred form of embodiment given as a non-limiting example with reference to the accompanying drawings.

[0043] It should also be understood that, although the present application has been described with reference to some specific examples, those skilled in the art will be able to realize many other equivalent forms of the present application that have the features described in the claims and are therefore within the scope of protection defined thereby.

[0044] The above and other aspects, features and advantages of the present application will become more apparent in view of the following detailed description when taken in conjunction with the accompanying drawings.

[0045] Specific embodiments of the present application are described hereinafter with reference to the accompanying drawings; however, it should be understood that the embodiments applied for are merely examples of the present application, which may be implemented in a variety of ways. Well-known and / or repeated functions and structures are not described in detail to determine the true intent based on the user's historical operations and to avoid unnecessary or redundant details that make the present application unclear. Therefore, the specific structural and functional details applied for herein are not intended to be limiting, but are merely used as the basis and representative basis for the claims to teach those skilled in the art to use the present application in a variety of ways with substantially any suitable detailed structure.

[0046] This specification may use the phrases "in one embodiment," "in another embodiment," "in yet another embodiment," or "in other embodiments," all of which may refer to one or more of the same or different embodiments according to the present application.

[0047] The embodiment of the present invention provides an image and video annotation system for small sample scenarios. Figures 1 to 4 As shown, it includes: a client and a server, wherein,

[0048] The client is used to request an image file, mark a detection frame and key points of an object in the image, or mark an event in a video, and after the marking is completed, upload the marked image file to the server in a set marking format;

[0049] The server is used to manage image data sets, video data sets and annotation data, monitor and control the annotation quality of the client, calculate user scores and distribute data according to rules, including annotation quality detection, data distribution strategy, annotation sampling strategy and data augmentation algorithm.

[0050] In the above embodiment, the annotation system adopts the Client / Server mode, and the overall system architecture is as follows: Figure 1As shown in the figure. On the Client side, users use annotation software to annotate images or videos; on the Server side, annotation quality is calculated in real time and an error correction mechanism is used to repair incorrect annotations, thus realizing image annotation quality monitoring and data distribution control. After data augmentation, efficient and high-quality image annotation is ensured.

[0051] The client side includes multiple users, requesting image files, supporting users to interactively annotate images and returning the annotation results. The user first accepts the image sent by the server side, uses open source annotation software such as labelimg or labelme to annotate the detection box and key points of the target in the image, or annotate the events in the video, and then uploads it to the server side in a unified annotation format after the annotation is completed.

[0052] The server is responsible for storing and managing image / video datasets and annotation data, monitoring and controlling the annotation quality of the client, calculating user scores and distributing data according to rules. It mainly includes four strategies or algorithms: annotation quality detection, data distribution strategy, annotation sampling strategy, and data augmentation algorithm. The detailed schemes of each strategy or algorithm are as follows:

[0053] Labeling quality inspection

[0054] The annotation quality describes the correctness of the annotation results of a client's image. We believe that the annotations that are widely recognized are correct annotations. Based on this, a annotation quality monitoring algorithm based on a voting strategy is proposed. In this algorithm, the annotation results of different clients for the same image are voted on. Based on the principle of minority obeys majority, the annotation with more votes is considered correct.

[0055] For a certain client, the server randomly extracts a certain proportion r of labeled data, and randomly selects another client to re-label, and calculates the difference D between the two labeled results. If the difference D ≥ β (β is the threshold), it is considered as suspicious labeled data and the server administrator is asked to make a judgment. The correct annotation is adopted, and the wrong annotation is discarded. At the same time, the scores of the two clients are updated and the inspection ratio is dynamically adjusted.

[0056] When calculating the difference of the annotation results, three aspects are considered: 1) the number of annotation boxes of the same annotation results is the same; 2) the spatial position of the annotation boxes of the same annotation results is the same; 3) the category of the annotation boxes (target category) of the same annotation results is the same. They are two annotation results for the same image, each row represents a annotation box (p 0 is the coordinate of the upper left corner of the annotation box, l 0 is the length and width of the annotation box, t 0 is the annotation box type), the annotation result difference D is defined as follows:

[0057] D(Li ,L j )=αNum(L i ,L j )+βPos(L i ,L j )+(1-α-β)Type(L i ,L j )

[0058] Among them, α=0.4 and β=0.1. Num(L i ,L j )=||N i -N j || / max(N i ,N j ) measures the difference in the number of annotation boxes of the annotation results, Ni and Nj represent the number of annotation boxes. Pos(L i ,L j ) measures the difference in the spatial position of the annotation boxes of the two annotation results, and uses the intersection-over-union ratio (IOU) between the annotation boxes (the IOU value of the intersection-over-union ratio of the two annotation boxes is the ratio of the intersection area of ​​the two annotation boxes to the area of ​​the two annotation boxes) as the weight, and uses the KM (Kuhn Munkres) algorithm to calculate the correspondence between the annotation boxes in the two annotation results; then, add the IOU values ​​between the corresponding annotation boxes to get Pos(L i ,L j ). Type(L i ,L j ) measures the difference between the annotation box types of the two annotation results, which is the number of corresponding annotation boxes with different types.

[0059] If D≥β(β=0.5), it is suspicious labeled data, please ask the server administrator to make a judgment, the client judged as wrong will score S=S-1; otherwise, the client score will not be affected. If D<β, it is considered to be correctly labeled, and the client score is S=S+1.

[0060] Label sampling strategy

[0061] A certain proportion of the labeled data is randomly selected for quality inspection. The inspection ratio is dynamically adjusted based on the real-time calculation of the annotation scores of each client. The inspection ratio of clients with high annotation quality is reduced, and the inspection ratio of clients with low annotation quality is increased to maximize the annotation quality and efficiency. The client inspection ratio r is defined as follows:

[0062]

[0063] Among them, r b =30%, r lower =10%.

[0064] Data distribution strategy

[0065] Taking into account the labeling efficiency, the data distribution ratio is dynamically adjusted based on the real-time calculation of the labeling scores of each client. The data distribution ratio of clients with high labeling quality is high, and the data distribution ratio of clients with low labeling quality is low, so as to maximize the labeling quality and labeling efficiency. The data distribution ratio p of the client is defined as p = 1 / r.

[0066] Multi-order data augmentation algorithm

[0067] A multi-order data augmentation technique based on an image transformation operator library is proposed. A new image is obtained by applying a series of image transformation operations to the original image to expand the data set. The image transformation operators in the operator library include: blur transformation, noise perturbation, convolution transformation, Euclidean transformation, contrast transformation, color channel transformation, shear transformation, flip transformation, pooling operation, fog transformation, cloud transformation, etc. The operator library categories and calculation process of the multi-order data augmentation algorithm are as follows: Figure 2 As shown in the figure. During the calculation, first randomly select 3 of the 6 major categories of operators and randomly determine the order of calculation, then randomly select 2 subcategories of the major categories and randomly determine the order of calculation, and obtain the first-order image augmentation calculation process. Similarly, determine the second-order, third-order, and other image augmentation calculation processes, and finally connect them in series to form a multi-order data augmentation process.

[0068] Through the technical solution, it can be seen that the above-mentioned image and video annotation system for small sample scenes provides a set of integrated tools for image target detection annotation and video event annotation, which solves the problem of single function of existing annotation tools and provides support for higher-level computer vision tasks; at the same time, it also improves the quality of data annotation. Small sample data is precious, and high-quality, error-free annotation is particularly important for the training of neural networks. Therefore, a data annotation process of distribution-annotation-sampling-redistribution of the Server-Client architecture is proposed. The administrator on the Server side is used as the annotation quality inspection center to perform annotation inspection and scoring on each Client side, and adjust the data sampling and distribution strategy according to the scoring situation. Furthermore, it also integrates data augmentation strategies into the annotation tool to improve data diversity. A multi-order image augmentation algorithm that integrates multiple image processing algorithms is constructed, and multiple randomness is introduced in the data augmentation process to improve the spatial complexity of a single sample and the difference between multiple samples, thereby improving the training efficiency and generalization ability of the neural network.

[0069] In order to facilitate the understanding of the above technical solution, the following is a detailed description with reference to specific examples, as follows:

[0070] According to the labeling task, the specific implementation method is divided into the target detection box labeling step and the video event labeling step.

[0071] (1) Object detection and annotation

[0072] The implementation flow chart of target detection and annotation is as follows: Figure 3 The detailed implementation steps are as follows:

[0073] Step 1: Data storage: Extract a set of the most distinctive frame images from the surveillance video, archive and classify them, and thus form a set of target image data sets.

[0074] Step 2: Data distribution. The server sends the image file path to the client, and the client obtains the file path and accesses the corresponding file. The annotation data file generated by the client has the same name as the image file and is stored in the same path.

[0075] Step 3: Data annotation. On the client side, the user uses image annotation software to annotate the image. The annotation data file content includes the location of the target object (the coordinates of the upper left corner and the lower right corner of the identification box), the category and the client ID.

[0076] Step 4: Annotation and storage. After the annotation is completed, the results are uploaded to the server.

[0077] Step 5: Labeling quality detection. In order to monitor the labeling quality, for a certain client, the server randomly extracts a certain proportion r of the labeling data, and randomly selects another client to re-label, and calculates the difference D between the two labeling results.

[0078] set up They are two annotation results for the same image, each row represents a annotation box (p 0 is the coordinate of the upper left corner of the annotation box, l 0 is the length and width of the annotation box, t 0 The difference D of the annotation result is calculated by the following formula:

[0079] D(L i ,L j )=αNum(L i ,L j )+βPos(L i ,L j )+(1-α-β)Type(L i ,L j )

[0080] Among them, α=0.4 and β=0.1. Num(L i ,L j )=||N i -N j || / max(N i ,N j ) measures the difference in the number of annotation boxes of the annotation results, Ni and Nj represent the number of annotation boxes. Pos(L i ,Lj ) measures the difference in the spatial position of the annotation boxes of the two annotation results, and uses the intersection-over-union ratio (IOU) between the annotation boxes (the IOU value of the intersection-over-union ratio of the two annotation boxes is the ratio of the intersection area of ​​the two annotation boxes to the area of ​​the two annotation boxes) as the weight, and uses the KM (Kuhn Munkres) algorithm to calculate the correspondence between the annotation boxes in the two annotation results; then, add the IOU values ​​between the corresponding annotation boxes to get Pos(L i ,L j ). Type(L i ,L j ) measures the difference between the annotation box types of the two annotation results, which is the number of corresponding annotation boxes with different types.

[0081] Step 6: Update the client score. If the difference D ≥ β (β = 0.5), it is considered suspicious annotation data and judged by the server. The client score S = S-1 is judged to be wrong, and the wrong annotation is discarded.

[0082] If D<β, it is considered to be correctly labeled, the labeling result is adopted, and the client score S=S+1.

[0083] Step 7: Adjust the sampling ratio. Calculate the sampling ratio r of the client by the following formula:

[0084]

[0085] Among them, r b =30%, r lower = 10%. The result of sampling ratio r is updated to the fourth step.

[0086] Step 8: Data distribution ratio adjustment. Calculate the client's data distribution ratio p = 1 / r. The result of the data distribution ratio p is updated to the second step.

[0087] Step 9: Data augmentation. A new image is obtained by applying a series of image transformation operations to the original image to expand the data set. During the calculation, first randomly select 3 of the 6 major categories of operators and randomly determine the order of calculation. Then randomly select 2 subcategories of the major categories and randomly determine the order of calculation to obtain the first-order image augmentation calculation process. Similarly, determine the second-order, third-order, and other image augmentation calculation processes, and finally connect them in series to form a multi-order data augmentation process.

[0088] Step 10: Augmented data storage. Classify the augmented images and annotations. Return to step 2.

[0089] (2) Video Event Annotation

[0090] The implementation flow chart of video event annotation is as follows: Figure 4 The detailed implementation steps are as follows:

[0091] Step 1: Data storage: Extract a set of the most distinctive frame images from the surveillance video, archive and classify them, and thus form a set of target image data sets.

[0092] Step 2: Data distribution. The server sends the image file path to the client, and the client obtains the file path and accesses the corresponding file. The annotation data file generated by the client has the same name as the image file and is stored in the same path.

[0093] Step 3: Data annotation. On the client side, the user uses image annotation software to annotate the image. The annotation data file content includes the location of the target object (the coordinates of the upper left corner and the lower right corner of the identification box), the category and the client ID.

[0094] Step 4: Annotation and storage. After the annotation is completed, the results are uploaded to the server.

[0095] Step 5: Labeling quality detection. In order to monitor the labeling quality, for a certain client, the server randomly extracts a certain proportion r of the labeling data, and randomly selects another client to re-label, and calculates the difference D between the two labeling results.

[0096] set up They are two annotation results for the same image, each row represents a annotation box (p 0 is the coordinate of the upper left corner of the annotation box, l 0 is the length and width of the annotation box, t 0 The difference D of the annotation result is calculated by the following formula:

[0097] D(L i ,L j )=αNum(L i ,L j )+βPos(L i ,L j )+(1-α-β)Type(L i ,L j )

[0098] Among them, α=0.4 and β=0.1. Num(L i ,L j )=||N i -N j || / max(N i ,N j ) measures the difference in the number of annotation boxes of the annotation results, Ni and Nj represent the number of annotation boxes. Pos(L i ,L j) measures the difference in the spatial position of the annotation boxes of the two annotation results, and uses the intersection-over-union ratio (IOU) between the annotation boxes (the IOU value of the intersection-over-union ratio of the two annotation boxes is the ratio of the intersection area of ​​the two annotation boxes to the area of ​​the two annotation boxes) as the weight, and uses the KM (Kuhn Munkres) algorithm to calculate the correspondence between the annotation boxes in the two annotation results; then, add the IOU values ​​between the corresponding annotation boxes to get Pos(L i ,L j ). Type(L i ,L j ) measures the difference between the annotation box types of the two annotation results, which is the number of corresponding annotation boxes with different types.

[0099] Step 6: Update the client score. If the difference D ≥ β (β = 0.5), it is considered suspicious annotation data and judged by the server. The client score S = S-1 is judged to be wrong, and the wrong annotation is discarded.

[0100] If D<β, it is considered to be correctly labeled, the labeling result is adopted, and the client score S=S+1.

[0101] Step 7: Adjust the sampling ratio. Calculate the sampling ratio r of the client by the following formula:

[0102]

[0103] Among them, r b =30%, r lower = 10%. The result of sampling ratio r is updated to the fourth step.

[0104] Step 8: Data distribution ratio adjustment. Calculate the client's data distribution ratio p = 1 / r. The result of the data distribution ratio p is updated to the second step.

[0105] Step 9: Data augmentation. A new image is obtained by applying a series of image transformation operations to the original image to expand the data set. During the calculation, first randomly select 3 of the 6 major categories of operators and randomly determine the order of calculation. Then randomly select 2 subcategories of the major categories and randomly determine the order of calculation to obtain the first-order image augmentation calculation process. Similarly, determine the second-order, third-order, and other image augmentation calculation processes, and finally connect them in series to form a multi-order data augmentation process.

[0106] Step 10: Augmented data storage. Classify the augmented images and annotations. Return to step 2.

[0107] In combination with the above embodiments, it can be seen that the image and video annotation system for small sample scenarios provided by the above embodiments of the present invention is conducive to improving the quality of data annotation. It specifically designs unified data distribution and data quality detection by the administrator server, so as to achieve consistency in the data annotation quality judgment standard, which is suitable for small data volume annotation scenarios. At the same time, it is also conducive to solving the problem of overfitting of neural network training caused by too little image data. It specifically proposes to embed image data augmentation technology based on image transformation in the annotation system, which achieves the effect of obtaining large batches of diversified image data from small batches of image data, and supports the subsequent research on related technologies of neural network target detection and event detection.

[0108] The above embodiments are only exemplary embodiments of the present invention and are not intended to limit the present invention. The protection scope of the present invention is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present invention within the essence and protection scope of the present invention, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present invention.

Claims

1. An image and video annotation system for small sample scenarios, characterized in that: include: Client and server, where The client is used to request an image file, mark a detection frame and key points of an object in the image, or mark an event in a video, and after the marking is completed, upload the marked image file to the server in a set marking format; The server is used to manage image data sets, video data sets and annotation data, monitor and control the annotation quality of the client, calculate user scores and distribute data according to rules, including annotation quality detection, data distribution strategy, annotation sampling strategy and data augmentation algorithm.

2. The image and video annotation system for small sample scenarios according to claim 1, characterized in that: The annotation quality detection is used to describe the correctness of the annotation results of the client on the image file, including the use of an annotation quality monitoring algorithm based on a voting strategy to determine the correctness. In this algorithm, the annotation results of different clients for the same image are voted on, and based on the principle of minority obeys majority, the annotation with more votes is considered correct.

3. The image and video annotation system for small sample scenarios according to claim 2, characterized in that: The labeling quality detection, for a certain client, is specifically as follows: The server randomly extracts a certain proportion r of labeled data, and randomly selects another client to re-label it, and calculates the difference D between the two labeled results; If the difference D ≥ β, where β is the threshold, it is considered suspicious labeled data and the server administrator is asked to make a judgment; The correct annotations are adopted and the incorrect annotations are discarded. At the same time, the scores of the two clients are updated and the inspection ratio is dynamically adjusted; When calculating the difference of the annotation results, three structures are included: the number of annotation boxes of the same annotation results is the same, the spatial position of the annotation boxes of the same annotation results is the same, and the category of the annotation boxes of the same annotation results is the same; set up These are two annotation results for the same image. Each row represents a annotation box, where p0 is the coordinate of the upper left corner of the annotation box, l0 is the length and width of the annotation box, and t0 is the type of the annotation box. The difference D of the annotation results is defined as follows: D(L i ,L j )=αNum(L i ,L j )+βPos(L i ,L j )+(1-α-β)Type(L i ,L j ) Among them, α=0.4, β=0.1, Num(L i ,L j )=||N i -N j || / max(N i ,N j ) is used to measure the difference in the number of annotation boxes of the annotation results, Ni and Nj represent the number of annotation boxes; Pos(L i ,L j ) is used to measure the difference in the spatial position of the annotation boxes of the two annotation results. The intersection-over-union (IOU) ratio between the annotation boxes is used as the weight. The IOU value of the two annotation boxes is the ratio of the intersection area of ​​the two annotation boxes to the area of ​​the two annotation boxes. The Kuhn-Munkres algorithm is used to calculate the correspondence between the annotation boxes in the two annotation results. Then, add the IOU values ​​between the corresponding annotation boxes to get Pos(L i ,L j ), Type(L i ,L j ) is used to measure the difference between the annotation box types of the two annotation results, which is the number of corresponding annotation boxes with different types; If D≥β, β=0.5, it is suspicious labeled data, please ask the server administrator to make a judgment, the client judged as wrong will score S=S-1; otherwise, the client score will not be affected; If D<β, it is considered to be correctly labeled, and the client score S=S+1.

4. The image and video annotation system for small sample scenarios according to claim 3, characterized in that: The labeling sampling strategy includes randomly selecting a certain proportion of the labeling data for quality inspection, and dynamically adjusting the inspection ratio based on the real-time calculation of the labeling score of each client. The inspection ratio of clients with high labeling quality is reduced, and the inspection ratio of clients with low labeling quality is increased, so as to maximize the labeling quality and labeling efficiency.

5. The image and video annotation system for small sample scenarios according to claim 4, characterized in that: The labeling sampling strategy is specifically as follows: The sampling ratio r of the client is defined as follows: Among them, r b =30%, r lower =10%.

6. The image and video annotation system for small sample scenarios according to claim 5, characterized in that: The data distribution strategy includes dynamically adjusting the data distribution ratio based on the annotation efficiency and calculating the annotation scores of each client in real time, with a high data distribution ratio for clients with high annotation quality and a low data distribution ratio for clients with low annotation quality, so as to maximize the annotation quality and annotation efficiency; The data distribution ratio p of the client is defined as p=1 / r.

7. The image and video annotation system for small sample scenarios according to claim 6, characterized in that: The data augmentation algorithm is a multi-order data augmentation algorithm, which obtains a new image by applying a series of image transformation operations to the original image to expand the data set. The image transformation operators in the operator library include: fuzzy transformation, noise perturbation, convolution transformation, Euclidean transformation, contrast transformation, and other transformations including color channel transformation, shear transformation, flip transformation, pooling operation, atomization transformation and cloud transformation; During the calculation, firstly, 3 types of operators from the above 6 major types are randomly selected, and the calculation order is randomly determined. Then, 2 sub-types of operators from the major types are randomly selected and the calculation order is randomly determined to obtain the first-order image augmentation calculation process.

Images