Method, system and product for annotating images

Abstract

The present disclosure relates to methods, systems, and program products for annotating images. The method comprises the steps of determining an identification result of an image based on a trained AI model, and the AI model is obtained by training a labeled image. The method further comprises the step of obtaining feedback of the user to the recognition result, wherein the feedback is used for indicating the recognition accuracy of the recognition result. And in response to the feedback, the method indicates that the recognition accuracy of the recognition result meets a preset condition, and the recognition result determined by the AI model is used as annotation data of the image. In response to the feedback, the method indicates that the recognition accuracy of the recognition result does not meet the preset condition, and the re-annotation of the image by the user is used as the annotation data of the image.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence (AI) technology, and more particularly, to a method, system, and program product for annotating images. Background Art

[0002] At present, the field of AI technology is developing rapidly. The development of AI technology has benefited from the annotation work of a large amount of basic images and other data. AI image annotation refers to the process of annotating images so that computers can recognize and understand the content of images by learning these annotated data. Specifically, the work of image annotators mainly includes image classification annotation, image detection annotation and image segmentation annotation. Among them, image classification annotation refers to dividing images into different categories, such as animals, plants, buildings, etc., which usually requires selecting appropriate labels from predefined labels to annotate the images. Image detection annotation is to frame or mark specific objects in the image, such as faces, vehicles, etc. This annotation method is widely used in fields such as autonomous driving and security monitoring. Image segmentation annotation is to mark objects in the image along the contours, including multiple types such as semantic segmentation.

[0003] For AI models to be accurate and effective, high-quality image annotations are required as training samples. Therefore, image annotators must have strong comprehension skills, able to understand image content and accurately annotate it. They also need strong communication skills, communicating effectively with other team members to ensure the accuracy and consistency of annotations. Furthermore, they must be patient and attentive, as image annotation requires a high degree of patience and care. Incorrect or inconsistent annotations will affect the learning effect of the AI ​​model. Summary of the Invention

[0004] In a first aspect of an embodiment of the present disclosure, a method for annotating an image is provided. The method includes determining a recognition result of an image based on a trained AI model, wherein the AI ​​model is obtained by training based on annotated images. The method also includes obtaining user feedback on the recognition result, wherein the feedback is used to indicate the recognition accuracy of the recognition result. In response to the feedback indicating that the recognition accuracy of the recognition result meets a predetermined condition, the method uses the recognition result determined by the AI ​​model as the annotation data of the image. The method also uses the user's re-annotation of the image as the annotation data of the image in response to the feedback indicating that the recognition accuracy of the recognition result does not meet the predetermined condition.

[0005] In a second aspect of an embodiment of the present disclosure, a system for annotating an image is provided. The system includes a processor and a memory coupled to the processor. The memory has instructions stored therein, which cause the processor to execute a method for annotating an image. The method includes determining a recognition result of an image based on a trained AI model, wherein the AI ​​model is obtained by training based on annotated images. The method also includes obtaining user feedback on the recognition result, wherein the feedback is used to indicate the recognition accuracy of the recognition result. And in response to the feedback indicating that the recognition accuracy of the recognition result meets a predetermined condition, the method uses the recognition result determined by the AI ​​model as the annotation data of the image. The method also uses the user's re-annotation of the image as the annotation data of the image in response to the feedback indicating that the recognition accuracy of the recognition result does not meet the predetermined condition.

[0006] In a third aspect of an embodiment of the present disclosure, a computer program product is provided. The computer program product is tangibly stored on a non-transitory computer-readable medium and includes machine-executable instructions, which, when executed, enable a machine to implement a method for annotating an image provided in accordance with the present disclosure. The method includes determining a recognition result of an image based on a trained AI model, wherein the AI ​​model is obtained by training based on annotated images. The method also includes obtaining user feedback on the recognition result, wherein the feedback is used to indicate the recognition accuracy of the recognition result. And in response to the feedback indicating that the recognition accuracy of the recognition result meets a predetermined condition, the method uses the recognition result determined by the AI ​​model as the annotation data of the image. The method also uses the user's re-annotation of the image as the annotation data of the image in response to the feedback indicating that the recognition accuracy of the recognition result does not meet the predetermined condition.

[0007] In a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein machine-executable instructions are stored on the computer-readable medium, and when the machine-executable instructions are executed, the machine implements a method for annotating an image provided in accordance with the present disclosure. The method includes determining a recognition result of an image based on a trained AI model, wherein the AI ​​model is obtained by training based on annotated images. The method also includes obtaining user feedback on the recognition result, wherein the feedback is used to indicate the recognition accuracy of the recognition result. And in response to the feedback indicating that the recognition accuracy of the recognition result meets a predetermined condition, the method uses the recognition result determined by the AI ​​model as the annotation data of the image. The method also uses the user's re-annotation of the image as the annotation data of the image in response to the feedback indicating that the recognition accuracy of the recognition result does not meet the predetermined condition.

[0008] It should be understood that the contents described in the Summary of the Invention section are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram of an image for annotation according to some embodiments of the present disclosure is shown;

[0011] Figure 2 A flowchart of a method for annotating an image according to some embodiments of the present disclosure is shown;

[0012] Figure 3 A flowchart of a method for annotating an image and generating an AI model according to some embodiments of the present disclosure is shown;

[0013] Figure 4 A flowchart of a method for obtaining user annotation data for a target image according to some embodiments of the present disclosure is shown;

[0014] Figure 5 A schematic diagram of block cutting based on the outline of an object is shown;

[0015] Figure 6 A flowchart of a method for obtaining user annotation data for a target image according to some other embodiments of the present disclosure is shown;

[0016] Figure 7 A schematic diagram for marking the boundaries of an object in an image according to some embodiments of the present disclosure is shown;

[0017] Figure 8 A flowchart of a method for labeling an object segmentation model according to some embodiments of the present disclosure is shown;

[0018] Figure 9 A flowchart of a method for labeling an object detection model according to some embodiments of the present disclosure is shown;

[0019] Figure 10 A flowchart of a method for labeling an image recognition model according to some embodiments of the present disclosure is shown;

[0020] Figure 11 A block diagram of an electronic device 1100 is shown in which various embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION

[0021] The following will describe embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure. The embodiments of the present disclosure described below with reference to the accompanying drawings are for illustrative purposes only.

[0022] As mentioned above, when training an AI model, it is usually necessary to annotate a large number of training images. In applications where different targets need to be segmented in an image, the annotators need to annotate along the edges of the target object as much as possible. For object detection scenarios, annotators are required to wrap the outer contour of the target object with a polygonal box. In image recognition (image classification) scenarios, annotators need to manually identify the image according to the requirements of the scenario, and then make judgments and annotations on the image. During the annotation operation, keyboard and mouse operations are currently commonly used. For scenes that require fine marking, the operation requirements for the annotators are high, and usually training and practice are required to meet the requirements. At the same time, since the amount of training data required by the AI ​​model is large, it is necessary to manually repeat the above fine process to annotate each image, which is time-consuming and labor-intensive, and it is difficult to ensure the stability of the annotation.

[0023] Specifically, the existing technology mainly has the following problems:

[0024] One type of problem is that for image segmentation applications, labelers need to manually use the keyboard and mouse to label along the edges of the target object as much as possible, which requires high operating skills.

[0025] Another problem is that for almost all AI model training scenarios, a large amount of image data needs to be labeled. This is because in many application scenarios, the targets to be identified are unique, making it difficult to automatically identify any target, especially unknown objects, using a general model. In order for the model to accurately identify and distinguish unknown targets, it still needs to be trained using a large number of labeled training images. Therefore, a tool that can quickly and easily label a large number of model training images is needed;

[0026] Another problem is that current touch annotation methods rely heavily on the annotator's touch control skills when tracking the details of the target's edges by simply counting the touch point locations and frequencies. This makes it difficult for inexperienced users to perform fine annotation along the target's edges.

[0027] Furthermore, current annotation technologies still require the use of the annotation results to train deep learning models for application cases. This process requires users to have a certain understanding of deep learning technology, which is a relatively high barrier to entry.

[0028] Therefore, there is a need for a method for annotating images, which enables annotators to annotate images quickly and easily without professional training. Alternatively, annotators can also perform precise and accurate annotation without professional training.

[0029] Optionally, some embodiments of the present disclosure provide a method and system for annotating images. This method, utilizing a user interface (UI), enables users to semi-automatically annotate image data used to train deep learning models using various methods, such as finger touch, stylus, keyboard, and mouse, combined with AI technology, on multiple hardware platforms, including PCs, tablets, and smart mobile devices. This method can be used for annotating image data used for training computer vision and generative machine learning models, as well as for training machine learning models for various application scenarios after annotation.

[0030] As mentioned above, when training an image processing deep learning model, the training images must first be annotated. During the annotation process, manual interpretation of the images is typically required to identify the objects to be annotated and then perform manual annotation. For various deep learning application scenarios, such as image segmentation, object detection, and image recognition, the manual annotation process is tedious, time-consuming, and labor-intensive. Especially for various image segmentation and annotation tasks, since the target annotation area must be drawn as close as possible to the edges of the target objects in the training image, the annotation process places high demands on the annotators, who often require a period of training to be competent. In addition, for some deep learning-based computer vision tasks, the definition of the annotation target range requires experienced annotators to discern and carefully define the annotated target range. In contrast, using one or more embodiments of the method or system of the present disclosure, after annotating the image data, the present disclosure can directly output the trained deep model, which can be directly deployed on supported hardware platforms. Using one or more embodiments of the method or system of the present disclosure, annotators can use touch-screen annotation, and for image segmentation annotation, the system can automatically identify the target edges. Therefore, the training requirements and experience of labelers can be greatly reduced, allowing almost any inexperienced user to use the UI-based method in this technical solution to quickly label data for different visual application scenarios and complete deep learning model training.

[0031] Figure 1A schematic diagram of an image 100 that can be used for annotation according to some embodiments of the present disclosure is shown. Image 100 schematically shows objects 110, object 120, and object 130 that need to be annotated. Of course, the images of the present disclosure are not limited to those shown in image 100. Image 100 is only used as an example to make one or more embodiments of the present disclosure easier to understand, and should not be regarded as limiting the scope of protection of the present disclosure. Those skilled in the art can completely select images that need to be annotated according to actual needs. For example, the image may include at least one of a medical image, a test strip reagent test result image, a human tissue lesion image, a computed tomography (CT) image, a magnetic resonance imaging (MRI) image, an X-ray film, a pathological section image, a microscope image, an ultrasound image, etc. Optionally, the object in the image may include various image elements that require attention. Therefore, any image suitable for the present disclosure should fall within the scope of protection of the present disclosure, and one or more embodiments of the present disclosure can be applied to any suitable practical application scenario, such as image annotation in the medical field and training of AI models.

[0032] Figure 2 A flowchart of a method 200 for annotating an image according to some embodiments of the present disclosure is shown. Each step of the method 200 may be performed by a system, device, or program product that implements the method. At box 210, a recognition result of the image is determined based on a trained AI model, wherein the AI ​​model is obtained by training based on annotated images. Optionally, the AI ​​model may be a deep learning model, and optionally, the AI ​​model may also be an incremental learning AI model. Optionally, the AI ​​model includes at least one of an AI model that recognizes objects in an image one by one, an AI model that recognizes a target object from objects in an image, or an AI model for overall recognition of objects in an image. Optionally, the recognition result includes at least one of a category, a contour, or a boundary of at least one object in the image, for example Figure 1 At least one of the categories, outlines, or boundaries of objects 110, 120, or 130 in the image processing unit. Optionally, the labeled images may include a small number of manually labeled images, or a small number of manually labeled images and images automatically labeled by the AI ​​model. The labeled images may also include only a small number of manually labeled images or only images automatically labeled by the AI ​​model. Those skilled in the art can fully select which labeled images to use as training samples based on actual needs. Any labeled images suitable for the present disclosure should fall within the scope of protection of the present disclosure.

[0033] At block 220 , user feedback on the recognition result is obtained, where the feedback indicates the recognition accuracy of the recognition result. Optionally, method 200 can provide a way for the user to provide feedback on the recognition result, such as a UI input interface. The user can judge the recognition result based on their own experience and provide feedback. User feedback indicates the recognition accuracy of the recognition result. For example, the user may believe that the AI ​​model accurately identified the object category of the object in the image. Optionally, this may also include the user believing that the AI ​​model accurately identified the object contour of the object in the image. Optionally, this may also include the user believing that the AI ​​model accurately identified the object boundary of the object in the image. In this case, the user can provide positive feedback on the recognition result, indicating that the AI ​​model's recognition result is correct. Alternatively, or conversely, for example, the user may believe that the AI ​​model incorrectly identified the object category of the object in the image. Optionally, this may also include the user believing that the AI ​​model incorrectly identified the object contour of the object in the image. Optionally, this may also include the user believing that the AI ​​model incorrectly identified the object boundary of the object in the image. In this case, the user can give negative feedback to the recognition result, which indicates that the recognition result of the AI ​​model is wrong.

[0034] At block 230, in response to feedback indicating that the recognition accuracy of the recognition result meets predetermined conditions, the recognition result determined by the AI ​​model is used as the annotated data for the image. Optionally, the predetermined conditions can be set based on actual needs. For example, if the trained model has very high requirements for annotated images and requires precise annotation, high predetermined conditions can be set for the recognition results determined by the AI ​​model. For example, all recognition result evaluation metrics (such as precision, mAP (mean average accuracy), IoU (intersection over union), etc.) must be accurate, that is, a 100% accuracy requirement is required, where the identified object categories, object outlines, and boundaries are all accurate. Conversely, if the trained model has less stringent requirements for annotated images, a relatively low predetermined condition can be set, such as a 90% accuracy requirement, where the identified object categories have an accuracy of at least 95% and the object outlines or boundaries have an accuracy of at least 90%. Those skilled in the art can select appropriate predetermined conditions based on actual needs. Any setting of predetermined conditions suitable for the present disclosure should fall within the scope of protection of the present disclosure. Optionally, the accuracy rate can be calculated by the ratio of accurately labeled images to the total images. Of course, those skilled in the art can choose a suitable method for calculating the accuracy rate according to actual needs. Any suitable method for calculating the accuracy rate suitable for the present disclosure should fall within the scope of protection of the present disclosure. It is worth noting that the above-mentioned accuracy rate as a predetermined condition is only an example. In the model for judging image segmentation or object detection, multiple indicators can also be used, such as the correct rate of recognition category, the overlap rate between the recognition area and the labeled area, etc. as comprehensive judgment criteria. Those skilled in the art can formulate corresponding predetermined conditions according to actual needs. Any setting of the predetermined conditions suitable for the present invention should fall within the scope of protection of the present disclosure.

[0035] At block 240, in response to feedback indicating that the recognition accuracy of the recognition result does not meet the predetermined condition, the user's re-annotation of the image is used as the image's annotation data. Optionally, as an example, the user believes that the AI ​​model's identification of the object category of the object in the image is incorrect. Optionally, this may also include the user believing that the AI ​​model's identification of the object outline of the object in the image is also incorrect. Optionally, this may also include the user believing that the AI ​​model's identification of the object boundary of the object in the image is also incorrect. In this case, the user may provide negative feedback on the recognition result, indicating that the AI ​​model's recognition result is incorrect. In this case, the recognition accuracy of the AI ​​model's recognition result fails to meet the predetermined condition. Alternatively, the AI ​​model's recognition result may be partially accurate while others are inaccurate, failing to achieve the 100% or 90% accuracy set by the predetermined condition. Optionally, method 200 may determine that the AI ​​model's recognition result is unusable and may provide a user-accessible UI interface to obtain the user's manual re-annotation of the image and use the user's re-annotation of the image as the image's annotation data.

[0036] It can be seen that by adopting the method 200 disclosed in the present invention, the outline or boundary of the target object can be automatically identified. Therefore, the training requirements and experience of the labeling personnel can be greatly reduced, so that almost any inexperienced user can use the UI-based method in this technical solution to achieve rapid data labeling for different visual application scenarios and complete AI model training. Moreover, the use of the semi-automatic labeling method of method 200 only requires a small amount of manually labeled data to train an AI model with good results. It greatly reduces the workload of the labeling personnel and also greatly saves labor costs and AI model training time.

[0037] Figure 3A flowchart of a method 300 for annotating an image and generating an AI model according to some embodiments of the present disclosure is shown. At block 310, user annotation data for a target image is obtained. Optionally, obtaining the user annotation data for the target image may include obtaining user input for the target image using an input / output device. Exemplarily, the input / output device may include various forms such as finger touch, a stylus, a keyboard, a mouse, a monitor, and the like, and the hardware platform may include a PC, a tablet computer, a smart mobile terminal, and the like. Exemplarily, if the input / output device includes a touch device, this greatly facilitates user image annotation. Applicants have found in long-term practice that using a touch device is highly intuitive when annotating the outline or boundary of an image. As the annotator performs touch annotation, what they see is what they get, saving time and effort and achieving twice the result with half the effort. Optionally, the annotation data may include at least one of category annotation data, outline annotation data, or boundary annotation data for objects in the image. Optionally, the category annotation data may be directly entered by the user, or a category type may be provided for the user to select. The contour annotation data or boundary annotation data can be drawn directly by the user, or can be annotated using a more convenient method further provided by the present disclosure. One or more embodiments of the relevant method will be described in detail later. Optionally, obtaining the user's annotation data for the target image can also include generating image annotation data based on the user's input. Optionally, after the user annotates the category, contour or boundary of the image through the input and output devices, the relevant image and the corresponding annotation information can be associated and stored to form the image annotation data.

[0038] At block 320 , an AI model is trained based on the target image and the annotated data. And at block 330 , a trained AI model is generated. Optionally, the AI ​​model may be an incremental learning AI model. For example, through extensive practice, it has been found that an incremental learning AI model is more suitable for the present disclosure. This allows for the use of a small number of manually annotated images. By combining automatic annotation with manual annotation using the incremental learning AI model, and by continuously and iteratively training the incremental learning AI model, an AI model product that meets the annotation requirements can ultimately be obtained.

[0039] It can be seen that by adopting the method 300 disclosed in the present invention, the outline or boundary of the target object can be automatically identified. Therefore, the training requirements and experience of the labeling personnel can be greatly reduced, so that almost any inexperienced user can use the UI-based method in this technical solution to achieve rapid data labeling for different visual application scenarios and complete AI model training. Moreover, the use of the semi-automatic labeling method of method 300 only requires a small amount of manually labeled data to train an incremental learning AI model with good results. It greatly reduces the workload of the labeling personnel, and also greatly saves labor costs and AI model training time, and realizes rapid labeling of images and rapid deployment of AI models.

[0040] Figure 4 A flowchart of a method 400 for obtaining user annotation data for a target image according to some embodiments of the present disclosure is shown. At box 410, the graphical features of at least one object in the target image are identified. Optionally, the graphical features include at least one of the edge, texture, or color of the object. Optionally, a preliminarily trained AI model can be used to automatically identify the graphical features of at least one object in the target image, which can greatly save the time and energy of the annotators. Optionally, in response to the situation where the preliminarily trained AI model is not very accurate in identifying the graphical features of the object, the annotator can be prompted to perform manual identification and manual annotation. Therefore, method 400 can use this method of close coordination between manual annotation and AI model recognition to perform graphical feature recognition of the object. This will greatly improve the efficiency of recognition and annotation compared to the method of simply having manual recognition and annotation performed, greatly reduce labor costs and reduce annotation time.

[0041] At block 420, at least one object is preliminarily distinguished based on the graphic features of the object. Optionally, the preliminarily trained AI model may be used to preliminarily distinguish at least one object based on the graphic features of the object. Optionally, preliminarily distinguishing at least one object may include dividing the target image into blocks based on the graphic features of the object to preliminarily distinguish each object. Figure 5 For example, the AI ​​model after initial training can automatically identify Figure 5 Optionally, based on the outline or boundary of the identified object 510, object 520, or object 530, the preliminarily trained AI model can divide the object 510, object 520, or object 530 into blocks based on the outline or boundary of the object 510, object 520, or object 530 to preliminarily distinguish the object 510, object 520, or object 530. For example, Figure 5The block cutting 540 based on the outline of the object 510 is schematically shown. Similarly, the same or similar block cutting can be performed on the object 520 or the object 530. To save space, this disclosure will not go into details. It should be noted that Figure 5 The preliminary distinction is only schematic and exemplary, and those skilled in the art can adopt appropriate cutting and dividing methods according to actual needs. Therefore, any preliminary distinction suitable for the present disclosure should be within the scope of protection of the present disclosure.

[0042] At block 430, user input regarding the object is obtained using an input / output device. Optionally, obtaining user input regarding the object using an input / output device includes obtaining user input regarding region differentiation of the target image's segmented regions obtained by segmentation. Optionally, the region differentiation input includes at least one of coloring the segmented regions or drawing the region outline of the object using a polygon tool. Optionally, the user colors the segmented regions using an input / output device such as a touch screen or mouse, and the colored regions become the labeled regions. After coloring the target region, a user can manually select or determine and input the category of the labeled object corresponding to the region. Compared to traditional labeling methods, which only track the object's edge details by counting touch point locations and frequencies, these methods are highly dependent on the labeler's touch control skills, making it difficult for inexperienced users to precisely label the object's edges. Method 400, by coloring the automatically segmented regions, significantly reduces the training and experience required of labelers, allowing virtually any inexperienced user to quickly label data for various visual application scenarios and complete deep learning model training using the UI-based method of method 400. Optionally, obtaining the user's input on the object using the input and output devices also includes obtaining the user's annotation input on the block area. Optionally, after completing the above-mentioned coloring of the target area, the user can manually select or judge and input the category of the annotated object corresponding to the area. In this way, the complete annotation of the target area is completed, that is, the annotation includes the annotation data of the category, outline or boundary of the object. Optionally, finally at box 440, based on the user's input, the annotation data is generated. In this way, the annotation is completed. Figure 5 's annotation.

[0043] Thus, by using method 400 to color-code the automatically divided areas, the training requirements and experience required of the annotators can be greatly reduced, allowing almost any inexperienced user to use the UI-based method of method 400 to quickly annotate data for different visual application scenarios. This also completes the training of the AI ​​model at a relatively low cost, resulting in an AI model that can be applied to actual automatic annotation. Furthermore, through iterative training according to one or more embodiments of the present disclosure, the incremental learning AI model of the present disclosure will become even more accurate.

[0044] Figure 6 A flowchart of a method 600 for obtaining user annotation data for a target image according to some other embodiments of the present disclosure is shown. At box 610, the user's input of a target object in an object in a target image is obtained using an input / output device. It should be noted that the target image may also include images that are not accurately recognized by the AI ​​model and require manual annotation. Optionally, the input / output device includes at least a bounding box tool, and the user can use the bounding box tool to determine the boundaries of the target object. As an example, Figure 7 shows a schematic image 700 to be annotated, Figure 7 There is object 710, object 720 or object 730. Optionally, the user can use the bounding box tool provided by the system to determine the boundary of the target object. For example, Figure 7 The object 730 in the image 700 is shown with a marked boundary 740. It is worth noting that Figure 7 The boundary marking is only schematic and exemplary. Those skilled in the art can adopt appropriate boundary marking methods according to actual needs. Any boundary marking method suitable for the present disclosure should be within the scope of protection of the present disclosure.

[0045] At box 620, the user's input for selecting the category of the target object is obtained. Optionally, in an object detection application scenario, its main feature is to use an AI model to find the target object in the image and identify these target objects using a bounding box. After the user determines the boundary of the target object in the image, method 600 can provide candidate object categories for the user to select. The applicant's long-term practice has shown that the use of this category selection input method is more conducive to the unification and standardization of the annotation data. For example, for Figure 7 If the object 730 in the image is freely input, different annotators may have different inputs, which is very detrimental to the unification and standardization of the annotation. However, method 600 adopts a method of providing object category selection, which unifies the input of different annotators and has a very good technical effect. In this way, the user selects the input object category based on the candidate object categories provided by method 600. Optionally, at box 630, annotation data is generated based on the user's input. Optionally, the annotation data includes boundary information and category information of the target object in the image. In this way, the annotation work of the target image is completed.

[0046] Figure 8A flowchart of a method 800 for annotating an object segmentation model according to some embodiments of the present disclosure is shown. Optionally, method 800 can be used for data annotation in object segmentation applications. Object segmentation applications primarily utilize AI models, such as deep learning, to segment and distinguish target objects from other objects in an image along their edges. Therefore, when annotating training data, it is necessary to follow the outline of the object as closely as possible. To assist in manual annotation tasks that require precise contour drawing, method 800 prompts the annotator to upload several images to be annotated to the platform at 802 or 808. After the initial or subsequent image uploads are complete, method 800, at 810, segments each image by identifying graphical features such as edges, texture, and color of different objects in the image. At 804, the user colors the annotated area using touch or mouse control, and the colored area becomes the annotated area. After coloring the target area, the user can manually enter or select the corresponding annotation category for that area. Optionally, in addition to the coloring marking method, the user can also choose to use the polygon tool to draw the outline of the target area by touching or clicking the mouse to complete the marking.

[0047] After completing the coloring and annotation of the uploaded image, the user will use the training function provided by method 800 to train the corresponding AI model at 806 using the newly annotated image. After training is complete, method 800 will prompt the user to upload a new image to be annotated at 808. After the user completes the upload, method 800 will use the trained AI model to identify the target annotated object in the image at 810 and use the AI ​​model to draw its outline. After the drawing is completed, method 800 will hand over the drawn image to the user for manual review at 812. During the review, the user will confirm whether the annotated area drawn by the AI ​​model is qualified. If it meets the annotation requirements, there is no need to manually annotate the image. Otherwise, the user can use the segmentation method described above to segment the image again at 804 and annotate the target object by coloring or polygon tracing. After the annotation is complete, method 800 can use the annotated image to further train the corresponding AI model. In this way, method 800 iteratively trains the AI ​​model, ultimately obtaining an AI model that can accurately annotate images.

[0048] It can be seen that by adopting method 800, by coloring and annotating the automatically divided areas, the training requirements and experience of the annotators can be greatly reduced, so that almost any inexperienced user can use the UI-based method of method 800 to achieve rapid data annotation for object segmentation application scenarios, and only a small amount of manually annotated images and the training of the AI ​​model are completed at a relatively low cost, thereby obtaining an AI model that can be applied to actual automatic annotation. Moreover, through iterative training of one or more embodiments of the present disclosure, the AI ​​model of the present disclosure will become more accurate. In addition, the trained AI model can be exported and deployed on a certified deployment platform.

[0049] Figure 9 A flowchart of a method 900 for annotating an object detection model according to some embodiments of the present disclosure is shown. Optionally, method 900 can be used in object detection applications. Object detection applications primarily utilize an AI model to locate target objects in an image and identify these objects using bounding boxes. Similar to method 800, the user first uploads an image to be annotated at 902. The user then uses the bounding box tool, using touch or mouse dragging, to enclose the target object with a bounding box and select a category. After the user completes image annotation, method 900 trains the AI ​​model for the corresponding application using the newly annotated image at 904. After training is complete, the user can upload a new image to be annotated. At 906, the trained AI model is used to identify the objects to be annotated in the newly uploaded image and draw bounding boxes. After recognition and drawing are complete, method 900 prompts the user to review the recognition and drawing results at 908. At 910, if the annotation requirements are met, manual annotation of the image is not required. Otherwise, the user can manually annotate the target image using touch or mouse dragging. After the annotation is completed, the user can use the annotated image to further train the corresponding application model. In this way, the method 900 iterates and trains the AI ​​model through such a loop, and finally obtains an AI model that can accurately annotate images.

[0050] It can be seen that by adopting method 900, through the automatic identification and demarcation of the target object, the training requirements and experience of the labeling personnel can be greatly reduced, so that almost any inexperienced user can use the UI-based method of method 900 to achieve rapid data labeling for object detection application scenarios, and only a small amount of manually labeled images and the training of the AI ​​model are completed at a relatively low cost, thereby obtaining an AI model that can be applied to actual automatic labeling. Moreover, through iterative training of one or more embodiments of the present disclosure, the AI ​​model of the present disclosure will become more accurate. In addition, the trained AI model can be exported and deployed on a certified deployment platform.

[0051] Figure 10 The flowchart of the method 1000 for labeling image recognition models according to some embodiments of the present disclosure is shown. Optionally, the method 1000 can be used in image recognition application scenarios. The main feature of image recognition applications is to use AI models to interpret the entire image and give the interpretation type results. For example, for Figure 1 Perform overall recognition. In method 1000, the user first uploads an unlabeled image, and can use touch or keyboard and mouse to label the image category. Then method 1000 can use the labeled image to train the AI ​​model for the corresponding application at 1010. After the training is completed, the user can upload a new image to be labeled at 1020, and then use the trained AI model to perform model recognition and draw a bounding box on the newly uploaded image at 1030. After the recognition is completed, method 1000 prompts the user to review the recognition result at 1040. Optionally, if it meets the labeling requirements, there is no need to manually label the image. On the contrary, the user can use touch or keyboard and mouse clicks to manually label the target object. After the labeling is completed, the user can use the labeled image again at 1010 to further train the corresponding application model. In this way, the method 1010 trains the AI ​​model through such a loop iteration, and finally obtains an AI model that can accurately label images.

[0052] It can be seen that by adopting method 1010, through the automatic identification of the target object and the drawing of the bounding box, the training requirements and experience of the annotation personnel can be greatly reduced, so that almost any inexperienced user can use the UI-based method of method 1010 to achieve rapid data annotation for image recognition application scenarios, and only a small amount of manually annotated images and the training of the AI ​​model are completed at a relatively low cost, thereby obtaining an AI model that can be applied to actual automatic annotation. Moreover, through the iterative training of one or more embodiments of the present disclosure, the AI ​​model of the present disclosure will become more accurate. In addition, the trained AI model can be exported and deployed on a certified deployment platform.

[0053] In summary, through one or more embodiments of the present disclosure described above, for object segmentation applications, by using touch or keyboard and mouse operations, after identifying the graphical features of objects in the image, such as texture, color, edges, etc., the image is divided into small blocks, and the user can color and mark the target object in the image by touching or dragging the mouse and select the marking category. After completion, the colored area is the marked area. For object detection applications, users also use touch or mouse drag operations to draw bounding boxes. For image recognition applications, users use touch or mouse clicks to mark image categories.

[0054] Optionally, one or more embodiments of the present disclosure also adopt a semi-automatic annotation form, guiding the user to upload and pre-annotate several target application images through the UI. The system then uses the uploaded images to train the corresponding application model. After the training is completed, the system guides the user to continue uploading new images, and then the trained model interprets the new images and annotates the target types listed by the user. For object segmentation applications, the model also draws the segmentation results. Similarly, for object detection applications, the model draws the bounding box. The system will then guide the user to review the model's depiction and annotation results, and guide the user to accept or re-annotate the image according to the annotation requirements.

[0055] By adopting one or more of the above embodiments of the present disclosure, it is possible to greatly reduce the user's need to label image data sets for AI model training in different application scenarios (such as image recognition, object detection, and object segmentation application scenarios), as well as the threshold for training and using these application models. The adoption of one or more of the above embodiments of the present disclosure can enable users who have no experience in AI and data labeling to rely on the UI and system prompts to quickly implement all code-free operations from training image data labeling to model training, and build their own AI applications. The above one or more embodiments of the present disclosure can greatly promote the application scope of AI in the field of images, reduce system development costs, and greatly reduce the technical threshold and deployment cycle of AI applications in the field of images, and have broad technical prospects and huge industrial value.

[0056] One or more embodiments of the present disclosure also include a system for annotating images. The system includes a processor and a memory coupled to the processor. The memory has instructions stored therein, which cause the processor to execute a method for annotating images. The method includes determining a recognition result of an image based on a trained AI model, wherein the AI ​​model is obtained by training based on annotated images. The method also includes obtaining user feedback on the recognition result, wherein the feedback is used to indicate the recognition accuracy of the recognition result. And in response to the feedback indicating that the recognition accuracy of the recognition result meets a predetermined condition, the method uses the recognition result determined by the AI ​​model as the annotation data of the image. Optionally, the method also uses the user's re-annotation of the image as the annotation data of the image in response to the feedback indicating that the recognition accuracy of the recognition result does not meet the predetermined condition.

[0057] In one or more embodiments of the present disclosure, the method further obtains the user's annotation data of the target image, and trains the AI ​​model based on the target image and the annotation data, and generates a trained AI model.

[0058] In one or more embodiments of the present disclosure, obtaining the user's annotation data for the target image includes obtaining the user's input to the target image using an input / output device, and generating the annotation data based on the input.

[0059] In one or more embodiments of the present disclosure, obtaining the user's annotation data for the target image includes identifying the graphical features of at least one object in the target image, and preliminarily distinguishing at least one object based on the graphical features of the object, and obtaining the user's input to the object using an input / output device, and generating annotation data based on the input.

[0060] In one or more embodiments of the present disclosure, the graphical features include at least one of the edge, texture, or color of an object, and the AI ​​model includes at least one of an AI model that recognizes objects in an image one by one, an AI model that recognizes a target object from objects in an image, or an AI model that recognizes objects in an image as a whole, and the annotation data includes at least one of category annotation data, contour annotation data, or boundary annotation data of the objects in the image, and the recognition result includes at least one of the category, contour, or boundary of the objects in the image. In addition, the input and output device includes a touch device, the target image includes an image manually annotated by a user, and the AI ​​model includes an incremental learning AI model.

[0061] In one or more embodiments of the present disclosure, preliminarily distinguishing at least one object includes segmenting a target image into blocks based on the object's graphical features to preliminarily distinguish the objects, and obtaining user input regarding the object using an input / output device includes obtaining user input regarding region differentiation of segmented regions obtained by segmenting the target image, and obtaining user input regarding annotations of the segmented regions. Furthermore, the region differentiation input includes at least one of coloring the segmented regions or drawing the region outline of the object using a polygon tool.

[0062] In one or more embodiments of the present disclosure, obtaining user annotation data for a target image includes obtaining user input regarding a target object in the target image using an input / output device, obtaining user input selecting a category for the target object, and generating annotation data based on the input. The input / output device includes at least a bounding box tool, and the user can use the bounding box tool to determine the boundaries of the target object.

[0063] Figure 11A block diagram of an electronic device 1100 in which multiple embodiments of the present disclosure can be implemented is shown. As shown, the device 1100 includes a central processing unit (CPU) 1101, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 1102 or loaded from a storage unit 1108 into a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the device 1100 can also be stored. The CPU 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0064] Various components in device 1100 are connected to an I / O interface 1105, including an input unit 1106, such as a keyboard and mouse; an output unit 1107, such as various types of displays and speakers; a memory 1108, such as a magnetic disk and optical disk; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1109 allows device 1100 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0065] The various processes and processing described above, such as methods 200 and 300, etc., may be executed by the processing unit 1101. For example, in some embodiments, methods 200 and 300, etc. may be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded into the RAM 1103 and executed by the CPU 1101, one or more actions of the methods 200 and 300, etc. described above may be performed.

[0066] The present disclosure may be a method, an apparatus, a device (system) and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.

[0067] Computer-readable storage media can be a tangible device that can hold and store instructions used by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. The computer-readable storage media used herein is not to be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by a fiber optic cable), or an electrical signal transmitted by a wire.

[0068] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0069] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions. The electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0070] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0071] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0072] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0073] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0074] While various embodiments of the present disclosure have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for annotating an image, comprising: Determining a recognition result of the image based on a trained artificial intelligence (AI) model, wherein the AI ​​model is trained based on the annotated image; Obtaining user feedback on the recognition result, wherein the feedback is used to indicate the recognition accuracy of the recognition result; In response to the feedback indicating that the recognition accuracy of the recognition result meets a predetermined condition, using the recognition result determined by the AI ​​model as the annotation data of the image; or In response to the feedback indicating that the recognition accuracy of the recognition result does not meet a predetermined condition, the re-annotation of the image by the user is used as the annotation data of the image.

2. The method according to claim 1, further comprising: Acquire the user's annotation data for the target image; Training the AI ​​model based on the target image and the annotated data; as well as Generate the trained AI model.

3. The method according to claim 2, wherein obtaining the user's annotation data for the target image comprises: Acquiring input of the target image by the user using an input / output device; as well as Based on the input, the annotation data is generated.

4. The method according to claim 2, wherein obtaining the user's annotation data for the target image comprises: Identifying graphical features of at least one object in the target image; Preliminarily distinguishing at least one of the objects based on the graphical features of the objects; Obtaining the user's input to the object using an input / output device; as well as Based on the input, the annotation data is generated. The method according to claim 4 , wherein the graphical feature comprises at least one of an edge, a texture or a color of the object.

6. The method according to claim 4, wherein preliminarily distinguishing at least one of the objects comprises: Based on the graphical features of the objects, the target image is cut into blocks to preliminarily distinguish the objects.

7. The method according to claim 6, wherein obtaining the user's input to the object using an input / output device comprises: Obtaining a region distinction input by the user for the block regions obtained by block cutting of the target image; as well as Obtain the user's annotation input for the block area. 8 . The method according to claim 7 , wherein the region differentiation input comprises at least one of coloring the segmented region or drawing the region outline of the object using a polygon tool.

9. The method according to claim 2, wherein obtaining the user's annotation data for the target image comprises: Obtaining input of the user on a target object in the object in the target image using an input / output device; Obtaining the user's input of selecting a category of the target object; as well as Based on the input, the annotation data is generated.

10. The method of claim 9, wherein the input-output device comprises at least a bounding box tool, and the user can use the bounding box tool to determine the boundaries of the target object.

11. The method according to claim 1, wherein the AI ​​model comprises at least one of the following AI models: An AI model for identifying objects in the image one by one; an AI model that identifies a target object from among the objects in the image; or An AI model for overall recognition of the object in the image. 12 . The method according to claim 1 , wherein the annotation data comprises at least one of category annotation data, contour annotation data, or boundary annotation data of the object in the image.

13. The method according to claim 1, wherein the recognition result comprises at least one of a category, a contour, or a boundary of the object in the image. The method according to claim 3 , wherein the input-output device comprises a touch-sensitive device.

15. The method of claim 2, wherein the target image comprises the image, and the target image and the image comprise a medical image.

16. The method of claim 1, wherein the AI ​​model comprises an incremental learning AI model.

17. A system for image annotation, comprising: processor; as well as a memory coupled to the processor, the memory having instructions stored therein, the instructions, when executed by the processor, causing the device to perform actions, the actions comprising: Determining a recognition result of the image based on a trained AI model, wherein the AI ​​model is trained based on the annotated image; Obtaining user feedback on the recognition result, wherein the feedback is used to indicate the recognition accuracy of the recognition result; In response to the feedback indicating that the recognition accuracy of the recognition result meets a predetermined condition, using the recognition result determined by the AI ​​model as the annotation data of the image; or In response to the feedback indicating that the recognition accuracy of the recognition result does not meet a predetermined condition, the re-annotation of the image by the user is used as the annotation data of the image.

18. The system of claim 17, wherein the actions further comprise: Acquire the user's annotation data for the target image; Training the AI ​​model based on the target image and the annotated data; as well as Generate the trained AI model.

19. The system of claim 17, wherein the instructions, when executed by a processor, cause the device to further perform the method of any one of claims 3-16.

20. A computer program product tangibly stored on a computer readable medium and comprising machine executable instructions which, when executed, cause a machine to perform the method according to any one of claims 1-16.

Images