Image annotation system and method

Abstract

To provide an image annotation system that enables annotating a microscopic image fast and easily.SOLUTION: An image annotation system (100) comprises: an output unit (104) that displays an image of a sample; and an input unit (108, 110) that captures a first user input sequence including a first movement of a cursor (112) over a first selection area of the image and at least one second user input sequence including a second movement of the cursor over a second selection area of the image. A processor (114) determines a first image area (122a) on the basis of the first selection area, redetermines the first image area on the basis of the first selection area and the second selection area when the second movement begins within the first selection area, determines a second image area (122b) on the basis of the second selection area when the second movement begins outside the first selection area, and determines an image area of the image of the sample.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image annotation system for attaching annotations to microscope images of samples. Furthermore, the present invention relates to a method for determining at least one image region in a microscope image of a sample based on user input.

Background Art

[0002] Machine learning algorithms are used to process large amounts of images collected by microscopes. When these machine learning algorithms are properly trained, they can autonomously classify, i.e., identify and label, objects in microscope images. For training, a set of annotated images called a training dataset is used. This training dataset provides examples of what the machine learning algorithm should reproduce. Depending on the task, the examples are composed of single pixels and / or the outlines of the entire objects that need to be classified by the machine learning algorithm. The outline of the entire object allows the machine learning algorithm to understand a larger context. For example, if the object to be classified is a circular object with only bright edges, it becomes difficult to correctly identify a dark signal in the center of the object. Furthermore, by drawing the outline of the entire object, the machine learning algorithm can understand concepts such as circles and inside / outside, thereby providing an opportunity to properly classify the entire object.

[0003] Examples of training datasets need to be provided by the user. To efficiently generate a training dataset, the user requires an image annotation system. For example, a visual interface can be used that allows the user to label individual pixels in a microscope image using an image processing tool. In particular, a visual interface that allows the user to draw labels on a microscope image.

Summary of the Invention

Problems to be Solved by the Invention

[0004] Therefore, the objective is to provide an image annotation system and method that allows users to quickly and easily annotate microscope images. [Means for solving the problem]

[0005] A proposed image annotation system for annotating a microscopic image of a sample comprises an output unit configured to display an image of the sample, and an input unit configured to capture a first user input sequence including a first movement of a cursor in a first selection region of the displayed image, and at least one second user input sequence including a second movement of a cursor in a second selection region of the displayed image. The image annotation system further comprises a processor. The processor is configured to determine at least one image region of the image of the sample by determining a first image region based on a first selection region, redetermining the first image region based on the first and second selection regions if the second movement begins within the first selection region, and determining a second image region based on the second selection region if the second movement begins outside the first selection region.

[0006] Microscope images are annotated by assigning a first image region and a second image region to different areas of the image. Each image region can correspond to a different object or area of ​​the sample. These different image regions are generated by cursor movement on the annotated microscope image. These gestures are similar to drawing with a brush on the microscope image, providing an intuitive way to annotate microscope images. Furthermore, if cursor movement begins within an already defined image region, the image region is expanded. This allows for quick modification of already generated image regions. Therefore, the image annotation system provides users with a fast and easy-to-use smart tool to support annotating microscope images, for example, to generate training datasets for machine learning algorithms.

[0007] In a preferred embodiment, the cursor has a cursor shape, the first selection region is the region of the displayed image that the cursor shape passes through during a first movement, and the second selection region is the region of the displayed image that the cursor shape passes through during a second movement. In particular, the cursor shape is an ellipse, a circle, a rectangle, or a square. Preferably, the cursor shape has dimensions that include multiple pixels of the microscope image. This allows the user to select multiple pixels at once, thereby speeding up processing. Preferably, the shape and / or size of the cursor can be selected by the user. By providing one or more cursor shapes, the user can select the appropriate tool depending on the details of the region of the microscope image that needs to be selected. This further increases the versatility of the image annotation system.

[0008] In another preferred embodiment, the processor is configured to determine whether a second movement begins within a first selection region by determining whether the centroid of the cursor shape is within the first selection region when the second movement begins. This provides an intuitive measure of whether the majority of the cursor shape is within the first selection region. This allows the user to easily expand the first selection region, thereby enabling quick correction of errors or further refinement of the shape of the first selection region.

[0009] In another preferred embodiment, if the second movement begins outside the first selection region, the first image region corresponds to the first selection region, and the second image region corresponds to the second selection region. If the second movement begins within the first selection region, the first image region corresponds to the set of the first and second selection regions. In this embodiment, the image region is identical to the selection region. This allows the user to select the image region with great precision. Alternatively or additionally, the processor may be configured to enlarge and / or reduce the image region for each selection region based, for example, on the image content of the microscope image.

[0010] In another preferred embodiment, the processor is configured to generate an object map consisting of image points, where each image point corresponds to an image point in the sample image. Each image point in the object map encodes information about which image region the corresponding image point in the sample image belongs to. The object map is separate from the microscope image and can be used to generate a training dataset for a machine learning algorithm.

[0011] In another preferred embodiment, the processor is configured to determine the start of a first and second movement based on a button press and the end of the first and second movement based on a button release. Using button press / release as a method for determining the start and end of the first and second movement is highly intuitive. This allows the user to select the first and second image regions very quickly, resulting in high throughput.

[0012] In another preferred embodiment, the input unit is a computer mouse. In this embodiment, the processor is configured to determine the first and second movements of the cursor based on the movement of the computer mouse. Alternatively, other input units such as a joystick or trackball may be used. In particular, the image annotation system is set up to accommodate many different input devices. This allows the user to select the most comfortable input device, ensuring intuitive and rapid use of the image annotation system.

[0013] In another preferred embodiment, the image annotation system includes a touchscreen that forms an output unit and / or an input unit. Preferably, the processor is configured to determine the start of a first and second movement based on whether an object is touching the touchscreen, and to determine the end of the first and second movement based on whether the object is no longer touching the touchscreen. The processor is configured to determine the first and second movements of a cursor based on the movement of an object on the touchscreen. The object may be the user's finger, a pen, or a stylus. The use of a touchscreen in combination with a pen or stylus is particularly intuitive as it closely resembles the use of a pen and paper. This allows the user to quickly annotate microscope images.

[0014] In another preferred embodiment, the image annotation system comprises a virtual reality and / or augmented reality interface configured to provide input and / or output units. The virtual and augmented reality provides an intuitive way to interact with the image annotation system.

[0015] In another preferred embodiment, the output unit is configured to display a first image region and a second image region. This allows the image annotation system to provide the user with feedback regarding a first user input sequence and a second user input sequence that define the first image region and the second image region, respectively. This further enhances usability.

[0016] The present invention also relates to a microscope system comprising an optical detection system configured to capture an image of a sample, and the image annotation system described above.

[0017] The present invention also relates to a method for determining at least one image region in a microscope image of a sample based on user input, the method comprising: displaying an image of the sample; capturing a first user input sequence including a first movement of a cursor in a first selection region of the displayed image of the sample; determining a first image region based on the first user input sequence; capturing a second user input sequence including a second movement of a cursor in a second selection region of the displayed image of the sample; if the second movement begins within the first selection region, redetermining the first image region based on the first and second user input sequences; and if the second movement begins outside the first selection region, determining a second image region based on a second user input sequence.

[0018] The method has the same advantages as the image annotation system described above. In particular, the method can be supplemented by using the features of each dependent claim directed toward the image annotation system.

[0019] The present invention further relates to a computer program, which includes program code for performing the methods described above when executed on a processor. The processor may comprise at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (Field-Programmable Gate Array), an ASIC (Application-Specific Integrated Circuit), and a DSP (Digital Signal Processor).

[0020] The above has primarily described image annotation systems in relation to machine learning. However, the applications of image annotation systems are not limited to generating image data that can be used as training image data for machine learning algorithms. For example, image annotation systems can be used to add comments or highlights to specific features of microscopic images for the observer.

[0021] The following describes specific embodiments with reference to the drawings.

Brief Description of the Drawings

[0022] [Figure 1] It is a schematic diagram of an image annotation system for attaching annotations to a microscopic image of a sample. [Figure 2] It is a flowchart of a method for determining at least one image region in a microscopic image of a sample based on user input. [Figure 3] It is a schematic diagram of a first image region and a second image region. [[ID=X]] [Figure 4] It is a schematic diagram of a first region.

Embodiments of the Invention

[0023] FIG. 1 is a schematic diagram of an image annotation system 100 for attaching annotations to a microscopic image of a sample 102.

[0024] In the present embodiment, the image annotation system 100 includes an output unit 104 exemplarily formed as a computer monitor having a display area 106, and two input units exemplarily formed as a computer mouse 108 and a keyboard 110, respectively. The computer mouse 108 is configured to receive user input and move a cursor 112 in the display area 106 of the output unit 104. The keyboard 110 is also configured to receive user input, particularly in the form of button presses / releases. The image annotation system 100 further includes a processor 114 connected to the output unit ******** and input units 108, 110. The processor 114 is further connected to a microscope 116 having an optical detection system 118 configured to capture a microscopic image of the sample 102. The image annotation system 100 and the microscope 116 may form a microscope system 120.

[0025] It should be noted that there is an incomplete expression in the original text where "output unit 104 and input units 108, 110" is followed by "********", which is likely an error or an incomplete part in the original. This translation is made based on the available text as accurately as possible.Processor 114 is configured to receive image data corresponding to the microscopic image of sample 102 from microscope 116. Processor 114 may receive the image data, for example, via a direct connection, a network connection, or a data medium. Processor 114 is configured to control output unit 104 to display the microscopic image of sample 102 in display area 106. Processor 114 is also configured to receive control data corresponding to user input to two input units 108, 110. Further, processor 114 is configured to perform a method for determining at least one image region 122a, 122b in the microscopic image of sample 102 based on user input. The method will be described below with reference to FIGS. 2 to 4.

[0026] FIG. 2 is a flowchart of a method for determining at least one image region 122a, 122b in the microscopic image of sample 102 based on user input.

[0027] In step S200, processing begins. In the optional step S202, the user determines the zoom level of the microscope image displayed in the display area 106 of the output unit 104 to determine the level of detail that can be viewed. In step S202, the user may select a cursor shape, such as a pointer, a circle, or any other suitable shape. In step S204, a button is pressed. This may be a button on a computer mouse 108 or a keyboard 110. In other embodiments, the button press may correspond to an object such as a stylus touching the surface of a touchscreen. The button press determines the start of a first movement. In step S206, the cursor 112 moves in a first selected area of ​​the microscope image displayed in the display area 106 of the output unit 104 in the first movement. The first movement may be prompted, for example, by the movement of a computer mouse 108 or the movement of a stylus on a touchscreen. In step S208, the button is released, and the end of the first movement is determined. The first input sequence consists of pressing a button, the first movement of the cursor 112, and releasing the button. Next, in step S210, the processor 114 determines a first image region 122a based on a first selection region. Preferably, the first image region 122a is identical to the first selection region. Alternatively, the processor 114 may determine the first image region 122a based on the image content of the microscope image. For example, the processor 114 may ignore pixels in the microscope image that have a brightness value below a certain threshold.

[0028] In step S214, the button is pressed again. At this time, the button press determines the start of the second movement. In step S216, the processor 114 determines whether the cursor 112 is within the first image region 122a when the second movement begins. For example, if the cursor shape is a pointer, the processor 114 determines whether the tip of the pointer is within the first image region 122a when the second movement begins. If the cursor shape is an extended shape, such as a circle, the processor 114 may determine whether the centroid of the cursor shape is within the first image region 122a when the second movement begins. In step S218, the cursor 112 moves in the second selected region of the microscope image displayed in the display region 106 of the output unit 104 during the second movement. In step S220, the button is released, and the end of the second movement is determined. The button press, the second movement of the cursor 112, and the release of the button constitute the second input sequence. If a second movement is initiated within the image selection region 122a, the processor 114 re-determines the first image region 122a in step S222 based on the first and second selection regions. Preferably, the re-determined first image region 122a is identical to the set of the first and second selection regions. If a second movement is initiated outside the first selection region, the processor 114 determines the second image region 122b in step S224 based on the second selection region. Preferably, the second image region 122b is identical to the second selection region. Steps S214 to S224 will be described in more detail below with reference to Figures 3 and 4.

[0029] Steps S212 to S224 can be repeated to determine additional image regions 122a and 122b. If steps S212 to S224 are repeated, in step S216, the processor 114 determines whether the cursor 112 is inside either of the already determined image regions 122a or 122b when the second movement begins. Next, in step S222, the processor 114 re-determines the image regions 122a and 122b from which the second movement began. Optionally, in step 214, if another button is pressed, or yet another button is pressed, the eraser mode is activated until the button is released and the second movement ends. In eraser mode, after the button is released, the processor 114 removes the second selected region from all previously defined image regions 122a and 122b. This allows for quick correction of errors.

[0030] In the optional step S226, the processor 114 generates an object map. The object map consists of image points, each corresponding to an image point in the sample 102 image. Each image point in the object map encodes information indicating which image regions 122a,122b the corresponding image point in the sample 102 image belongs to. The object map can be used to generate a training dataset for a machine learning algorithm. The process then terminates in step S228.

[0031] Figure 3 is a schematic diagram of the first image region and the second image regions 122a and 122b.

[0032] In Figure 3, the second movement of the cursor 112 is shown by the dashed line 300. In Figure 3, the starting point of the second movement is shown by the arrow P1. As can be seen in Figure 3, the second movement starts outside the first image region 122a. Therefore, the processor 114 determines the second image region 122b to be the region where the cursor shape was moved during the second movement, i.e., the second selection region.

[0033] Figure 4 is a schematic diagram of the first image region 122a.

[0034] In Figure 4, the second movement of the cursor 112 is shown by the dashed line 400. In Figure 4, the starting point of the second movement is shown by the arrow P2. As can be seen in Figure 4, the second movement starts within the first image region 122a. Therefore, the processor 114 redefines the first image region 122a to include the region 402 to which the cursor shape was moved during the second movement, i.e., the second selection region.

[0035] Elements that function identically or similarly are designated with the same reference numeral in all figures. As used herein, the term "and / or" includes all possible combinations of one or more of the related items and may be abbreviated as " / ".

[0036] Both the individual features of the embodiment and all combinations of features are deemed to be disclosed. Furthermore, the individual features of the embodiment are deemed to be disclosed in combination with the individual features or sets of features described herein and / or in combination with the individual features or sets of features of the claims.

[0037] While several embodiments have been described in the context of the apparatus, it is clear that these embodiments also represent descriptions of the corresponding methods, where blocks or apparatus correspond to steps or features of steps. Similarly, embodiments described in the context of steps also represent descriptions of the corresponding blocks, items, or features of the corresponding apparatus. [Explanation of symbols]

[0038] 100 Image Annotation Systems 102 samples 104 Output Unit 106 Display area 108,110 input units 112 Cursor 114 processors 116 Microscope 118 Optical detection system 120 Microscope Systems 122a,122b Image area 300,400 lines 402 area P1, P2 arrows

Claims

1. An image annotation system (100) for annotating a microscopic image of a sample (102), said image annotation system (100) comprising: an output unit (104) configured to display an image of said sample (102); an input unit (108, 110) configured to capture a first user input sequence including a first movement of a cursor (112) in a first selected area of ​​the displayed image and at least one second user input sequence including a second movement of the cursor (112) in a second selected area of ​​the displayed image; a processor (114); Equipped with The processor (114) determining a first image region (122a) based on the first selected region; if the second motion begins within the first selected area, redetermine the first image area (122a) based on the first selected area and the second selected area; If the second movement begins outside the first selected area, determine a second image area (122b) based on the second selected area. and determining at least one image region (122a, 122b) of the image of the sample (102) by An image annotation system (100).

2. The cursor (112) has a cursor shape, the first selection area is an area of ​​the displayed image through which the cursor shape passes during the first movement; the second selection area is an area of ​​the displayed image through which the cursor shape passes during the second movement; The image annotation system (100) of claim 1.

3. the processor (114) is configured to determine whether the second movement begins within the first selection area by determining whether a center of gravity of the cursor shape is within the first selection area when the second movement begins. The image annotation system (100) of claim 2.

4. The cursor shape is an ellipse, a circle, a rectangle, or a square. The image annotation system (100) of claim 2.

5. if the second movement begins outside the first selected area, the first image area (122a) corresponds to the first selected area and the second image area (122b) corresponds to the second selected area; if the second movement begins within the first selected region, the first image region (122a) corresponds to the union of the first selected region and the second selected region; The image annotation system (100) of claim 1.

6. the processor (114) is configured to generate an object map of image points, each image point corresponding to an image point of the image of the sample (102), and each image point of the object map encoding information about to which image region (122a, 122b) the corresponding image point of the image of the sample (102) belongs; The image annotation system (100) of claim 1.

7. The processor (114) is configured to determine the beginning of the first movement and the second movement based on a button press and to determine the end of the first movement and the second movement based on a button release. The image annotation system (100) of claim 1.

8. The input unit (108, 110) is a computer mouse (108); the processor (114) is configured to determine the first movement and the second movement of the cursor (112) based on a movement of the computer mouse (108); The image annotation system (100) of claim 1.

9. the image annotation system (100) comprises a touch screen forming the output unit (104) and / or the input unit (108, 110); The image annotation system (100) of claim 1.

10. the processor (114) is configured to determine a beginning of the first movement and the second movement upon determining that the object is touching the touchscreen, and to determine an end of the first movement and the second movement upon determining that the object is no longer touching the touchscreen; the processor (114) is configured to determine the first movement and the second movement of the cursor (112) based on a movement of the object on the touch screen. The image annotation system (100) of claim 9.

11. the output unit (104) is configured to display the first image area and the second image area (122a, 122b); The image annotation system (100) of claim 1.

12. A microscope system (120), comprising: an optical detection system (118) configured to capture an image of the sample (102); An image annotation system (100) according to any one of claims 1 to 11, A microscope system (120) comprising:

13. 1. A method for determining at least one image region (122a, 122b) in a microscopic image of a sample (102) based on user input, the method comprising: displaying an image of the sample (102); capturing a first user input sequence including a first movement of a cursor (112) in a first selected area of ​​the displayed image of the sample (102); determining a first image region (122a) based on the first user input sequence; capturing a second user input sequence including a second movement of the cursor (112) in a second selected area of ​​the displayed image of the sample (102); if the second movement begins within the first selected area, redetermining the first image area (122 a) based on the first user input sequence and the second user input sequence; if the second movement begins outside the first selected region, determining a second image region (122b) based on the second user input sequence; A method comprising:

14. A computer program comprising a program code for performing the method of claim 13 when the computer program is run on a processor (114).

Images