Image annotation system and method

Abstract

To provide a method that enables annotating a microscopic image in a fast and easy way.SOLUTION: An image annotation system comprises an input unit that captures a first and second user input sequences including a first and second movements of a cursor over a displayed image and defining a first and second closed paths. A processor determines a first image area of an image of a sample corresponding to an area of the displayed image enclosed by the first closed path, determines whether or not a second path includes a proximity section where the second path is close to a border of the first image area on the basis of a proximity condition, determines a second image area of the image of the sample corresponding to an area of the displayed image enclosed by the second closed path when the second path does not include the proximity section, and determines the second image area corresponding to the area of the displayed image enclosed by the second closed path and corresponding to the area of the displayed image between the border of the first image area and the proximity section when the second path includes the proximity section.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image annotation system for annotating microscope images of samples. The present invention further relates to a method for identifying at least one image region in a microscope image of a sample based on user input.

Background Art

[0002] Machine learning algorithms are used for the purpose of processing a large number of images collected by microscopes. If properly trained, these machine learning algorithms can autonomously classify objects in microscope images, that is, identify and label them. For training, a set of annotated images called a training dataset is used. This training dataset provides an example of what the machine learning algorithm should reproduce. Depending on the task, these examples contain single pixels and / or contours of one complete object that must be classified by the machine learning algorithm. The contour of the complete object enables the machine learning algorithm to better understand a broader context. For example, if the object to be classified is a circular object with only high-brightness edges, it is difficult to accurately identify the faint signal in the center of the object. Furthermore, by drawing the contour of the complete object, the machine learning algorithm can understand the concepts of circle and inside / outside, and thus has a chance to properly classify the entire object.

[0003] The examples in the training dataset must be provided by the user. For the purpose of efficiently generating a training dataset, the user needs an image annotation system. For example, a visual interface can be used, which allows the user to label individual pixels in a microscope image using an image processing tool. In particular, it is a visual interface that enables the user to draw labels on the top surface of the microscope image. [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] Therefore, the challenge is to provide an image annotation system and a method that allows users to quickly and easily annotate microscope images. [Means for solving the problem]

[0005] A proposed image annotation system for annotating a microscope image of a sample includes an output unit configured to display an image of the sample, and an input unit configured to capture a first user input sequence including a first movement of a cursor on the displayed image that defines a first closed path, and at least one second user input sequence including a second movement of a cursor on the displayed image that defines a second closed path. The image annotation system further includes a processor. This processor is configured to identify a first image region of the sample image corresponding to the region of the display image enclosed by the first closed path, determine whether the second path includes at least one proximity interval located near the boundary of the first image region based on at least one proximity condition, identify a second image region of the sample image corresponding to the region of the display image enclosed by the second closed path if the second path does not include a proximity interval, and identify a second image region of the sample image corresponding to the region of the display image enclosed by the second closed path and corresponding to at least one region of the display image between the boundary of the first image region and at least one proximity interval if the second path does include a proximity interval.

[0006] Annotation is applied to a microscope image by assigning a first image region and a second image region to different regions of the microscope image, respectively. Each image region can correspond to a different object or region of the sample. The first image region is generated by defining a first closed path, that is, by moving a cursor on the display image around the region of the display image that the user wants to define as the first image region. Similarly, the second image region is defined by moving a cursor on the display image around the region that the user wants to define as the second image region, that is, by drawing a second closed path. However, if the second closed path includes a section located near the boundary of the first image region, which is defined as a proximity section in this specification, the image annotation system automatically adds the region of the microscope image located between the first image region and the proximity section of the second path to the second image region. This allows the user to define image regions that are adjacent to each other, for example, when defining image regions for the purpose of annotating different objects or regions of a sample that are in contact with each other, without having to be extremely precise. In other words, by allowing the user to roughly define a second image region and fill in the gaps, the image annotation system supports the user and therefore enables them to work more quickly and efficiently. This provides the user with a fast and easy-to-use smart tool for annotating microscope images, for example, for the purpose of generating training datasets for machine learning algorithms.

[0007] According to one preferred embodiment, the processor is configured to determine whether a second path contains at least one proximity interval by performing the following steps: 1. Determine the distance for each image point on the second closed path between an image point on the second closed path and an image point on the boundary of the first image region closest to the image point on the second closed path. 2. Compare each distance to a threshold. 3. Determine that the second path contains at least one proximity interval if at least a predetermined number of distances are less than the threshold. The processor is further configured to identify at least one proximity interval based on image points whose distances are less than the threshold. Preferably, the threshold is determined based on user input and / or the zoom level of the displayed image. In particular, the threshold is lower at higher zoom levels, i.e., when the displayed image is magnified. This allows the user to magnify the details of the displayed image for the purpose of more accurately defining the second image region. The processor can be configured to determine the threshold either alternatively or additionally based on the image content of a microscope image.

[0008] In another preferred embodiment, the processor is configured to re-identify the second image region in order to exclude areas of the display image where the first and second image regions overlap. In this embodiment, the user can draw on the first image region when drawing the second closed path. The image annotation system automatically adds only the portions of the display image that have not yet been defined as image regions to the second image region. This allows the user to work even more quickly.

[0009] In another preferred embodiment, the first and second user input sequences include button presses and releases, where the interval of the first closed path or the second closed path is defined by the cursor movement from button press to button release. Using button presses / releases as a method for identifying the start and end of the first and second movements is highly intuitive. This allows the user to select the first and second image regions very quickly, thereby enabling high throughput.

[0010] According to another preferred embodiment, the first and second user input sequences include intervals of at least two image points in the displayed image, where the interval of the first closed path or the second closed path is defined by a line between the two selected image points. In this embodiment, the user can define a straight line by selecting two image points. This further speeds up the annotation process, thereby allowing the user to work more efficiently.

[0011] According to another preferred embodiment, the processor is configured to generate an object map containing multiple image points, each image point corresponding to one image point of the sample image. Each image point in the object map encodes information about which image region the corresponding image point of the sample image belongs to. The object map is separate from the microscope image and can be used to generate a training dataset for a machine learning algorithm.

[0012] In another preferred embodiment, the input unit is a computer mouse. In this embodiment, the processor is configured to determine the first and second movements of the cursor based on the movement of the computer mouse. Alternatively, other input units such as a joystick or trackball may be used. In particular, the image annotation system is set up to use a number of different input devices. This allows the user to select the most comfortable input device while ensuring intuitive and rapid use of the image annotation system.

[0013] According to another preferred embodiment, the image annotation system includes a touchscreen that forms an output unit and / or input unit. Preferably, the processor is configured to identify the start of a first and second motion by determining that an object is touching the touchscreen, and to identify the end of the first and second motions by determining that the object is no longer touching the touchscreen. The processor is configured to identify the first and second motions of a cursor based on the movement of an object on the touchscreen. This object can be the user's finger, a pen, or a stylus. Using the touchscreen, especially in combination with a pen or stylus, is very intuitive as it closely resembles using a pen and paper. This allows the user to quickly annotate microscope images.

[0014] According to another preferred embodiment, the image annotation system includes a virtual reality interface and / or an augmented reality interface configured to provide an input unit and / or an output unit. The virtual reality and augmented reality provide an intuitive method of interaction with the image annotation system.

[0015] In another preferred embodiment, the processor is configured to determine, during input of the second input sequence, whether the second path includes at least one adjacent interval. In this embodiment, the image annotation system can provide the user with instantaneous feedback on whether or not the second path includes an adjacent interval. This allows the user to have better control over defining the second image region, thereby making the image annotation system easier to use.

[0016] In another preferred embodiment, the output unit is configured to display a first image region and a second image region. This allows the image annotation system to provide the user with feedback regarding a first user input sequence and a second user input sequence that define the first image region and the second image region, respectively. This further enhances ease of use.

[0017] The present invention also relates to a microscope system comprising an optical detection system configured to capture an image of a sample, and the image annotation system described above.

[0018] The present invention also relates to a method for identifying at least two image regions within a microscope image of a sample based on user input. This method may include the following steps: displaying an image of the sample; capturing a first user input sequence including a first movement of a cursor on the displayed image that defines a first closed path; capturing at least one second user input sequence including a second movement of a cursor on the displayed image that defines a second closed path; identifying a first image region of the sample image corresponding to a region of the displayed image enclosed by the first closed path; identifying a second image region of the sample image corresponding to a region of the displayed image enclosed by the second closed path, if the second path does not include a proximity interval; and identifying a second image region corresponding to a region of the displayed image enclosed by the second closed path and corresponding to at least one region of the displayed image between the boundary of the first image region and at least one proximity interval, if the second path does include a proximity interval.

[0019] This method has the same advantages as the image annotation system described above. In particular, this method can be complemented by the features of the dependent claims relating to the image annotation system.

[0020] This method further relates to a computer program having program code for performing the above-described method when the computer program is executed on a processor. This processor may include at least one of the following: CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (Field-Programmable Gate Array), ASIC (Application-Specific Integrated Circuit), and DSP (Digital Signal Processor).

[0021] The image annotation system has been mainly described so far in relation to machine learning. However, the use of this image annotation system is not limited to the generation of image data that can be used as training image data for use with machine learning algorithms. For example, this image annotation system may be used to provide comments on or highlight certain features of a microscopic image to a human observer.

[0022] Hereinafter, specific embodiments will be described with reference to the drawings.

Brief Description of the Drawings

[0023] [Figure 1] FIG. 1 is a schematic diagram of an image annotation system for annotating a microscopic image of a sample. [Figure 2] FIG. 2 is a flowchart of a method for identifying at least two image regions within a microscopic image of a sample based on user input. [Figure 3] FIG. 3 is a flowchart of an exemplary sub - process for identifying a closed path. [Figure 4] FIG. 4 is a schematic diagram of a display microscopic image, a first image region, and a closed path. [Figure 5] FIG. 5 is a schematic diagram of a display microscopic image and an image region.

Modes for Carrying Out the Invention

[0024] FIG. 1 is a schematic diagram of an image annotation system 100 for annotating a microscopic image of a sample 102.

[0025] According to this embodiment, the image annotation system 100 includes an output unit 104 exemplary formed as a computer monitor having a display area 106, and two input units exemplary formed as a computer mouse 108 and a keyboard 110, respectively. The computer mouse 108 is configured to receive user input and move a cursor 112 on the display area 106 of the output unit 104. The keyboard 110 is also configured to receive user input, particularly in the form of button presses / releases. The image annotation system 100 further includes a processor 114, which is connected to the output unit 104 and the input units 108,110, and is further connected to a microscope 116, which has an optical detection system 118 configured to capture a microscopic image of a sample 102. The image annotation system 100 and the microscope 116 can constitute a single microscope system 120.

[0026] The processor 114 is configured to receive image data corresponding to the microscope image of the sample 102 from the microscope 116. The processor 114 can receive image data, for example, via direct connection, network connection, or data medium. The processor 114 is configured to control the output unit 104 to display the microscope image of the sample 102 on the display area 106. The processor 114 is also configured to receive control data corresponding to user input to two input units 108 and 110. Furthermore, the processor 114 is configured to perform a method for identifying at least one image region 122a, 122b within the microscope image of the sample 102 based on user input. This method will be described below with reference to Figures 2 to 4.

[0027] Figure 2 is a flowchart of a method for identifying at least two image regions 122a and 122b within a microscope image of sample 102, based on user input.

[0028] In step S200, the process starts. In the optional step S202, the user specifies the zoom level of the microscope image displayed on the display area 106 of the output unit 104 for the purpose of determining the level of detail visibility. In step S204, a button is pressed. This button can be a button on a computer mouse 108, in particular the left mouse button, or a button on a keyboard 110. According to another embodiment, the button press can be associated with an object such as a stylus touching the surface of a touchscreen. The button press specifies the start of a first movement. In step S206, the cursor 112 is moved on the microscope image displayed on the display area 106 of the output unit 104 during the first movement. The first movement can be smoothly performed, for example, by the movement of a computer mouse 108 or the movement of a stylus on a touchscreen, and this movement specifies a first closed path. In step S208, the processor 114 specifies the end of the first movement. Steps S204 to S208 will be explained in more detail later with reference to Figure 3, particularly how the first closed path is identified. The button press and the first movement of the cursor 112 constitute the first input sequence. Next, in step S210, the processor 114 identifies the first image region 122a based on the first closed path. Preferably, the first image region 122a is the same as the region of the display microscope image enclosed by the first path. Alternatively, the processor 114 can also identify the first image region 122a based on the image content of the microscope image. For example, the processor 114 can ignore pixels of the microscope image that have a brightness value below a predetermined threshold.

[0029] In step S212, the button is pressed again. This time, the button press identifies the start of the second movement. In step S214, the cursor 112 is moved during the second movement on the microscope image displayed in the display area 106 of the output unit 104. The second movement identifies the second closed path. In step S216, the processor 114 identifies the end of the first movement. In step S218, the processor 114 determines whether the second closed path includes at least one proximity interval where the second path is located near the boundary of the first image area. This determination is based on at least one proximity condition. For example, the proximity condition can be whether the distance between an image point on the second closed path and the boundary of the first image area 122a is below a predetermined threshold. Step S218 can be performed simultaneously with step 214. Steps S212 to S218 will be explained in detail later, with reference to Figure 3, particularly regarding the identification of the second closed path and the adjacent section.

[0030] If the second closed path does not include a neighboring interval, in step S220, the processor 114 identifies a second image region of the sample image corresponding to the region of the display image enclosed by the second closed path. If the second path includes a neighboring interval, in step S222, the processor 114 adds the region of the display image located between the boundary of the first image region 122a and the neighboring interval to the region enclosed by the second closed path, for the purpose of identifying the second image region. In step S224, the processor 114 subtracts from the second image region any region in which the first image region 122a and the second image region 122b overlap. Steps S212 to S224 can be repeated for the purpose of identifying additional image regions 122a and 122b.

[0031] In the optional step S226, the processor 114 generates an object map. The object map contains multiple image points, each corresponding to a single image point in the image of sample 102. Each image point in the object map encodes information about which image region 122a, 122b the corresponding image point in the image of sample 102 belongs to. This object map can be used to generate a training dataset for a machine learning algorithm. The process then terminates in step S228.

[0032] Figure 3 is a flowchart of exemplary subprocesses for identifying closed paths.

[0033] These subprocesses correspond to steps S204-S208 and S212-S218, respectively, as explained earlier with reference to Figure 2. In step S300, the subprocesses start.

[0034] In step S302, the button is pressed. The position of the cursor 112 on the display image while the button is pressed is saved as the first position. Additionally, the position of the cursor 112 on the display image while the button is pressed for the first time in this subprocess is saved as the initial position. In step S304, the button is pressed again. The position of the cursor 112 on the display image while the button is pressed in step 304 is saved as the second position. If at least one image region has already been defined, in step S306, the processor 114 determines the distance between the currently saved first position and the boundary of the image region, and the distance between the second position and the boundary of the image region. If both distances are below a predetermined threshold, the processor 114 identifies the line between the first position and the second position as a proximity interval. In step S308, the line between the currently saved first position and the second position is added to a closed path. Subsequently, steps S302 to S308 are repeated until user input is received in step S310 and the subprocess is stopped. This user input can include, for example, pressing a different button or double-clicking. Once the subprocess is stopped, a line is added to the closed path between the last saved position of cursor 112 and the initial position of cursor 112.

[0035] There are several alternative methods for defining a closed path. For example, a closed path, or a segment of a closed path, can be identified by the path the cursor 112 moves from the time a button is pressed until it is released.

[0036] Figure 4 is a schematic diagram of the displayed microscope image, the first image region 122a, and the second closed path 400.

[0037] The second path 400 is shown as a dashed line in Figure 4 and includes section 402, which consists of a line where termination points 404a and 404b lie on the boundary of the first image region 122a. The termination points 404a and 404b are identified by button presses, as shown by the two arrows P1 and P2 in Figure 4. In step S306, the processor 114 determines that section 402 is a neighboring section. Therefore, the region 406 between section 402 and the boundary of the first image region 122a is added to the second image region 122b. This is shown in Figure 5, which is a schematic diagram of the displayed microscope image and the first and second image regions 122a and 122b.

[0038] Identical elements or elements that function similarly are indicated by the same reference numeral in all figures. As used herein, the term "and / or" includes all possible combinations of one or more of the related items and may be abbreviated as " / ".

[0039] Both the individual features of the embodiments and all combinations of features are deemed to be disclosed. Furthermore, the individual features of the embodiments are deemed to be disclosed in combination with the individual features or feature groups described above and / or in combination with the individual features or feature groups of the claims.

[0040] While several embodiments have been described in the context of the apparatus, it is clear that these embodiments also represent descriptions of the corresponding methods, where blocks or apparatus correspond to steps or features of steps. Similarly, embodiments described in the context of steps also represent descriptions of the corresponding blocks, items, or features of the corresponding apparatus. [Explanation of symbols]

[0041] 100 Image Annotation Systems 102 samples 104 Output Unit 106 Display area 108,110 input units 112 Cursor 114 processors 116 Microscope 118 Optical detection systems 120 Microscope Systems 122a,122b Image area 400 routes 402 sections 404a, 404b Point 406 area P1, P2 arrows

Claims

1. An image annotation system (100) for annotating a microscopic image of a sample (102), the image annotation system (100) comprising: an output unit (104) configured to display an image of said sample (102); an input unit (108, 110) configured to capture a first user input sequence including a first movement of a cursor (112) on a displayed image defining a first closed path, and to capture at least one second user input sequence including a second movement of the cursor (112) on the displayed image defining a second closed path; a processor (114); Including, The processor (114) identifying a first image region (122a) of the image of the sample (102) corresponding to an area of ​​the displayed image enclosed by the first closed path; determining whether the second path includes at least one adjacent section located near a boundary of the first image region (122a) based on at least one proximity condition; if the second path does not include the adjacent section, identifying a second image region (122b) of the image of the sample (102) corresponding to an area of ​​the displayed image enclosed by the second closed path; if the second path includes the adjacent section, identifying the second image region (122b) corresponding to the region of the display image enclosed by the second closed path and corresponding to at least one region of the display image between the boundary of the first image region (122a) and at least one of the adjacent sections; It is configured as follows: An image annotation system (100).

2. The processor (114) determines whether the second route includes at least one of the adjacent sections by: determining, for each image point on the second closed path, a distance between the image point on the second closed path and an image point on the boundary of the first image area (122a) that is closest to the image point on the second closed path; Compare each distance to a threshold, If at least a predetermined number of distances are smaller than the threshold, it is determined that the second route includes at least one of the adjacent sections. The method is configured to determine the the processor (114) is configured to identify at least one of the adjacent sections based on the image points for which the distance is less than the threshold. The image annotation system (100) of claim 1.

3. the processor (114) is configured to re-identify the second image region (122b) to exclude areas of the displayed image where the first image region (122a) and the second image region (122b) overlap. The image annotation system (100) of claim 1.

4. the first user input sequence and the second user input sequence include a button press and a button release, and a section of the first closed path or the second closed path is defined by a movement of the cursor (112) from the button press to the button release, respectively; The image annotation system (100) of claim 1.

5. the first user input sequence and the second user input sequence include a section of at least two image points of the displayed image, and a section of the first closed path or the second closed path, respectively, is defined by a line between two selected image points; The image annotation system (100) of claim 1.

6. the processor (114) is configured to generate an object map comprising image points each corresponding to an image point of the image of the sample (102), each image point of the object map encoding information about to which image region the corresponding image point of the image of the sample (102) belongs. The image annotation system (100) of claim 1.

7. the input unit (108, 110) is a computer mouse (108), and the processor (114) is configured to determine the first movement and the second movement of the cursor (112) based on a movement of the computer mouse (108); The image annotation system (100) of claim 1.

8. The image annotation system (100) includes a touch screen forming the output unit (104) and / or the input unit (108, 110). The image annotation system (100) of claim 1.

9. The processor (114) is configured to identify a start of the first movement and the second movement by determining that an object is touching the touchscreen, and to identify an end of the first movement and the second movement by determining that the object is no longer touching the touchscreen, and the processor (114) is configured to identify the first movement and the second movement of the cursor (112) based on a movement of the object on the touchscreen. The image annotation system (100) of claim 8.

10. the processor (114) is configured to determine, during input of the second user input sequence, whether the second path includes at least one of the adjacent sections. The image annotation system (100) of claim 1.

11. the output unit (104) is configured to display the first image area (122a) and the second image area (122b); The image annotation system (100) of claim 1.

12. an optical detection system (118) configured to capture an image of the sample (102); An image annotation system (100) according to any one of claims 1 to 11, A microscope system (120) comprising:

13. 1. A method for identifying at least two image regions within a microscopic image of a specimen (102) based on user input, the method comprising: displaying an image of the sample (102); capturing a first user input sequence including a first movement of a cursor (112) on a displayed image that defines a first closed path; capturing at least one second user input sequence including a second movement of the cursor (112) on the displayed image that defines a second closed path; identifying a first image region (122a) of the image of the sample (102) corresponding to an area of ​​the displayed image enclosed by the first closed path; determining whether the second path includes at least one adjacent section located near a boundary of the first image region (122a); if the second path does not include the adjacent section, identifying a second image region (122b) of the image of the sample (102) corresponding to an area of ​​the displayed image enclosed by the second closed path; if the second path includes the adjacent interval, identifying the second image region (122b) corresponding to the region of the displayed image enclosed by the second closed path and corresponding to at least one region of the displayed image between the boundary of the first image region (122a) and at least one of the adjacent intervals; A method comprising:

14. 14. A computer program comprising a program code for performing the method of claim 13 when the computer program is run on a processor (114).

Images