System, method, and program for supporting annotation

Abstract

To support annotation.SOLUTION: An image annotation support system includes a classification unit for generating classification information that includes a plurality of target regions constituting a target image, classified based on a feature seen in the target image, the target image being an image as a candidate to be annotated; and a control unit that arranges, on a screen of a display device, a classified image including the visualized classification information, in a comparable manner with the target image.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The disclosure herein relates to a system, method, and program that supports image annotation. [Background technology]

[0002] Currently, in the field of cell culture, a trained model constructed by performing supervised learning is used to grasp the culture state non-invasively from images. However, the construction of a trained model used for image processing requires a large amount of training data in which images are tagged with tags (correct answer labels). The process of creating such training data is called annotation.

[0003] In annotation, a large number of images are tagged manually by a human being one by one. This is a huge amount of work, and there is a demand for technology that can reduce the annotation workload. Technology related to this issue is described in, for example, Patent Document 1. Patent Document 1 discloses a user interface suitable for annotation. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-009314 A Summary of the Invention [Problem to be solved by the invention]

[0005] By adopting the technology described in Patent Document 1, it is possible to support user operations in annotation, and as a result, it is expected that the efficiency of annotation can be improved. However, the technology described in Patent Document 1 supports user operations, and does not support various decisions required of the user in annotation. For example, the decision of which area of ​​an image to tag is still left to the user, and no technology is described to support such decisions.

[0006] In view of the above circumstances, an object of one aspect of the present invention is to provide a new technique for supporting annotation. [Means for solving the problem]

[0007] A system according to one aspect of the present invention is a system for supporting annotation of images, the system including: a classification unit that generates classification information by classifying a plurality of target regions constituting a target image, which is a candidate image to which annotations are to be added, based on features that appear in the target image; and a control unit that arranges a classification image in which the classification information is visualized on a screen of a display device so that the classification information can be compared with the target image. The control unit superimposes the classification image on the target image, and determines a transmittance of the classification image based on the reliability of the classification image. do.

[0008] A system according to another aspect of the present invention is a system for supporting annotation of an image, comprising: The present invention is provided with a classification unit that generates classification information obtained by classifying a plurality of target regions constituting a target image, which is an image that is a candidate for adding annotation, based on features that appear in the target image, and a control unit that arranges a classification image in which the classification information is visualized on a screen of a display device so that the classification information can be compared with the target image, wherein the classification information includes, for each of the plurality of target regions, class information indicating the class into which the target region is classified, and score information indicating a probabilistic score for the classification into the class, and the control unit superimposes the classification image on the target image and determines the transmittance of the plurality of classification regions constituting the classification image based on the score information of a corresponding target region among the plurality of target regions. .

[0009] A method according to one aspect of the present invention is a method for supporting annotation of an image, which includes generating classification information by classifying a plurality of target regions constituting a target image, which is a candidate image to which annotations are to be added, based on features appearing in the target image, and arranging a classification image in which the classification information is visualized so as to be contrasted with the target image on a screen of a display device. The step of comparing the images includes superimposing the classification image on the target image and determining a transparency of the classification image based on the reliability of the classification image. .

[0010] According to another aspect of the present invention, there is provided a method for assisting in annotation of an image, the method comprising the steps of: A method for generating classification information by classifying a plurality of target regions constituting a target image, which is a candidate image to which an annotation is to be added, based on features appearing in the target image, and arranging a classification image in which the classification information is visualized, on a screen of a display device, so as to be contrasted with the target image, the classification information including class information indicating a class into which each of the plurality of target regions is classified, and score information indicating a probabilistic score for classification into the class, the arranging in a manner that allows comparison includes superimposing the classification image on the target image, and determining a transmittance of a plurality of classification regions constituting the classification image based on the score information of a corresponding target region among the plurality of target regions. .

[0011] A program according to one aspect of the present invention is a program for supporting image annotation, which generates classification information by classifying a plurality of target regions constituting a target image, which is a candidate image to which annotations are to be added, based on features appearing in the target image, and arranges a classified image in which the classification information is visualized on a screen of a display device so as to be contrasted with the target image. The step of comparing the images includes superimposing the classification image on the target image and determining a transparency of the classification image based on the reliability of the classification image. The processing is executed by a computer.

[0012] A program according to another aspect of the present invention is a program for supporting image annotation, comprising: A method for generating classification information by classifying a plurality of target regions constituting a target image, which is a candidate image to which an annotation is to be added, based on features appearing in the target image, and arranging a classification image in which the classification information is visualized, on a screen of a display device, so as to be contrasted with the target image, the classification information including class information indicating a class into which each of the plurality of target regions is classified, and score information indicating a probabilistic score for classification into the class, the arranging in a manner that allows comparison includes superimposing the classification image on the target image, and determining a transmittance of a plurality of classification regions constituting the classification image based on the score information of a corresponding target region among the plurality of target regions. The processing is executed by a computer. Effect of the Invention

[0013] According to the above aspect, it is possible to provide a new technique for supporting annotation. [Brief description of the drawings]

[0014] [Figure 1] 1 is a diagram illustrating a configuration of a system 200 according to a first embodiment. [Diagram 2] 1 is a diagram illustrating an example of the configuration of an imaging device 100. [Diagram 3] 1 is a diagram illustrating an example of the configuration of a light source unit 104 and an imaging unit 105. FIG. [Figure 4] FIG. 2 is a diagram illustrating an example of the configuration of a control device 130. [Diagram 5] 4 is a flowchart showing an example of processing performed by the system 200 according to the first embodiment. [Figure 6] FIG. 13 is a diagram showing an example of an annotation screen. [Figure 7] FIG. 13 is a diagram showing another example of the annotation screen. [Figure 8] FIG. 13 is a diagram showing yet another example of the annotation screen. [Figure 9] FIG. 13 is a diagram showing yet another example of the annotation screen. [Figure 10] 10 is a flowchart showing an example of a process performed by the system 200 in a learning stage. [Figure 11] 13 is a flowchart illustrating an example of a learning process of a feature extraction method. [Figure 12] FIG. 2 is a diagram showing an example of a feature extraction model M1. [Figure 13] 13 is a flowchart illustrating an example of a learning process of a normalization method. [Figure 14] 13 is a flowchart showing an example of a learning process of an information amount reduction method. [Figure 15] 13 is a flowchart illustrating an example of a classification method learning process. [Figure 16] 13 is a flowchart illustrating an example of a classification process performed by the system 200. [Figure 17] FIG. 13 is a diagram illustrating the first half of input and output in classification processing. [Figure 18] FIG. 13 is a diagram illustrating the latter half of input and output in the classification process. [Figure 19] FIG. 11 is a diagram for explaining a method of generating classification images. [Figure 20] FIG. 13 is a diagram showing yet another example of the annotation screen. [Figure 21] FIG. 13 is a diagram showing how to change the color assigned to a class. [Figure 22] 22A and 22B are diagrams illustrating classified images before and after the color allocation change shown in FIG. 21. [Figure 23] FIG. 11 is another diagram showing how to change the colors assigned to classes. [Figure 24] 24 is a diagram illustrating classified images before and after the color allocation change shown in FIG. 23. [Diagram 25] FIG. 13 is a diagram showing yet another example of the annotation screen. [Figure 26] 11 is a diagram for explaining a method for determining the transmittance of a classification image. FIG. [Figure 27] FIG. 13 is a diagram showing another example of a classification image. [Figure 28] FIG. 13 is a diagram showing yet another example of a classification image. [Figure 29] 10 is a flowchart showing an example of processing performed by the system according to the second embodiment. [Diagram 30] 13 is a flowchart illustrating an example of a time lapse photography process. [Diagram 31] FIG. 13 is a diagram showing an example of classified images arranged in chronological order. [Diagram 32] FIG. 13 is a diagram showing a state in which a selection operation is performed on classified images arranged in chronological order. [Diagram 33] FIG. 13 is a diagram showing an example of a culture data list screen. [Diagram 34] FIG. 13 is a diagram showing an example of a culture data top screen. [Diagram 35] FIG. 13 is a diagram showing an example of a time series display screen. [Diagram 36] FIG. 13 is a diagram showing an example of a composite image display screen. [Figure 37] FIG. 13 is a diagram showing an example of a composite element image display screen. [Figure 38] 13 is a flowchart showing an example of processing performed by the system according to the third embodiment. [Figure 39] FIG. 11 is a diagram for explaining a method of selecting a classification image. [Diagram 40] FIG. 13 is a diagram showing another example of classified images arranged in chronological order. [Diagram 41] 13 is a flowchart showing an example of processing performed by a system according to a fourth embodiment. [Diagram 42] 11 is a diagram for explaining a method for determining a transmittance of a classification image. FIG. [Diagram 43] 13 is a flowchart showing an example of processing performed by the system according to the fifth embodiment. [Diagram 44] FIG. 13 is a diagram showing yet another example of the annotation screen. [Diagram 45] FIG. 13 is a diagram for explaining a method for adjusting an annotation. [Figure 46] FIG. 13 is a diagram for explaining a method for adjusting an annotation. [Figure 47] FIG. 13 is a diagram for explaining a method for adjusting an annotation. [Figure 48]FIG. 13 is a diagram showing an example of a graph display of classification information. [Figure 49] FIG. 3 is a diagram illustrating an example of the configuration of a system 300. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] [First embodiment] Fig. 1 is a diagram illustrating the configuration of a system 200 according to this embodiment. Fig. 2 is a diagram illustrating the configuration of an imaging device 100. Fig. 3 is a diagram illustrating the configuration of a light source unit 104 and an imaging unit 105. Fig. 4 is a diagram illustrating the configuration of a control device 130. The configuration of the system 200 will be described below with reference to Figs. 1 to 4.

[0016] 1 is a cell culture system that photographs cells contained in a container C while culturing them. The system 200 is also a system that supports image annotation. Here, annotation refers to the task of adding information (tags) to an image to create teacher data, or information (tags) added to an image by the annotation task.

[0017] The system 200 includes one or more imaging devices 100 that capture images of cultured cells contained in a container C from below the container C, and a control device 130 that controls the imaging devices 100. Each of the imaging devices 100 and the control device 130 only need to be able to exchange data with each other. Therefore, each of the imaging devices 100 and the control device 130 may be connected to each other so as to be able to communicate with each other via a wire, or may be connected to each other so as to be able to communicate with each other wirelessly. The container C that contains the cultured cells is, for example, a flask. However, the container C is not limited to a flask, and may be other culture containers such as a dish or a well plate.

[0018] In order to photograph the cultured cells without removing them from the incubator 120, the imaging device 100 is used while being placed in the incubator 120, for example, as shown in FIG. 1. More specifically, the imaging device 100 is placed in the incubator 120 with a container C placed on a transmission window 101 of the imaging device 100, as shown in FIG. 1, and acquires an image of the sample (cells) in the container C according to an instruction from a control device 130. The transmission window 101 is a transparent top plate that constitutes the upper surface of a housing 102 of the imaging device 100, and constitutes a mounting surface on which the container is placed. The transmission window 101 is made of, for example, glass or transparent resin.

[0019] 1, the imaging device 100 includes a box-shaped housing 102 having a transparent transmission window 101 on the upper surface through which a container C is placed, and a positioning member 110 that positions the container C at a predetermined position on the transmission window 101 (mounting surface). The positioning member 110 is fixed to the housing 102. However, the positioning member 110 can be removed as necessary, and may be replaced with another positioning member having a different shape depending on the container to be used.

[0020] 2 and 3, the imaging device 100 further includes a stage 103 that moves within the housing 102, a pair of light source units 104 that illuminate the cultured cells, and an imaging unit 105 that captures images of the cultured cells. The stage 103, the light source unit 104, and the imaging unit 105 are housed within the housing 102. The light source unit 104 and the imaging unit 105 are installed on the stage 103, and move relative to the container C as the stage 103 moves within the housing 102.

[0021] The stage 103 changes the relative position of the photographing unit 105 with respect to the container C. The stage 103 is movable in X and Y directions that are parallel to the transmission window 101 (mounting surface) and perpendicular to each other. However, the stage 103 may also move in the Z direction (height direction) that is perpendicular to both the X and Y directions.

[0022] 2 and 3 show an example in which the light source unit 104 and the photographing unit 105 are installed on the stage 103 and move together within the housing 102, but the light source unit 104 and the photographing unit 105 may move independently within the housing 102. Also, while an example in which a pair of light source units 104 are arranged on the left and right sides of the photographing unit 105 is shown in FIG. 2 and 3, the arrangement and number of the light source units 104 are not limited to this example. For example, three or more light source units 104 may be provided on the stage 103, or only one light source unit 104 may be provided.

[0023] 3, the light source unit 104 includes a light source 106 and a diffusion plate 107. The light source 106 includes, for example, a light emitting diode (LED). The light source 106 may include a white LED, or may include a plurality of LEDs that emit light of a plurality of different wavelengths, such as R (red), G (green), and B (blue). The light emitted from the light source 106 is incident on the diffusion plate 107.

[0024] The diffusion plate 107 diffuses the light emitted from the light source 106. The diffusion plate 107 is not particularly limited, but may be, for example, a frosted diffusion plate having an uneven surface. However, the diffusion plate 107 may be an opal diffusion plate having a coated surface, or may be another type of diffusion plate. Furthermore, a mask 107a may be formed on the diffusion plate 107 to limit the emission area of ​​the diffused light. The light emitted from the diffusion plate 107 travels in various directions.

[0025] 3, the photographing unit 105 includes an optical system 108 and an image sensor 109. The optical system 108 collects light that has passed through the transmission window 101 and entered the housing 102. The optical system 108, which is focused on the bottom surface of the container C in which the cultured cells are present, collects the light that has entered the housing 102 onto the image sensor 109, thereby forming an optical image of the cultured cells on the image sensor 109.

[0026] The image sensor 109 is an optical sensor that converts detected light into an electrical signal. Although not particularly limited, the image sensor 109 may be, for example, a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS).

[0027] In the imaging device 100 configured as above, oblique illumination is adopted to visualize the sample S (cultured cells) in the container C, which is a phase object. Specifically, the light emitted by the light source 106 is diffused by the diffusion plate 107 and emitted outside the housing 102. That is, the light source unit 104 emits light traveling in various directions toward the outside of the housing 102 without passing through the optical system 108. Then, a part of the light emitted outside the housing 102 is reflected, for example, by the upper surface of the container C, and is deflected above the sample S. Furthermore, a part of the light deflected above the sample S is irradiated onto the sample S and enters the housing 102 by passing through the sample S and the transmission window 101. Then, a part of the light entering the housing 102 is collected by the optical system 108 to form an image of the sample S on the image sensor 109. Finally, the imaging device 100 generates an image of the sample S (cultured cells) based on the electrical signal output from the image sensor 109, and outputs the image to the control device .

[0028] The control device 130 is a device that controls the image capturing device 100. The control device 130 transmits image capturing instructions to the image capturing device 100 placed in the incubator 120, and receives images captured by the image capturing device 100.

[0029] The control device 130 is an image processing device that processes the image acquired by the photographing device 100. The control device 130 generates classification information that classifies the areas constituting the image into several classes based on the image features extracted from the image.

[0030] Furthermore, the control device 130 is a display control device that visualizes and displays the classification information. The control device 130 visualizes the classification information and arranges it on the screen in response to a request from a user. Hereinafter, an image in which the classification information is visualized will be referred to as a classification image.

[0031] The control device 130 may include one or more processors and one or more non-transitory computer-readable media, and may be, for example, a general computer. More specifically, the control device 130 may include, for example, one or more processors 131, one or more storage devices 132, an input device 133, a display device 134, and a communication device 135, as shown in FIG. 4, and may be connected via a bus 136.

[0032] Each of the one or more processors 131 is hardware including, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), etc., and performs programmed processing by executing a program 132a stored in one or more storage devices 132. The programmed processing includes, for example, a classification processing for generating classification information, a display control processing for arranging classified images on a screen, etc. In other words, the processor 131 is an example of a classification unit of the system 200, and is also an example of a control unit of the system 200. Furthermore, the one or more processors 131 may include an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), etc.

[0033] Each of the one or more storage devices 132 may include, for example, one or more arbitrary semiconductor memories and may further include one or more other storage devices. The semiconductor memories may include, for example, volatile memories such as RAM (Random Access Memory), and non-volatile memories such as ROM (Read Only Memory), programmable ROM, and flash memory. The RAM may include, for example, DRAM (Dynamic Random Access Memory), SRAM (Static Random Access Memory), and the like. The other storage devices may include, for example, magnetic storage devices including magnetic disks, optical storage devices including optical disks, and the like.

[0034] Note that the one or more storage devices 132 are non-transitory computer-readable media and are an example of a storage unit of the system 200. At least one of the storage devices 132 stores trained data 132b used to generate a classified image.

[0035] The input device 133 is a device that is directly operated by a user, and is, for example, a keyboard, a mouse, a touch panel, etc. The display device 134 is, for example, a liquid crystal display, an organic EL display, a CRT (Cathode Ray Tube) display, etc. The display may have a built-in touch panel. The communication device 135 may be a wired communication module or a wireless communication module.

[0036] 4 is an example of a hardware configuration of the control device 130, and the control device 130 is not limited to this configuration. The control device 130 is not limited to a general-purpose device, and may be a dedicated device.

[0037] FIG. 5 is a flowchart showing an example of a process performed by the system 200 according to the present embodiment. FIGS. 6 to 9 are diagrams showing examples of annotation screens. The annotation screen is a screen on which an annotation can be added to an image based on a user's operation. The system 200 performs the process shown in FIG. 5 to provide a working environment for adding annotations to an image together with information for supporting various decisions that a user should make in the annotation. Hereinafter, a specific example of a method for supporting annotations performed by the system 200 will be described with reference to FIGS. 5 to 9. The process shown in FIG. 5 is started, for example, by the processor 131 executing the program 132a.

[0038] 5 is started, first, the system 200 acquires an image that is a candidate for adding an annotation (step S1). Here, the photographing device 100 photographs the cultured cells in the container C and acquires an image that is a candidate for adding an annotation (hereinafter, referred to as a target image). Furthermore, the photographing device 100 outputs the acquired target image to the control device 130.

[0039] Next, the system 200 classifies each area of ​​the target image based on the features appearing in the target image (step S2). Here, the control device 130 acquires the target image output by the photographing device 100. Furthermore, in the control device 130, the processor 131, which is an example of a classification unit, uses the learned data 132b stored in the storage device 132 to generate classification information in which a plurality of target areas constituting the target image are classified based on the features appearing in the target image.

[0040] Each of the multiple target regions is, for example, one or more pixels that make up the target image. One pixel may make up one target region, or multiple pixels (e.g., 3 x 3 pixels) may make up one target region. The number of divisions for each of the multiple target regions may be freely determined by the user or the system. For example, a case where the number of pixels is 3*3 has been exemplified as the division unit, but the number of pixels vertically and horizontally does not necessarily have to be the same. The classification information includes at least class information corresponding to each of the multiple target regions. The class information is information indicating the class into which each target region is classified.

[0041] When the classification information is generated, the system 200 arranges a classification image in which the classification information is visualized on the screen of the display device 134 so as to be contrasted with the target image (step S3). Here, in the control device 130, the processor 131, which is an example of a control unit, generates a classification image in which the classification information is visualized based on the classification information, for example, by assigning to each region a color or pattern corresponding to the class into which the region is classified. The processor 131 further arranges the classification image on the annotation screen so as to be contrasted with the target image.

[0042] FIG. 6 shows a state where "photographed image" is selected in the region R1, and as a result, a target image (image 1) photographed by the photographing device 100 is arranged in the region R0. FIG. 7 shows a state where "classified image" is selected in the region R1, and as a result, a classified image (image 8) is arranged in the region R0. In step S3, as shown in FIG. 6 and FIG. 7, the processor 131 may arrange the classified image in a manner that allows comparison with the target image by switching between image 1, which is a target image, and image 8, which is a classified image, and arranging them on the screen. Note that the switching between image 1 and image 8 may be performed by a user's operation (e.g., selection of a radio button in the region R1), or may be performed automatically without a user's operation. For example, image 1 and image 8 may be switched at regular intervals.

[0043] FIG. 8 shows a state where "captured image and classified image" is selected in area R1, and as a result, a target image (image 1) and a classified image (image 8) are arranged overlapping in area R0. That is, the control device 130 superimposes the classified image on the target image. Furthermore, the control device 130 can change the transparency of the classified image superimposed on the target image in response to a predetermined instruction (e.g., an operation on area R2). This allows the user to check both images at the same time. In step S3, the processor 131 may arrange the target image, image 1, and the classified image, image 8, overlapping on the screen as shown in FIG. 8, thereby arranging the classified image in a manner that allows comparison with the target image.

[0044] 8 shows an example in which the transmittance of the classification image can be adjusted in region R2, but it is sufficient if the transmittance of at least one of the target image and the classification image can be adjusted. The transmittance adjustment may be performed automatically. For example, the transmittance may be adjusted in response to an operation requesting an enlarged display of an image.

[0045] After the classified image is arranged on the screen, the system 200 accepts an input from the user and adds annotations to the image (step S4). First, the user determines that the target image displayed on the screen is an image to which an annotation should be added, and performs an input to add an annotation to the target image. In response to this, the processor 131 adds an annotation to the target image according to the user's input, and generates training data.

[0046] 5, the system 200 can support annotation by a user. This will be described in detail below.

[0047] First, by displaying the classification image, the user can easily guess the part to which annotation should be added. This is because, as described above, the classification image is an image that visualizes classification information (including class information) that is classified based on the characteristics of the target image.

[0048] In more detail, the classification information can be considered to be generated by the classification unit capturing the cells in the target image as a pattern and classifying the differences in texture in the target image caused by differences in the size and contrast of the cells. Therefore, it is considered that cells having similar morphologies are captured in the regions of the target image classified into the same class. Conversely, it is considered that cells having different morphologies are captured in the regions of the target image classified into different classes. For example, assuming that an undifferentiated region and a differentiated region are respectively annotated for an image of iPS cells in culture, the undifferentiated region and the differentiated region have different morphologies and are classified into different classes, whereas the undifferentiated regions (or differentiated regions) have similar morphologies and are classified into the same class. Therefore, when a target image in which undifferentiated regions and differentiated regions are mixed is captured, the user can predict that a region corresponding to one of the classes of the classified image generated from the target image is an undifferentiated region, and based on that premise, the user can identify the undifferentiated region from the classified image. Similarly, the differentiated region can be identified from the classified image.

[0049] Furthermore, by arranging the classification images so as to be contrasted with the target image, the user can check the target image for the portion that has been identified using the classification images and decide whether or not the portion should be annotated. Note that FIG. 9 shows a state in which annotation 9 is added to the target image (image 1) visible beneath the classification image while referring to the classification image (image 8). The type of annotation to be added can be selected, for example, in area R3. In this way, the user only needs to check the target image in detail for the range narrowed down to a certain extent using the classification images, making it possible to add annotations accurately and efficiently.

[0050] As described above, according to the system 200, by displaying classified images, it is possible to provide information to an annotation worker to assist in various judgments. Therefore, it is expected that the efficiency and reliability of annotation will be improved. In addition, by providing a user with information to assist in judgments necessary for annotation, even a person who does not necessarily have specialized knowledge can accurately annotate, and the constraints on selecting an annotation worker are also alleviated. Therefore, since it is easier to secure an annotation worker while improving the work efficiency, it becomes possible to efficiently annotate a large number of images compared to the conventional method.

[0051] In the present embodiment, the control device 130 controls the image capture device 100 and generates classified images used to support annotation. However, these may be performed by separate devices. For example, the control device 130 may control the image capture device 100, and a device other than the control device 130 may generate classified images, and the device other than the control device 130 may display classified images in response to a request from a user. Also, the control device 130 may control the image capture device 100 and generate classified images, and a device other than the control device 130 may display classified images in response to a request from a user. The device other than the control device 130 may be, for example, a client terminal, including a tablet, a smartphone, a computer, etc. These client terminals may be configured to be able to communicate with the control device 130. The client terminal may include one or more processors and one or more non-transitory computer-readable media, just like the control device 130.

[0052] FIG. 10 is a flowchart showing an example of processing performed by the system 200 in the learning stage. FIG. 11 is a flowchart showing an example of learning processing of the feature extraction method. FIG. 12 is a diagram showing an example of the feature extraction model M1. FIG. 13 is a flowchart showing an example of learning processing of the normalization method. FIG. 14 is a flowchart showing an example of learning processing of the information amount reduction method. FIG. 15 is a flowchart showing an example of learning processing of the classification method. FIG. 16 is a flowchart showing an example of classification processing by the system 200. FIG. 17 is a diagram showing an example of the first half of input / output in the classification processing. FIG. 18 is a diagram showing an example of the second half of input / output in the classification processing. Hereinafter, the classification processing of step S2 in FIG. 5 and the learning processing performed in advance to enable the classification processing will be described in detail with reference to FIGS. 10 to 18.

[0053] The classification process in step S2 in FIG. 5 described above is performed using trained data obtained by machine learning. First, the learning process performed to obtain trained data will be described with reference to FIG. 10 to FIG. 15. Note that the learning process is desirably performed for each annotation target. For example, in the case of annotating differentiated and undifferentiated regions of iPS cells, it is desirably performed learning using multiple images acquired during the iPS cell culture process.

[0054] The learning process shown in Fig. 10 is performed, for example, by the control device 130, but may be performed by a device different from the control device 130. In addition, the images used for learning are acquired, for example, by the imaging device 100, but may be acquired by a device different from the imaging device 100. In other words, as long as the device performing the classification uses learned data obtained by learning, learning and classification may be performed by different devices. In the following, an example will be described in which the control device 130 executes the learning process using images of cultured cells acquired by the imaging device 100 as learning images.

[0055] The learning process shown in FIG. 10 includes a process of learning a feature extraction method (step S10), a process of learning a method of normalizing the feature extraction results (step S20), a process of learning a method of reducing the amount of information (step S30), and a process of learning a classification method (step S40).

[0056] In step S10 of learning the feature extraction method, the control device 130 first acquires learning images (step S11), and learns autoencoding by deep learning using the acquired learning images (step S12), as shown in Fig. 11. The control device 130 repeats the processes of steps S11 and S12 for all learning images.

[0057] In step S12, the same learning image is set for both the input data and the teacher data of the neural network, and the neural network is repeatedly trained. That is, a plurality of learning images are used to train the neural network, thereby constructing an autoencoder that extracts image features. A specific example of the network configuration is shown in FIG. 12, but the model M1 shown in FIG. 12 is merely an example, and the number of channels and the number of layers can be changed as appropriate. Note that FIG. 12 illustrates an autoencoder with five intermediate layers. Furthermore, the learning in step S12 is performed so that a total of five loss functions, from when only one intermediate layer of the model M1 is used to when all five intermediate layers are used, are averaged.

[0058] After performing the processes of steps S11 and S12 for all learning images, the control device 130 determines whether to end learning (step S13). Here, the determination may be based on whether the loss function is equal to or less than a reference value. If it is determined that the loss function is equal to or less than the reference value, the control device 130 ends learning (step S13 YES) and outputs the autoencoder constructed in step S12 (hereinafter, referred to as trained data A for feature extraction) as trained data (step S14). Note that the trained data A may be all or a part of the neural network constituting the autoencoder (for example, information from the input layer to the fifth hidden layer).

[0059] In step S20 of learning the normalization method, as shown in FIG. 13, the control device 130 first acquires a learning image (step S21), and then generates an intermediate layer image using the learned data A obtained in step S10 (step S22).

[0060] In step S22, the control device 130 inputs a learning image to the model M1, for example, and outputs intermediate layer data from which the learning image is compressed and features are extracted as an intermediate layer image. If the intermediate layer data is a one-dimensional array rather than a two-dimensional array (image format), it is converted to a two-dimensional array (image format) and output as an intermediate layer image. For example, if all of the first to fifth intermediate layers of the model M1 are used, intermediate layer images with a total of 368 (=16+32+64+128+128) channels are generated in step S22.

[0061] When the intermediate layer image is generated in step S22, the control device 130 performs statistical processing on the intermediate layer image for each channel to generate a statistical image (step S23). Here, the statistical processing refers to an image filter process that performs statistical calculations, such as calculations of the average or variance. In other words, the statistical processing is a spatial filtering process that uses information on the pixel of interest and its surrounding pixels.

[0062] The statistical calculation performed in the statistical processing of step S23 may be one type (for example, only averaging) or two or more types (for example, averaging and variance calculation). When two or more types of statistical calculations are performed, the control device 130 may output the respective calculation results as separate channels.

[0063] The control device 130 repeats the processes from step S21 to step S23 for all learning images (step S24 NO). When the processes have been performed for all learning images (step S24 YES), the control device 130 learns normalization for each channel (step S25) and outputs the learned data (step S26).

[0064] In step S25, the control device 130 extracts the maximum pixel value and the minimum pixel value for each channel from the multiple statistical images generated in step S23 as normalization parameters. In step S26, the control device 130 outputs the extracted normalization parameters (hereinafter, referred to as learned data B for normalization) as learned data.

[0065] Finally, the control device 130 normalizes the statistical image for each channel using the learned data B for normalization to generate a feature image, and outputs the generated feature image (step S27). Here, the control device 130 converts the statistical image using the learned data B to generate a feature image whose pixel values ​​fall within the range from 0 to 1.

[0066] In step S30 of learning the information amount reduction method, the control device 130 first acquires a characteristic image (step S31), as shown in Fig. 14. After that, the control device 130 reduces the image size of the characteristic image to generate a second characteristic image, and outputs the generated second characteristic image (step S32).

[0067] In step S32, the control device 130 may reduce the image size by thinning out pixels from the feature image based on a predetermined rule, or may reduce the image to an arbitrary size using an interpolation method.

[0068] The control device 130 repeats the processes of steps S31 and S32 for all the feature images (step S33 NO). When the processes have been performed for all the feature images (step S33 YES), the control device 130 performs principal component analysis on the feature vector of the second feature image whose image size has been reduced (step S34). Note that the principal component analysis may be performed after converting the second feature image into a one-dimensional array as necessary.

[0069] The control device 130 outputs a transformation matrix (hereinafter, referred to as trained data C for reducing the amount of information) that outputs the principal components identified by the principal component analysis in step S34 for the input of the second feature image as trained data (step S35). Note that it is preferable to set the number of principal components output by the transformation matrix appropriately according to the complexity of the classification target, and it is preferable to leave more principal components as the classification target becomes more complex.

[0070] 15, in step S40 of learning the classification method, the control device 130 first acquires a second feature image (step S41), and then generates a third feature image using the learned data C obtained in step S30 (step S42). Note that the third feature image is an image composed of the principal components of the second feature image, and has a lower number of dimensions of the feature vector (i.e., the number of channels) than the third feature image.

[0071] The control device 130 repeats the processes of steps S41 and S42 for all second feature images (step S43 NO). When the processes have been performed for all second feature images (step S43 YES), the control device 130 performs cluster analysis on the feature vectors of the third feature image with the reduced number of dimensions (step S44).

[0072] For example, the K-means method may be adopted as the cluster analysis performed in step S44. The K-means method is preferable because it allows the learned classification rules to be stored, but the method of cluster analysis is not limited to the K-means method. Note that the cluster analysis may be performed after converting the third feature image into a one-dimensional array as necessary.

[0073] The control device 130 outputs the classification rules (hereinafter, referred to as learned data D for classification) created in the cluster analysis in step S44 as learned data (step S45). Note that it is preferable to set the number of clusters (number of classes) of the classification result appropriately according to the classification target before the cluster analysis in step S44 is performed.

[0074] The system 200 stores the learned data 132b (learned data A to D) obtained by the above-mentioned learning process in the storage device 132. When the user performs annotation, the system 200 uses the learned data to assist the user in annotation.

[0075] Next, the classification process in step S2 in FIG. 5 will be described in detail with reference to FIGS. 16 to 18, focusing on a method of utilizing the learned data generated in the learning process shown in FIGS.

[0076] The control device 130 first acquires a target image (step S101). Here, for example, in step S1, the control device 130 acquires an image of cells cultured in a container C captured by the imaging device 100 as the target image. The target image (image 1) is, for example, one image (one channel) as shown in FIG.

[0077] When the target image is acquired, the control device 130 generates an intermediate layer image using the learned data A (step S102). Here, the processor 131 inputs the intermediate layer image to the autoencoder stored in the storage device 132 as the learned data A, and acquires the intermediate layer data as the intermediate layer image. The intermediate layer image (image 2) from which the features of the target image (image 1) are extracted has, for example, the same number of channels as the number of channels in the intermediate layer, as shown in FIG. 17.

[0078] When the intermediate layer image is generated, the control device 130 performs statistical processing on the intermediate layer image for each channel to generate a statistical image (step S103). Here, the processor 131 generates a statistical image by performing spatial filtering processing on the intermediate layer image. Note that the spatial filtering processing performed in this step is similar to the processing in step S23 in FIG. 13. As shown in FIG. 17, the statistical image (image 3) has the same number of channels as the intermediate layer image (image 2).

[0079] When the statistical image is generated, the control device 130 normalizes the statistical image for each channel using the learned data B to generate a feature image (step S104). Here, the processor 131 converts the statistical image into a feature image whose pixel values ​​fall within the range from 0 to 1, using normalization parameters stored in the storage device 132 as the learned data B. Since the normalization parameters are stored for each channel, the normalization parameters corresponding to the channels of the statistical image are used in step S104. The feature image (image 4) has the same number of channels as the statistical image (image 3), as shown in FIG. 17.

[0080] When the feature image is generated, the control device 130 reduces the image size of the feature image to generate a second feature image (step S105). Here, the processor 131 may reduce the image size by thinning out pixels from the feature image based on a predetermined rule, as in the process of step S32 in Fig. 14, or may reduce the image to an arbitrary size using an interpolation method. The second feature image (image 5) has the same number of channels as the feature image (image 4), as shown in Fig. 18.

[0081] When the second feature image is generated, the control device 130 generates a third feature image by reducing the number of dimensions of the feature vector from the second feature image using the learned data C (step S106). Here, the processor 131 converts the second feature image into a third feature image consisting of the principal components of the second feature image using a transformation matrix stored in the storage device 132 as the learned data C. The third feature image (image 6) has a smaller number of channels (three channels in this example) than the second feature image (image 5), as shown in FIG. 18.

[0082] When the third feature image is generated, the control device 130 classifies the feature vector (three-dimensional in this example) of the third feature image using the learned data D to generate an index image (classification information) (step S107). Here, the processor 131 clusters a plurality of feature vectors corresponding to a plurality of pixels constituting the third feature image using the classification rules stored in the storage device 132 as the learned data D. The processor 131 further generates an index image by two-dimensionally arranging indexes (class information) indicating the classified classes. The index image (image 7) is, for example, a single image (one channel) as shown in FIG. 18, and is an example of the above-mentioned classification information.

[0083] The system 200 stores the index images obtained by the above-mentioned classification process in the storage device 132. Then, when a user performs annotation, the system 200 generates classified images by visualizing the index images according to the indexes, and arranges the classified images on the screen so as to be contrasted with the target image, as shown in Fig. 19. This makes it possible to support the worker who performs annotation.

[0084] FIG. 20 is a diagram showing yet another example of the annotation screen. The relationship between the classification image and the target image may be confirmed on the annotation screen as shown in FIG. 20. In FIG. 20, a window W is displayed on the annotation screen to show the relationship between the color on the classification image and the legend of the area in the target image classified into that color for each class. Note that the number of legends is not limited to one for each class, and multiple legends may be displayed. Also, as shown in FIG. 20, an explanation of the object (or class) indicated by the legend and color can be input in the window W. For example, an expert in this field can input an explanation, and an inexperienced worker can refer to the explanation input by the expert, so that the classification in the classification image can be correctly understood.

[0085] FIG. 21 is a diagram showing how colors assigned to classes are changed. FIG. 22 is a diagram showing an example of classified images before and after the color allocation change shown in FIG. 21. FIG. 23 is another diagram showing how colors assigned to classes are changed. FIG. 24 is a diagram showing an example of classified images before and after the color allocation change shown in FIG. 23. FIG. 25 is a diagram showing yet another example of an annotation screen. FIG. 26 is a diagram for explaining a method of determining the transmittance of classified images. FIG. 27 is a diagram showing another example of classified images. FIG. 28 is a diagram showing yet another example of classified images. The settings related to visualization of index images may be changed by the user as appropriate. Hereinafter, with reference to FIG. 21 to FIG. 28, an example in which a user changes settings to change the display mode of classified images will be described.

[0086] As shown in Figs. 21 and 22, the control device 130 may change the colors assigned to the classes in response to an operation on the window W performed by the user. Fig. 21 shows an example in which the color assigned to the third class is changed to the same color as the color assigned to the second class, and further, the explanation of the third class is made common to the explanation of the second class. In response to such an operation by the user, the control device 130 may change the classification image arranged on the screen from image 8 to image 8a, as shown in Fig. 22. Image 8a is a classification image in which the index image is visualized according to the changed settings shown in Fig. 21, and is an image in which the same color is assigned to the second class and the third class. In this way, an environment in which annotation is easy to perform may be created by a human making adjustments to the classification by the system 200.

[0087] As shown in Figs. 23 and 24, the control device 130 may change the color assigned to a class to a transparent color in response to an operation on the window W performed by the user. Fig. 23 shows an example in which the color assigned to the second class is changed to a transparent color. In response to such an operation by the user, the system 200 may change the classification image arranged on the screen from image 8a to image 8b as shown in Fig. 24. By changing from image 8a to image 8b, the area classified into the class changed to the transparent color becomes transparent, and the target image (image 1) arranged below the classification image can be seen. In this way, only the areas classified into some classes may be made transparent, and the state of the target (cell) may be confirmed in detail in the actual image (target image).

[0088] As shown in FIGS. 21 to 24, the control device 130 may change the display mode assigned to the class information in the classification image in response to a predetermined instruction (for example, an operation by a user).

[0089] Although Fig. 23 and Fig. 24 show an example in which the transmittance of a classification image is changed for each class, the system 200 may change the transmittance of a classification image for each region. As shown in Fig. 25 and Fig. 26, the control device 130 may automatically determine the transmittance of a classification image based on the reliability of classification. Fig. 25 shows an example in which the control device 130 automatically changes the transmittance of a classification image superimposed on a target image for each region in response to a predetermined instruction from a user (for example, an operation to select "Auto" in region R2). Note that the transmittance of the classification image 8c shown in Fig. 25 is determined, for example, according to the reliability of classification.

[0090] In order to realize such a transmittance according to reliability, when generating an index image (classification information), as shown in Fig. 26, the reliability of classification, that is, score information indicating a probabilistic score for classification into a class, may be output for each region together with the index (class information). Therefore, the classification information may include class information indicating the classified class for each of a plurality of target regions, and score information indicating a probabilistic score for classification into the class. For example, when the K-means method is used in the cluster analysis when generating the index image, the control device 130 may output the distance from the center of gravity of the cluster (or information normalizing the distance) as the probabilistic score.

[0091] Since the classification information includes score information, the control device 130 may determine the transmittance of the classification regions constituting the classification image based on the score information of the corresponding target region among the target regions. More specifically, by setting the transmittance high when the probabilistic score is low, it becomes possible to better confirm the actual state of cells in the target image for regions with low classification reliability. On the other hand, for regions with high classification reliability, annotation can be performed mainly based on the information of the classification image. Note that FIG. 26 shows an example in which the transmittance of each region of the classification image is adjusted using a score of 70% as a threshold.

[0092] In the above, an example was shown in which classification information is visualized using differences in color, but various methods can be used to assign colors. For example, an index may be assigned to H (hue) in the HLS color space. In addition, a color that is easy for humans to distinguish may be selected and an index may be assigned to the selected color. Furthermore, indexes may be assigned to any color according to the purpose, without being limited to the above examples.

[0093] Also, the classification information may be visualized using shades of color instead of differences in color. Furthermore, the classification information may be visualized using patterns. For example, as shown in FIG. 27, the control device 130 may arrange an image 8d in which regions are visualized using different patterns for each class on the screen as a classification image.

[0094] In addition, in the above, an example was shown in which classification information was visualized by filling an area with a specific color or pattern, but the classification information may be visualized by, for example, outlining an area instead of filling it, as long as it can distinguishably divide areas of different classifications. For example, as shown in FIG. 28, the control device 130 may arrange an image 8e, which depicts the outlines of areas classified into the same class, on the screen as a classification image. The outlines may be drawn with lines of different colors for each class, lines of different thicknesses for each class, lines of different line types for each class, or combinations of these.

[0095] [Second embodiment] Fig. 29 is a flowchart showing an example of processing performed by the system according to this embodiment. Fig. 30 is a flowchart showing an example of time lapse photography processing. Figs. 31 and 32 are diagrams showing examples of classified images arranged in chronological order. Hereinafter, processing performed by the system according to this embodiment will be described with reference to Figs. 29 to 32. The processing shown in Fig. 29 is started, for example, by the processor 131 executing the program 132a.

[0096] The system according to this embodiment assists in the selection of an image to which an annotation should be added by performing the process shown in Fig. 29. The system according to this embodiment (hereinafter simply referred to as the system) differs from the system 200 according to the first embodiment in that the system performs the process shown in Fig. 29 instead of or in addition to the process shown in Fig. 5, but is similar to the system 200 in other respects.

[0097] When the process shown in FIG. 29 is started, the system first performs time lapse photography (step S200). Specifically, as shown in FIG. 30, the control device 130 acquires the conditions of time lapse photography such as the photography time and the photography position (step S201), and when the photography time comes (step S202 YES), transmits a photography instruction to the photographing device 100 (step S203). The photographing device 100 that has received the photography instruction from the control device 130 performs photography according to the photography instruction and transmits the acquired target image to the control device 130. The control device 130 that has received the target image performs, for example, a classification process shown in FIG. 16 (step S204) and outputs the classification information generated by the classification process (step S205). The control device 130 repeats the above process until the end time of the time lapse photography is reached (step S206 YES).

[0098] When the time lapse photography is completed, the control device 130 acquires a plurality of pieces of classification information obtained by the time lapse photography (step S210). Note that the plurality of pieces of classification information acquired here correspond to a plurality of target images captured at mutually different times.

[0099] Thereafter, the control device 130 generates a plurality of classified images by visualizing the acquired plurality of pieces of classification information, and then arranges the generated plurality of classified images on the screen based on the shooting times of the plurality of target images corresponding to the plurality of classified images, as shown in Fig. 31 (step S220).

[0100] Fig. 31 shows a state in which a plurality of classified images (images P1 to P24) are arranged in chronological order. Specifically, Fig. 31 shows a plurality of classified images generated based on target images taken every four hours starting from 3 pm on the first day, and arranged in the order of the time of their photographing.

[0101] An arrow marked with "ME" indicates that a medium change has been performed, and an arrow marked with "P" indicates that a passaging has been performed. Information regarding the time when the medium change or passaging is performed is recorded when an operator presses an operation button provided on the imaging device 100. The control device 130 may display a display indicating the timing of the medium change or passaging together with the classified image based on the time information recorded in the imaging device 100. Since the culture environment changes significantly due to the medium change or passaging, providing information indicating when these were performed is useful for selecting annotation images.

[0102] FIG. 32 shows a state where a user selects an image to which an annotation should be added from among a plurality of classified images arranged based on the shooting time. In this example, three images, image P6, image P17, and image P19, are selected. Furthermore, when the user is in the middle of a step of actually adding annotations, it is possible to display not only images to which annotations should be added, but also images to which annotations have already been added by the user. Specifically, a mark may be displayed at the shooting time when one or more annotations were added to the image. Also, a mark may be displayed on images to which annotations have already been added. This can further reduce the effort required for annotation work.

[0103] After the multiple classified images are arranged on the screen, the system then accepts an input from the user and adds annotations to the images (step S230). Note that the process of step S230 is the same as the process of step S4 in FIG.

[0104] By performing the process shown in Fig. 29, the system can assist the user in selecting an image to which an annotation should be added. This point will be described in detail below.

[0105] In order to achieve high learning efficiency with a relatively small number of images in machine learning, it is desirable that the teacher data created by annotation contains a variety of images. In the system according to this embodiment, since a plurality of classified images are arranged on the screen based on the shooting time, the user can easily grasp the changes that have occurred in the target image by comparing the plurality of classified images. For example, the user can select a variety of images simply by selecting images based on a criterion such as prioritizing images with large changes. Therefore, the user can easily and appropriately select the image to be annotated. In particular, by displaying a plurality of classified images side by side, rather than displaying a plurality of target images side by side, the user can grasp the changes that have occurred in the image at a glance. Therefore, it is possible to select the image to be annotated in a short time, and the efficiency of the annotation work is improved. In addition, by combining with the annotation support method according to the first embodiment, the process from selecting the image to be annotated to the annotation work is consistently supported, so that the user can perform annotation more efficiently.

[0106] In the above, an example has been described in which an image to be annotated is selected after the time-lapse photography is completed, but the image to be annotated may be selected during the time-lapse photography period. The control device 130 may display classified images created from images already acquired in chronological order, allowing the user to select an image to be annotated at any time.

[0107] Fig. 33 is a diagram showing an example of a culture data list screen. Fig. 34 is a diagram showing an example of a culture data top screen. Fig. 35 is a diagram showing an example of a time-series display screen. Fig. 36 is a diagram showing an example of a composite image display screen. Fig. 37 is a diagram showing an example of a composite element image display screen. Hereinafter, with reference to Figs. 33 to 37, an example of screen transitions until an image to be annotated is selected will be specifically described.

[0108] First, as shown in Fig. 33, the control device 130 displays a list of culture data acquired by time lapse photography on the display device 134 and allows the user to select. The culture data may include data from ongoing time lapse photography. Fig. 33 shows that "Test 4", which is culture data from ongoing time lapse photography on a device with device ID "Device2", has been selected by the user.

[0109] When the culture data is selected, the control device 130 arranges a container image (image CT) simulating the culture container on the screen of the display device 134. The control device 130 may further arrange, on the container image, a classification image based on a target image obtained by photographing a sample in the container. The classification image may be arranged on the container image based on the photographing position of the target image on the culture container.

[0110] 34 shows an example in which the culture vessel is a multi-well plate. A vessel image (image CT) simulating a multi-well plate includes well images W1 to W6 simulating multiple wells. The control device 130 may arrange, at positions corresponding to the multiple well images, multiple classification images (image 8) corresponding to multiple target images capturing images of cells in the corresponding wells.

[0111] When the culture vessel is a vessel having multiple culture regions, such as a multi-well plate as shown in Fig. 34, it is desirable to arrange a classification image showing the state of each culture region for each culture region. This allows the user to grasp the state of each culture region at a glance, making it easy to select a culture region to be checked in more detail. It is even more desirable if the classification image corresponds to the latest target image.

[0112] When a culture area (for example, well image W6) is selected on the container image, the control device 130 arranges on the screen a plurality of classification images generated based on a plurality of target images captured in the culture area, based on the capture time, as shown in Fig. 35. Note that images Pa to Pi shown in Fig. 35 are classification images made by pasting together 5 x 5 target images captured by scanning the inside of each well.

[0113] Furthermore, when a specific classified image (composite image) is selected from a plurality of classified images (composite images) arranged in chronological order, the control device 130 enlarges and displays the selected classified image (image Pc which is a composite image), as shown in Fig. 36. Note that when it is desired to display classified images (composite images) corresponding to different times, it is sufficient to press the buttons arranged to the left and right of the classified image.

[0114] Thereafter, when an area in the classified image (image Pc) that the user wishes to observe in particular detail is selected, the control device 130 enlarges and displays the classified image (image Pc9, which is a composite element image) containing the selected area, as shown in Fig. 37. The user checks the sufficiently enlarged classified image (image Pc9) and decides whether or not to annotate the corresponding target image. When a predetermined operation by the user (e.g., pressing the "annotation" button) is detected, the control device 130 transitions to an annotation screen.

[0115] As described above, the system according to the present embodiment can support the user in selecting images before transitioning to the annotation screen by using the chronological display of classified images. This allows the user to select appropriate images, and therefore allows efficient annotation by avoiding the need to redo image selection.

[0116] In this embodiment, an example has been shown in which only classified images are displayed until transition to the annotation screen, but the target image may be used in addition to the classified image to select an image to be annotated. For example, similar to the first embodiment, the classified image and the target image may be displayed overlapping each other, or the classified image and the target image may be displayed by switching between them. That is, a plurality of classified images may be arranged on the screen in chronological order so as to be contrasted with the target image.

[0117] In addition, in the present embodiment, an example is shown in which classified images of the shooting time are displayed in a list in chronological order, but it is not necessary to display all classified images of the shooting time. For example, a user may select an image to be hidden on the screen shown in FIG. 32, and the user may select an image that has less change compared to adjacent images as an image to be hidden. The system may omit displaying the image selected according to the user's selection, thereby narrowing down the images to be displayed in a list. By suppressing the number of images to be displayed in a list, the visibility is improved, making it easier to compare images. This makes it even easier to select an image to be annotated.

[0118] [Third embodiment] FIG. 38 is a flowchart showing an example of processing performed by the system according to the present embodiment. FIG. 39 is a diagram for explaining a method for selecting classified images. FIG. 40 is a diagram showing another example of classified images arranged in time series. Hereinafter, processing performed by the system according to the present embodiment will be described with reference to FIG. 38 to FIG. 40. The system according to the present embodiment supports the selection of an image to which an annotation should be added by performing the processing shown in FIG. 38. The system according to the present embodiment (hereinafter, simply referred to as the system) is different from the system according to the second embodiment in that it performs the processing shown in FIG. 38 instead of the processing shown in FIG. 29, but is similar to the system according to the second embodiment in other respects. The processing shown in FIG. 38 is started, for example, by the processor 131 executing the program 132a.

[0119] When the process shown in Fig. 38 is started, first, the system performs time lapse photography (step S310), and acquires a plurality of pieces of classification information obtained by the time lapse photography (step S320). Note that the processes in steps S310 and S320 are similar to the processes in steps S200 and S210 in Fig. 29.

[0120] Thereafter, the control device 130 selects a plurality of classified images to be arranged on the screen based on a comparison of the plurality of classification information acquired in step S320 (step S330). Here, the control device 130 may select images in which a large change has occurred preferentially, and the specific selection method is not particularly limited. For example, as shown in FIG. 39, the control device 130 may calculate the change rate between two adjacent images in a plurality of classified images (images Pa, Pb, Pc, etc.) arranged in chronological order, and select a classified image (in this example, image Pc) in which the change rate is equal to or greater than a threshold value. Also, instead of selecting an image based on the change rate of adjacent images, the control device 130 may select the next image (for example, image Pc) based on the change rate relative to the selected image (for example, image Pa).

[0121] When the multiple classified images are selected, the control device 130 arranges the selected multiple classified images on the screen based on the shooting times of the corresponding multiple target images (step S340). Note that the process of step S340 is similar to the process of step S220 in Fig. 29, except that instead of arranging all of the classified images obtained by time lapse photography on the screen, only the classified images selected in step S330 are arranged on the screen.

[0122] Fig. 40 shows a state in which a plurality of classified images (images Pc, Pd, Pi) selected in step S330 are arranged in chronological order. Images not selected in step S330 may be completely removed from the screen, or may be arranged so as to be hidden behind similar images, as shown in Fig. 40.

[0123] After the selected group images are arranged on the screen, the system then accepts an input from the user and adds annotations to the images (step S350). Note that the process of step S350 is the same as the process of step S4 in FIG.

[0124] By performing the process shown in FIG. 38, the system can assist the user in selecting an image to be annotated. In particular, in this embodiment, a change in a target image is automatically detected based on a comparison of classified images, and classified images to be arranged on the screen are automatically selected based on the magnitude of the change. Therefore, even when selecting an image to be annotated from a large number of images, the number of images to be checked by the user can be reduced by omitting the display of similar images, and the burden on the user can be significantly reduced. Furthermore, since the display of similar images is omitted, it is possible to avoid unintentionally annotating only similar images, and it is possible to create teacher data with high learning efficiency.

[0125] In this embodiment, an example of automatically selecting classified images to be arranged on the screen based on the magnitude of change has been described, but this selection does not necessarily have to be performed automatically by the system. For example, even if the system does not perform automatic selection, if each classified image is displayed in chronological order, the user can visually grasp the amount of change at each time. For this reason, as described above in the third embodiment, the user only needs to annotate images that are determined to have a large amount of change, and may manually select and hide images that are not candidates for images to be annotated. In addition, in this embodiment, since the rate of change between images is calculated, change rate information may be displayed near the multiple classified images displayed in chronological order. The user may manually select images to be hidden based on the displayed change rate information, or may select images to be annotated based on the change rate information.

[0126] As described above, the system according to this embodiment, like the system according to the second embodiment, can assist the user in selecting images before transitioning to the annotation screen by using a chronological display of classified images.

[0127] [Fourth embodiment] FIG. 41 is a flowchart showing an example of processing performed by the system according to this embodiment. FIG. 42 is a diagram for explaining a method for determining the transmittance of a classified image. Hereinafter, processing performed by the system according to this embodiment will be described with reference to FIG. 41 and FIG. 42. Note that the system according to this embodiment supports the selection of an image to which an annotation should be added by performing the processing shown in FIG. 41. The system according to this embodiment (hereinafter, simply referred to as the system) is different from the system according to the second embodiment in that it performs the processing shown in FIG. 41 instead of the processing shown in FIG. 29, but is similar to the system according to the second embodiment in other respects. Note that the processing shown in FIG. 41 is started, for example, by the processor 131 executing the program 132a.

[0128] When the process shown in Fig. 41 is started, first, the system performs time lapse photography (step S410), and obtains a plurality of pieces of classification information obtained by the time lapse photography (step S420). Note that the processes in steps S410 and S420 are similar to the processes in steps S200 and S210 in Fig. 29.

[0129] Thereafter, the control device 130 selects the transmittance of the multiple classified images to be arranged on the screen based on the comparison of the multiple classification information acquired in step S420 (step S430). Here, the control device 130 may select a high transmittance for an image in which a large change has occurred, and a specific selection method is not particularly limited. As shown in FIG. 42, the control device 130 may, for example, calculate the change rate between two adjacent images in multiple classified images (images Pa, Pb, Pz (or Py), ...) arranged in chronological order, and select a high transmittance for classified images (in this example, images Pz and Py) in which the change rate is equal to or greater than a threshold value. Note that the transmittance may be adjusted for each image by comparing images (for example, see image Pz), or the transmittance may be adjusted for each composite image by comparing images (for example, see image Py). Also, the transmittance may be adjusted for each predetermined area (for example, 10 x 10 pixels) in the image by comparing.

[0130] When the transparency is selected, the control device 130 arranges the multiple classification images on the screen at the selected transparency based on the shooting times of the corresponding multiple target images (step S440). Here, the control device 130 arranges the multiple classification images at the selected transparency by superimposing them on the corresponding multiple target images. This allows the target image with a large change to be seen through the classification images, and can be used as a reference for selecting images to be annotated.

[0131] After the multiple classified images are arranged on the screen, the system then accepts an input from the user and adds annotations to the images (step S450). Note that the process of step S450 is similar to the process of step S4 in FIG.

[0132] As described above, the system according to this embodiment, like the system according to the second embodiment, can assist the user in selecting images before transitioning to the annotation screen by using a chronological display of classified images.

[0133] [Fifth embodiment] FIG. 43 is a flowchart showing an example of processing performed by the system according to the present embodiment. FIG. 44 is a diagram showing yet another example of an annotation screen. FIG. 45 to FIG. 47 are diagrams for explaining a method for adjusting an annotation. Hereinafter, processing performed by the system according to the present embodiment will be described with reference to FIG. 43 to FIG. 47. Note that the system according to the present embodiment supports the selection of an image to which an annotation should be added by performing the processing shown in FIG. 43. The system according to the present embodiment (hereinafter, simply referred to as the system) is different from the system according to the second embodiment in that the system performs the processing shown in FIG. 43 instead of the processing shown in FIG. 29, but is similar to the system according to the second embodiment in other respects. Note that the processing shown in FIG. 43 is started, for example, by the processor 131 executing the program 132a.

[0134] When the process shown in Fig. 43 is started, the system first performs time lapse photography (step S510) and acquires a plurality of pieces of classification information obtained by the time lapse photography (step S520). The system further arranges the generated plurality of classification images on the screen based on the shooting times of the plurality of target images corresponding to the plurality of classification images, as shown in Fig. 31 (step S530). Note that the process from step S510 to step S530 is similar to the process from step S200 to step S220 in Fig. 29.

[0135] Thereafter, when the user selects a target image to which annotations should be added based on the multiple classification images arranged on the screen, the control device 130 adds annotations to the target image selected by the user (step S540). Here, the control device 130 may automatically add annotations to the target image selected by the user based on the classification image corresponding to the target image selected by the user. Specifically, the control device 130 may add annotations to an area belonging to a predetermined class. For example, the control device 130 may identify an area belonging to a predetermined class based on the classification image corresponding to the target image selected by the user, and add annotations to the identified area of ​​the target image selected by the user. Note that the control device 130 may add annotations to an area belonging to a predetermined class and classified with high reliability.

[0136] When the annotation is automatically added, the control device 130 causes the display device 134 to display the relationship between the automatically added annotation and the class (step S550). Here, the control device 130 may cause the display device 134 to display, for example, an annotation screen shown in Fig. 44. The annotation screen shown in Fig. 44 includes setting 1001 which is a setting for automatically adding annotations, setting 1002 which is a setting for manually adding annotations, and an image 1003 with annotations added. Setting 1001 indicates the relationship between classes and annotations, and in this example, a setting is shown for automatically adding annotations to the second class from the top.

[0137] As the teacher data, an image with annotations added to a target image is used, but the image with annotations 1003 displayed on the annotation screen is not limited to an image with annotations added to a target image. As shown in Fig. 44, an image with annotations added to a classification image may be displayed. Also, an image with annotations added to an image in which a classification image and a target image are superimposed may be displayed, or an image with annotations added to a target image may be displayed. That is, it is sufficient that at least one of the classification image and the target image (an image with annotations added) is displayed.

[0138] The user checks the annotated image 1003 on the annotation screen and determines whether the annotation has been applied appropriately. Then, when the user recognizes the need to correct the annotation, the user instructs the control device 130 to correct the annotation by performing an operation such as changing the automatic annotation assignment setting or enabling the manual annotation assignment setting. The control device 130 accepts the user's input and corrects the annotation (step S560).

[0139] In the following, an example is shown in which the user inputs an annotation correction operation by clicking with a mouse, but the user may also input directly with a finger or a stylus. Information (annotation information) input by the user via the input device 133 is received by a reception unit of the control device 130. The reception unit is, for example, a processor within the control device 130. When outputting the received annotation information to the display device, the reception unit may appropriately convert the information into a form suitable for output.

[0140] 45 shows how a user modifies an annotation by changing the automatic annotation setting. Here, by changing from setting 1001, which modifies only the second class, to setting 1001a, which modifies the second and third classes, the annotated image changes from image 1003 to image 1003a. In this way, a user may modify an annotation by changing the automatic annotation setting, and in this case, the modification can be easily made by simply changing the setting.

[0141] FIG. 46 also shows how a user activates the manual annotation setting and annotates any region of an image. When a user changes the manual annotation setting from OFF to ON (changing the manual annotation setting from setting 1002 to setting 1002a), grid lines are displayed on the annotated image to assist in manually annotating the image. The grid lines are lines that divide the image into a number of small regions (N×M small regions in this example) specified in the setting, and the small regions are regions surrounded by the grid lines. Annotations that are manually added can be added in small region units. That is, the control device 130 can add annotations to the target image in small region units surrounded by grid lines according to the user's input. Note that the small region units may be the same as or different from the units of the target region used when generating the classification information. In the example shown in FIG. 46, the image to which annotations have been added has changed from image 1003 to image 1003b by manually adding annotations. In this way, the user may enable the manual annotation setting to modify the annotation, in which case the annotation can be modified with a greater degree of freedom.

[0142] Also, the user may change the number of grids as necessary, for example, as shown in FIG. 47, from setting 1002a to setting 1002b. That is, the control device 130 may change the size of the small region according to the user's input. In the example shown in FIG. 47, the image to which annotations have been added changes from image 1003b to image 1003c by manually adding annotations while changing the number of grids. In this way, the user can efficiently add annotations by adding annotations with a relatively small number of grids. On the other hand, by adding annotations with a relatively large number of grids, it is possible to deal with cases where detailed region designation is required, such as the edge of a cell, and annotations can be added accurately according to the shape of the region to be annotated. Therefore, by manually adding annotations while changing the number of grids as necessary, efficient and appropriate annotations can be performed. Furthermore, by displaying grid lines, the user can accurately grasp which range is to be annotated by one selection.

[0143] As described above, the system according to this embodiment can achieve the same effect as the system according to the second embodiment. Moreover, according to the system according to this embodiment, annotations are automatically added to images selected by the user, so that the burden of annotation work on the user can be significantly reduced. Furthermore, the reliability of annotation can be maintained at a high level equivalent to that of manual annotation by the user checking the automatically added annotations and correcting only the necessary parts. Therefore, appropriate annotation can be efficiently performed. Furthermore, by allowing the user to freely set the unit of annotation, the efficiency and accuracy of annotation work can be compatible at an even higher level.

[0144] The above-mentioned embodiments are specific examples shown to facilitate understanding of the invention, and the present invention is not limited to these embodiments. Modifications of the above-mentioned embodiments and alternative forms that replace the above-mentioned embodiments may be included. That is, each embodiment can have its components modified within the scope of its purpose and scope. In addition, new embodiments can be implemented by appropriately combining multiple components disclosed in one or more embodiments. In addition, some components may be deleted from the components shown in each embodiment, or some components may be added to the components shown in the embodiment. Furthermore, the processing procedures shown in each embodiment may be performed in a different order as long as there is no contradiction. That is, the system, method, and program for supporting annotation of the present invention can be variously modified and changed within the scope of the claims.

[0145] In the above-described embodiment, an example has been shown in which the classified images are arranged in a chronological order and displayed to allow the user to easily grasp changes that have occurred in the target image. However, the control device 130 may display the classified images one by one in chronological order. Furthermore, instead of or in addition to the chronological display of the classified images, the control device 130 may display on the display device 134 a graph G1 showing the change over time in the area ratio of the class and a glass G2 showing the change over time in the growth rate of the class, as shown in FIG. 48. Such a display can also assist the user in grasping changes that have occurred in the target image.

[0146] In the above-described embodiment, the photographing device 100 that acquires the target image is an apparatus that photographs cultured cells in an incubator, but the photographing device is not particularly limited. For example, it may be an endoscope, a biological microscope, or an industrial microscope. The target image is not limited to an image of cultured cells, and may be, for example, an image used in pathological diagnosis, or an image of an industrial product. The photographing device and the control device may be connected via the Internet. As shown in FIG. 49, the system 300 that supports annotation may include multiple photographing devices 310 (photographing device 311, microscope 312, photographing device 313) and may be connected to the control device 320 via the Internet 330. The control device 320 may arrange classified images on the screen of a client terminal 340 (client terminal 341, client terminal 342, client terminal 343) that is accessed via the Internet. Furthermore, the control device 320 may function only as a classification unit that generates classification information, and the client terminal may function as a control unit that arranges classified images on the screen. That is, the classification unit and the control unit may be provided in any device in the system, and these roles may be shared by different devices. Annotation input by a user may be accepted from the client terminal 340. In this case, the client terminal 340 may transmit information regarding the accepted annotation to the control device 320. A processor included in the client terminal 340 or the control device 320 may convert data into a form suitable for each device when exchanging data between them. In this case, the user can make annotations regardless of location. [Explanation of symbols]

[0147] 100, 310, 311, 312, 313 Imaging device 130, 320 Control device 131 processors 132 Storage device 132a Program 132b Trained Data 133 Input Devices 134 Display device 200, 300 systems C container S sample

Claims

1. A system for supporting image annotation, comprising: a classification unit that generates classification information by classifying a plurality of target regions constituting a target image, which is a candidate image to which an annotation is to be added, based on features that appear in the target image; a control unit that displays a classification image in which the classification information is visualized on a screen of a display device so as to be contrasted with the target image; The control unit is superimposing the classification image on the target image; A transmittance of the classification image is determined based on the reliability of the classification image. A system characterized by:

2. A system for supporting image annotation, comprising: a classification unit that generates classification information by classifying a plurality of target regions constituting a target image, which is a candidate image to which an annotation is to be added, based on features that appear in the target image; a control unit that displays a classification image in which the classification information is visualized on a screen of a display device so as to be contrasted with the target image; The classification information includes, for each of the plurality of target regions, Class information indicating the classified class; score information indicating a probabilistic score for the classification into the class; The control unit is superimposing the classification image on the target image; determining transmittances of a plurality of classification regions constituting the classification image based on the score information of corresponding target regions among the plurality of target regions; A system characterized by:

3. In the system according to claim 1 or 2, The control unit changes a transparency of the classification image superimposed on the target image in response to a predetermined instruction. A system characterized by:

4. The system according to any one of claims 1 to 3, The control unit arranges, on a screen of the display device, a plurality of classified images obtained by visualizing a plurality of pieces of classification information generated by the classification unit, the plurality of classified images corresponding to a plurality of target images photographed at different times, based on the photographing times of the plurality of target images. A system characterized by:

5. 5. The system of claim 4, The control unit selects the plurality of classification images to be arranged on the screen of the display device based on a comparison of the plurality of classification information generated by the classification unit. A system characterized by:

6. In the system according to claim 4 or claim 5, The control unit is superimposing the plurality of classification images on the corresponding plurality of target images; The transmittance of the plurality of classification images is determined based on a comparison of a plurality of classification information corresponding to the plurality of classification images. A system characterized by:

7. A system according to any one of claims 1 to 6, the classification information includes class information indicating a classified class, The control unit changes a display mode assigned to the class information in the classification image in response to a predetermined instruction. A system characterized by:

8. A system according to any one of claims 1 to 7, The target image is an image of cells cultured in a culture vessel, The control unit is A vessel image simulating the culture vessel is arranged on a screen of the display device, The classification image corresponding to the target image is arranged on the container image based on the photographing position of the target image on the culture container. A system characterized by:

9. 9. The system of claim 8, the culture vessel is a multi-well plate, the container image includes a plurality of well images that mimic a plurality of wells included in the multi-well plate; The control unit arranges a plurality of classification images corresponding to a plurality of target images obtained by photographing cells in the corresponding wells at positions corresponding to the plurality of well images. A system characterized by:

10. A system according to any one of claims 1 to 9, The classification unit includes all or a part of a neural network that constitutes an autoencoder. A system characterized by:

11. A system according to any one of claims 1 to 10, The control unit adds an annotation to the target image based on the classified image. A system characterized by:

12. A system according to any one of claims 1 to 11, The control unit is Placing grid lines on at least one of the target image and the classification image arranged on the screen of the display device; According to a user's input, annotations are added to the target image in units of small regions enclosed by the grid lines. A system characterized by:

13. 13. The system of claim 12, The control unit changes the size of the small region according to an input from the user. A system characterized by:

14. In the system according to claim 2, The control unit generates the distance from the center of gravity of the cluster as the probabilistic score. A system characterized by:

15. 1. A method for assisting in annotation of an image, comprising: generating classification information by classifying a plurality of target regions constituting a target image, which is a candidate image to which an annotation is to be added, based on features appearing in the target image; A classification image in which the classification information is visualized is arranged on a screen of a display device so as to be contrasted with the target image; The comparative arrangement is superimposing the classification image on the target image; determining a transmittance of the classification image based on the reliability of the classification image; A method comprising:

16. A method for assisting in annotation of an image, comprising: generating classification information by classifying a plurality of target regions constituting a target image, which is a candidate image to which an annotation is to be added, based on features appearing in the target image; A classification image in which the classification information is visualized is arranged on a screen of a display device so as to be contrasted with the target image; The classification information includes, for each of the plurality of target regions, Class information indicating the classified class; score information indicating a probabilistic score for the classification into the class; The comparative arrangement is superimposing the classification image on the target image; determining transmittances of a plurality of classification regions constituting the classification image based on the score information of corresponding target regions among the plurality of target regions. A method comprising:

17. A program for supporting image annotation, comprising: generating classification information by classifying a plurality of target regions constituting a target image, which is a candidate image to which an annotation is to be added, based on features appearing in the target image; A classification image in which the classification information is visualized is arranged on a screen of a display device so as to be contrasted with the target image; The comparative arrangement is superimposing the classification image on the target image; determining a transmittance of the classification image based on a reliability of the classification image; A program that causes a computer to execute a process.

18. A program for supporting image annotation, comprising: generating classification information by classifying a plurality of target regions constituting a target image, which is a candidate image to which an annotation is to be added, based on features appearing in the target image; A classification image in which the classification information is visualized is arranged on a screen of a display device so as to be contrasted with the target image; The classification information includes, for each of the plurality of target regions, Class information indicating the classified class; score information indicating a probabilistic score for the classification into the class; The comparative arrangement is superimposing the classification image on the target image; determining transmittances of a plurality of classification regions constituting the classification image based on the score information of corresponding target regions among the plurality of target regions. A program that causes a computer to execute a process.

Images